How to monitor particular kit with Oversight: which sensor to use, where on the device to set up access, which user function and extraction rows read the response, and how to write the rules that turn it into an alarm. Search for a vendor, a product or an error message, or ask a question in your own words.
Echo agent for the echo sensors: is this leg of the network sound?
The echo sensors test one leg of a network: a LAN link, a leased line, a 4G or Starlink uplink, a tunnel between two sites. A small program, oversight-echo, runs on a machine at the far end of the leg, and a probe at the near end tests against it. Every packet and every byte the probe sends is numbered and its contents known, so anything lost, reordered, duplicated or damaged on the way is counted, in each direction.
There are two sensors, usually used together:
- UDP echo stream sends a set number of packets, one a millisecond, and gets each one back. It answers whether the leg is sound: loss, jitter, round trip, and whether the loss is scattered (weather, a dirty dish, a failing cable) or one outage (a handover, a cell change).
- TCP echo transfer sends a set size up and then back down. It answers how fast the leg is in each direction and how hard TCP had to work for it: retransmits and stalls.
Everything is layer 3. Nothing uses ICMP, which is not reliably forwarded across WAN links.
What it measures
The UDP sensor reads packets sent, received and lost, with the loss split by direction, since the echo counts what reaches it; packets out of order, duplicated or corrupt; round trip minimum, average and maximum; jitter; the longest run of lost packets; and scattered loss, which is the loss leaving out any single gap over 300 ms.
The TCP sensor reads upload and download speed and time, retransmits in each direction as counted by the sending end's own TCP stack, the longest stall in each direction, and corrupt bytes, which TCP should never let through.
What to alarm on
Damaged data is never normal, so any corrupt packet or byte is critical. Scattered loss is the figure rain fade and a dirty dish move, and a phone call starts breaking up at a few per cent, while satellite handovers do not trip it, so it is the one to warn on, with the longest gap alongside where outages matter as well. On a leg that should be clean, retransmits climbing beyond a few dozen deserve a warning. An echo that is not running, or a leg that passes nothing, gives no reading at all, which is critical as for any sensor.
Pinning a fault down
Run two echoes at once on the same path, one near and one far, each with its own device and sensor from the same probe. When the far one shows loss and the near one does not, in the same minute, the fault is between them. Move the echoes and repeat. On a tunnel, an echo through the tunnel and another reached directly over the same link tell the tunnel's faults apart from the link's.
A sensor at 1000 packets every 5 minutes is a light watch. If it shows trouble, raise it to 10000 packets every minute for the hunt, and back again when the fault is found.
Running the echo
The echo is a single program for Linux, Windows and macOS, with nothing to configure, run on a machine at the far end of the leg. It can run in a terminal for a quick test, or as a locked-down service for a watch lasting days, and it is its own client too, so an engineer can run the same tests by hand and see the same figures the sensors record. It has no password: it is meant for private legs and for days at a time, not for the internet, so stop it when the work is done.
The test runs from whichever probe polls the sensor, so use a probe group whose probes sit at the near end of the leg, ideally one probe, or the figures describe a different path each time the sensor moves.
Both are 2 credits a reading. Mind the data on a metered link: 1MB each way every minute is about 1.4GB a day in each direction. The UDP sensor uses very little.
Before you start
- Run the echo at the far end. Download the build for the machine (Linux on amd64, arm64 or armv7, Windows on amd64, macOS on arm64 or amd64) and run it. Nothing to configure.
The command or address for this step is for signed-in customers. Sign in to see it.It listens on port 29732, TCP and UDP, until it is stopped.
-portpicks another port. It logs each test it serves. To leave it running on a Linux machine, through reboots, see Running it as a service on Linux below. - Open the port. The probe must reach the echo on 29732, TCP and UDP. Windows asks the first time the echo runs; a firewall in between needs a rule.
- The echo has no password. It is meant for private legs and for days at a time, not for the internet. Stop it when the work is done.
- Find a forgotten echo by scanning for port 29732.
- Mind the probe groups. The test is from whichever probe polls the sensor. Use a group whose probes sit at the near end of the leg, ideally one probe, or the figures describe a different path each time the sensor moves.
Running it as a service on Linux
For a quick test, running the echo in a terminal is enough. For a watch lasting days or longer, on a Linux machine at the far end of the leg, run it under systemd so that it survives a logout and a reboot. Unlike the Linux agent, the echo has no --install, so this is done by hand, as root:
Put the Linux build for the machine at /opt/oversight/echo, make it executable (chmod 755) and check it runs with /opt/oversight/echo -version:
Then write the service as /etc/systemd/system/oversight-echo.service:
[Unit]
Description=Oversight echo
After=network-online.target
Wants=network-online.target
[Service]
ExecStart=/opt/oversight/echo
DynamicUser=yes
Restart=always
RestartSec=5
NoNewPrivileges=yes
CapabilityBoundingSet=
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
RestrictAddressFamilies=AF_INET AF_INET6
[Install]
WantedBy=multi-user.target
For another port, put it on the end of the ExecStart line, as /opt/oversight/echo -port 40000. DynamicUser runs it as a throwaway user that systemd makes for it, so there is no account to create, and it can write nothing on the machine. Start it, and have it start at boot:
systemctl daemon-reload
systemctl enable --now oversight-echo
systemctl status oversight-echo
Its log, one line for each test it serves, is in the journal: journalctl -u oversight-echo. Check it from the machine itself before going near Oversight: /opt/oversight/echo -c 127.0.0.1 should report no loss.
The machine's firewall needs a rule for both TCP and UDP, or the check above works while the sensor sees nothing:
firewall-cmd --permanent --add-port=29732/tcp --add-port=29732/udp && firewall-cmd --reload # firewalld
ufw allow 29732 # ufw
To upgrade, download the new build over /opt/oversight/echo as above and systemctl restart oversight-echo. When the work is done, take it away, and close the port again:
systemctl disable --now oversight-echo
rm /etc/systemd/system/oversight-echo.service /opt/oversight/echo
systemctl daemon-reload
Setting it up
Create a device for the machine running the echo, with its address, and add the sensors to it.
| Setting | What to put |
|---|---|
| Packets (UDP) | How many to send, up to 10000. 1000, the default, takes a second. Start there and raise it to pin a fault down. |
| Size each way (TCP) | Written as 512KB, 1MB or 1GB, up to 1GB; the default is 1MB. A small size mostly measures TCP getting going, so a larger one reads closer to the leg's real speed, at the cost of the data used. |
| Port | Empty for 29732. |
| Interval | Every 5 minutes is a sensible watch; every minute while hunting a fault. The packets must finish within the interval, so a short interval limits the count. |
What it reads
UDP echo stream
| Slot | Reading |
|---|---|
| SV01, SV02, SV03 | Sent, received and lost, in packets. |
| SV04 | Loss, as a percentage of those sent. |
| SV05, SV06 | Lost on the way out and lost on the way back. The echo counts what reaches it, which splits the loss by direction. |
| SV07, SV08, SV09 | Out of order, duplicated and corrupt (arrived, but not as sent). |
| SV10, SV11, SV12 | Round trip minimum, average and maximum, in milliseconds. |
| SV13 | Jitter: how much each packet's round trip differs from the one before, averaged, in milliseconds. |
| SV14 | Longest gap: the longest run of lost packets, in milliseconds (one packet is one millisecond). |
| SV15 | Scattered loss: loss leaving out any single gap over 300 ms, as a percentage. |
TCP echo transfer
| Slot | Reading |
|---|---|
| SV01 | Size each way, in bytes. |
| SV02, SV03 | Upload and download, in Mbps. |
| SV04, SV05 | Upload and download time, in milliseconds. |
| SV06, SV07 | Retransmits up and down, counted by the sending end's own TCP stack. -1 where that end cannot say. |
| SV08, SV09 | Longest stall up and down: the longest wait between pieces arriving, in milliseconds. |
| SV10, SV11 | Corrupt bytes up and down. TCP checks its own data, so anything here got past it and is serious. |
Connect time and response time are filled in too: the average round trip for UDP and the whole run for TCP.
Rules worth having
- SV09 greater than 0 gives CRIT on the UDP sensor, and SV10 or SV11 greater than 0 gives CRIT on the TCP one. Damaged data is never normal.
- SV15 greater than 1 gives WARN, greater than 5 gives CRIT. Scattered loss is the figure rain fade and a dirty dish move, and a phone call starts breaking up at a few per cent. Satellite handovers do not trip it.
- SV14 greater than 2000 gives WARN if outages matter as well: two seconds with nothing through.
- SV06 or SV07 greater than a few dozen gives WARN for a leg that should be clean. A busy link retransmits a little; a sound one should not climb.
An echo that is not running, or a leg that passes nothing, gives no reading at all, and the failure threshold turns that into CRIT as for any sensor.
Running it by hand
The echo is its own client, so an engineer can run the same tests at a command line: oversight-echo -c 10.1.1.15 sends 1000 packets and then 1MB each way, and prints the same figures the sensors record. -udp 10000 and -tcp 100MB change how much; -udp 0 or -tcp 0 leaves that test out.
When it fails
| Error | What it means |
|---|---|
| REFUSED | The machine answered but nothing is listening on the port: the echo is not running, or is on another port. |
| RESPONSETIMEOUT | UDP: not one packet came back. The echo is not running, or a firewall drops the port, or the leg is down. TCP: the transfer stopped part way for longer than the response timeout; the message says how far it got. |
| CONNECTTIMEOUT | TCP: no answer at all to the connection, usually a firewall dropping the port or the machine being unreachable. |