Articles setup, updated 2026-11-16

Log sender: what matters from a host's own logs

The Oversight log sender is a small program that reads a Linux host's own logs, picks out the lines that matter using rules you write, a filter for each log, and sends just those to Oversight every five minutes, where they appear on the Logs page. Who logged in. Which addresses fail2ban banned. What a nightly job reported. It reads log files, the systemd journal (journald), or whatever is piped into it, and it can run as a service or once from cron.

Each thing it sends is a channel, with a name of its own, such as secure or fail2ban, and its own rules. One host can send several channels; each is its own service, with its own config.

For syslog from a site's routers and switches, use the Syslog sender instead. The two share the rules, the limits and the Logs page.

Flavours

Most hosts want the same few things, so they come ready-made, as flavours, and GEN adds more as they are asked for:

FlavourSends
secureSuccessful logins: SSH, su, the console and Cockpit. From /var/log/secure (RHEL, AlmaLinux, Rocky), /var/log/auth.log (Debian, Ubuntu) or, where neither exists, the journal.
fail2banEvery ban and unban, in every jail, from /var/log/fail2ban.log.

Installing is one command, as root, naming the flavours wanted. A host without the log a flavour reads skips it, so one command can name every flavour for a whole estate. For each one, the install finds the log, writes the channel's rules (keeping any you have already edited), makes the log readable by the sender the way the host already intends, checks that log rotation keeps it that way, labels what it wrote where SELinux is enforcing, and starts a service that follows the log through rotation.

The first install on a host prints a token, which GEN registers against your account; until then Oversight refuses what the host sends. A host has one token whatever it sends, shared by every channel on it and a syslog sender there, so a host is registered once.

Worked example: fail2ban

fail2ban's log is mostly noise: every address it notices, every jail starting, every ban it restores when it restarts. What matters is the bans and unbans:

2026-10-15 09:12:01,234 fail2ban.actions        [1234]: NOTICE  [sshd] Ban 203.0.113.7
2026-10-15 09:22:01,301 fail2ban.actions        [1234]: NOTICE  [sshd] Unban 203.0.113.7

The fail2ban flavour sends those two, in any jail and at any level, and nothing else. Found, already banned, start-up chatter and Restore Ban stay on the host.

Each line arrives in Oversight with its rule's name, so the Logs page can show just the bans. On a host whose SSH faces the internet that can be a few hundred a day.

Worked example: who logged in

The login log is mostly failed attempts by strangers, and housekeeping: cron opening sessions every minute, sudo, systemd starting each user's session. What matters is who actually got in:

Oct 15 12:52:22 mail sshd-session[56905]: Accepted password for root from 172.16.128.40 port 52499 ssh2
Oct 15 09:06:02 mail su[3401]: pam_unix(su-l:session): session opened for user root(uid=0) by richard(uid=1000)

The secure flavour sends successful SSH logins by any method, su, logins at the console and logins to Cockpit, each under its own rule: ssh, su, console, cockpit.

Two things are worth knowing:

  • Newer OpenSSH logs each connection as sshd-session, not sshd: AlmaLinux 9 does now. The rule takes either. A rule written for one alone misses every login on the other, silently.
  • sudo is left out on purpose. It is logged for every command, and a host run by scripts or Ansible would send thousands. A rule of your own can send it all the same.

A machine that logs in all day by itself, a backup server say, can be kept out with an exclude rule. On a host that keeps its logs only in the journal, minimal Debian 12 for one, the flavour reads the journal instead, and the same rules work.

Your own channel

Anything else is a channel of your own: a name, where to read, and rules. It can follow an application's log file through rotation, or the systemd journal for one service, and Oversight can be told how often to expect it, so GEN is told if it stops. A channel with no rules at all sends every line, for input that is already exactly what should go.

The sender does not have to run all the time, either. Given a file, or something piped into it, it reads to the end, sends what matches and stops, so a job that produces a short report each hour can send it from cron as a channel, and a job that stops running is noticed.

Rules can be tried before they are relied on: a dry run over a log that is already there prints what each rule would have sent, a month of logins in a second, and sends nothing.

Things to know

  • Log rotation is followed whichever way it is done: renamed and recreated, or copied and emptied in place. A log that is deleted is waited for, and read from its start when it comes back.
  • Permission is the commonest fault: the service runs as oversight, and many logs are root's alone. The journal says permission denied and names the file. Once it is put right, the sender carries on from that moment, without sending the log's history.
  • Limits, which no config can raise: a batch covers at least five minutes and holds at most 8,192 lines of up to 4KB each. Lines beyond the limit are counted, and the Logs page shows the batch in orange. Batches that cannot be sent wait an hour on the host, then are counted and shown as a gap.
  • Only plain text goes. Accented letters and anything that is not plain ASCII are taken out of a line before the rules see it.

Installing a flavour

Most hosts want the same few things, so they come ready-made, as flavours. As root, naming the ones you want:

The command or address for this step is for signed-in customers. Sign in to see it.

/opt/oversight/sender --flavours lists what there is; GEN adds more as they are asked for. For each one named, the install:

  1. Finds the log on this host. A host without it skips that flavour, so one line can name every flavour for a whole estate.
  2. Writes the channel's config, /etc/oversight/sender-secure.json and so on, with its rules. A config already there is kept as it is, so rules you have edited are never overwritten.
  3. Makes the log readable by the oversight user, the way the host already intends: on Debian and Ubuntu it joins the group that reads logs (adm); on RHEL and AlmaLinux, where the log is root's alone, the log's group becomes oversight. It says which it did.
  4. Checks that logrotate will keep that when it rotates the log, and says so, or prints the one line to change if it would not.
  5. On a host with SELinux enforcing, labels what it wrote correctly and checks for denials once the service is running. AppArmor does not affect it.
  6. Installs and starts oversight-sender-secure.service and so on, which start at boot and follow the log through rotation.

It prints a token the first time:

token: 9f2c6e0a4b7d1385c2e9f04a6b8d3e7f1a5c9b2d6e0f4a8c3b7d1e5f9a2c6b0d
register this under Probes > Senders before it can upload

Send that token to GEN. We register it against your account; until then Oversight refuses what the host sends. A host has one token, whatever it sends: every channel on it, and a syslog sender there, share it, so a host is registered once. Running the same line again upgrades the program and restarts its services, and changes nothing else.

Worked example: fail2ban, the config

The fail2ban flavour's config:

{
  "channel": "fail2ban",
  "token": "9f2c6e0a...",
  "files": ["/var/log/fail2ban.log"],
  "include": [
    {"name": "ban", "match": "fail2ban\\.actions\\b.*\\] Ban "},
    {"name": "unban", "match": "fail2ban\\.actions\\b.*\\] Unban "}
  ]
}

Each line arrives in Oversight with its rule's name, so the Logs page can show just the bans. For bans alone, delete the unban rule.

By hand, without the flavour, the same thing is three steps on AlmaLinux:

  1. Write that config to /etc/oversight/sender-fail2ban.json, with a token from openssl rand -hex 32 or this host's existing one.
  2. Install it with the install line, adding --config /etc/oversight/sender-fail2ban.json:
    The command or address for this step is for signed-in customers. Sign in to see it.
  3. Let it read the log, which is root's alone: chgrp oversight /var/log/fail2ban.log && chmod 0640 /var/log/fail2ban.log. AlmaLinux's logrotate recreates each log as the old one was, so this lasts.

Worked example: who logged in, the rules

The secure flavour's rules:

"include": [
  {"name": "ssh", "match": "\\bsshd(-session)?\\[\\d+\\]: Accepted \\S+ for \\S+ from "},
  {"name": "su", "match": "\\bsu\\[\\d+\\]: pam_unix\\(su(-l)?:session\\): session opened for user "},
  {"name": "console", "match": "\\blogin\\[\\d+\\]: pam_unix\\(login:session\\): session opened for user "},
  {"name": "cockpit", "match": "\\bcockpit-session\\[\\d+\\]: pam_unix\\(cockpit:session\\): session opened for user "}
]

Two things are worth knowing:

  • Newer OpenSSH logs each connection as sshd-session, not sshd: AlmaLinux 9 does now. The rule takes either. A rule written for one alone misses every login on the other, silently.
  • sudo is left out on purpose. It is logged for every command, and a host run by scripts or Ansible would send thousands. To send it all the same, add {"name": "sudo", "match": "\\bsudo\\[\\d+\\]: pam_unix\\(sudo:session\\): session opened for user "}.

If one machine logs in all day by itself, a backup server say, keep it out with an exclude: "exclude": [{"name": "backups", "match": "Accepted \\S+ for root from 10\\.1\\.1\\.20 "}].

On a host that keeps its logs only in the journal, minimal Debian 12 for one, the flavour reads the journal instead, through journalctl, from the moment it starts. The same rules work, because the journal's lines are written out the same way.

Your own channel

Anything else is a channel of your own: a name, where to read, and rules. For an application's log, run the install line as root with --channel myapp --follow /var/log/myapp/error.log:

The command or address for this step is for signed-in customers. Sign in to see it.

That installs oversight-sender-myapp.service and writes /etc/oversight/sender-myapp.json with a token. Add the rules to that file; the service picks the change up within five minutes, or at once with systemctl reload oversight-sender-myapp. Every channel needs a name: without one, nothing is installed. The oversight user must be able to read the file, as above.

KeyWhat it does
channelThe name. Lower-case letters, digits, dots, dashes and underscores.
tokenThe host's token.
filesLog files to follow, from now on, through rotation.
journalInstead of files, journal matches to follow, such as ["_SYSTEMD_UNIT=myapp.service"].
intervalSeconds between uploads: 300 or more.
expectHow often Oversight should hear from it, in seconds. Following a log, every interval, and GEN is told if it stops; 0 for never late.
exclude, includeThe rules, as for the Syslog sender, except that there is no severity or facility in a plain log file, and source is the file's path.

A channel with no rules at all sends every line. That is for input that is already exactly what should go, below.

Once, from cron

The sender does not have to run all the time. Given a file, or something piped into it, it reads to the end, sends what matches and stops. A job that produces a short report each hour can send it as a channel:

0 * * * * /usr/local/bin/clocking-report | /opt/oversight/sender --channel clocking --expect 1h

With no rules, every line of the report goes. --expect 1h tells Oversight to expect it hourly, so a job that stops running is noticed. Set the token once in /etc/oversight/sender-clocking.json as {"channel": "clocking", "token": "..."}; the install line with --channel clocking writes that for you, with no service, since there is nothing to follow. The cron job must run as root or as a member of the oversight group.

Two runs closer together than five minutes do not both send: the second is held and goes with the next run. And a run sends everything it is given: give it the same growing file every hour and it sends the whole file every hour. For a log that grows, follow it as a service.

Trying rules before relying on them

A dry run reads, applies the rules and prints what it would have sent, and sends nothing. Against the whole of a log that is already there:

/opt/oversight/sender --dry-run --config /etc/oversight/sender-secure.json /var/log/secure

It ends with a count: lines read, sent by each rule. A month of logins in a second. | jq -r '.lines[].msg' shows just the lines. --validate checks a config for mistakes without running anything.

In a dry run every line is given the time it was read, so over an old log they all carry the same time; the real time is in the line itself. Following a log as a service, the two are the same.

If it does not work

Each channel's service writes to the journal: journalctl -u oversight-sender-secure. One line every five minutes, or each run, is the healthy state:

uploaded secure, 12:20:00 to 12:25:00: 1 shipped of 41 received
The journal saysWhat to do
open /var/log/...: permission deniedMake the log readable by oversight, as above. Running the flavour again does it for you.
Oversight does not know this tokenThe token is not registered yet. Send it to GEN.
config: ...A mistake in the config, named. The last good one carries on meanwhile.
... held ...: under 5 minutes since this channel's lastNothing wrong: a run from cron came within five minutes of the last, and the next run sends it.
journalctl: ...Reading the journal failed. The sender starts journalctl again ten seconds later; check oversight is in systemd-journal.

Uploading, but nothing shipped: the rules do not match what the log really says. A dry run over the log shows exactly what they catch, and the log itself shows what they should.