One long-running service and two timers. The service is a daemon rather than a cron job because cron’s floor is sixty seconds, and the point of the tailer is that a request appears in the panel about five seconds after it was served.
sudo systemctl enable --now loghound-tail.service loghound-score.timer loghound-retention.timer
One command enables and starts all three and brings them back after a reboot. The panel repeats it under Settings › Finish setting up, together with whether ingestion is actually running.
| Unit | When | What it does |
|---|---|---|
| tail service | continuously | Polls each source once a second, parses new lines, enriches them, assigns them to sessions, and indexes them in batches. Handles rotation, truncation and inode reuse. |
| score timer | every 60s | Closes sessions idle past the timeout, publishes open ones so the panel is not empty for half an hour on a fresh install, merges the beacon data, computes the fingerprint clusters with one facet per hour bucket, runs the ruleset, writes the session documents and rebuilds the daily rollup. |
| retention timer | daily | Deletes past the retention window. This is the only thing in the product allowed to issue a deletion. |
On a box with no systemd, the installer falls back to cron for the two timers and says plainly that the daemon then needs a supervisor of your own.
As the loghound user, in the adm group for log read access, never as root. The units set no-new-privileges, a strict read-only system view, a private temporary directory, private devices, an empty capability set, a restricted set of address families, a system-call filter and a memory cap. The only writable path is the installation’s own var directory; the log directory is explicitly read-only. Application code is owned by root and only read by the service user, so a compromise of the daemon cannot rewrite the code that runs next time.
Figure 1 — the life of a session. Publishing open sessions is why the panel shows traffic within a minute of ingestion starting rather than after the idle timeout, which on a fresh install would read as a broken product.
The daemon writes a status file about itself, atomically, so a reader can never see a half-written one. Reading it is a read-only client of the running daemon — it does not talk to it:
loghound-tail --status --human
The three numbers to look at:
- Lag — how far behind the end of the file the reader is. Steady and small is healthy.
- Parse errors — every unparseable line is counted, never silently dropped, and a capped sample is written with its source file and byte offset. A sudden climb is usually a format change; it is also what a log-injection attempt looks like.
- Solr errors — indexing failures.
Plus a missed-rotation count, which is how the daemon reports honestly that a rotated file was compressed before it could drain it. Set delaycompress if you see it climbing.
The machine-readable form is the same command without the human flag, which is what to point a monitoring check at.
| Path | What is in it |
|---|---|
var/state.db | The tailer’s byte cursor in every source, open sessions, staged beacon rows, enrichment caches, rate-limit buckets. Not safe to simply delete — see retention. |
var/badlines.log | A capped sample of unparseable lines, with control characters stripped before writing — because that file will be read in a terminal and a crafted User-Agent should not be able to emit escape sequences into it. |
var/panel-jobs.db | The state of any stepped job, which is what makes one survive a page refresh. |
var/quota.json | The cached plan usage behind the storage meter. |
/var/log/loghound-install.log | Everything the installer did, in plain text. The API key is never written to it. |
| the daemon’s own journal | journalctl -u loghound-tail -n 50 --no-pager |
It would ingest its own beacon traffic and inflate every number it reports. The panel’s own log files belong to your logrotate, and the uninstaller deliberately leaves them alone.
# Is it running, and is it keeping up? loghound-tail --status --human systemctl status loghound-tail.service --no-pager # What did it say for itself? journalctl -u loghound-tail -n 50 --no-pager # Stop and start it (a reload onto a half-finished config is refused and logged) sudo systemctl stop loghound-tail.service sudo systemctl start loghound-tail.service # Run the scorer or the retention job by hand sudo -u loghound php /opt/loghound/bin/loghound-score sudo -u loghound php /opt/loghound/bin/loghound-retention --dry-run # Are the live indexes still in step with the code? Run after every upgrade. php /opt/loghound/bin/loghound-schema
When something is wrong, troubleshooting has the symptom-by-symptom list. The command reference has every flag.