Retention & disk

Two mechanisms doing different jobs. One is a policy you choose. The other is what keeps Loghound running on a small plan instead of switching your own dashboard off.

A number of days, defaulting to ninety. Set it in Settings › Privacy as Delete hits and sessions older than, from 0 to 3,650 days; neither installer asks about it. The How much data you keep card shows it beside the size rule.

The deletion is done by a command on a daily timer, and it is a real deletion job rather than a paragraph of documentation saying you ought to have one. To run it now:

sudo -u loghound php /opt/loghound/bin/loghound-retention

And to see what it would remove without removing it:

sudo -u loghound php /opt/loghound/bin/loghound-retention --dry-run

The cutoff is computed as an explicit instant rather than handed to Solr as an expression, for two reasons: the same instant is used for both indexes so they cannot drift apart mid-run, and an absolute timestamp is what gets printed in the dry run — somebody about to delete data should see a date, not an expression. There is a floor of one day, because a rounding accident that produced half a day would delete the data the dashboard is currently showing, and there is no undo.

The daily rollup documents are kept indefinitely by default. They are aggregate counts with no per-visitor field in them, they cost a few hundred bytes each, and they are what makes long-term trends survive the deletion of the detail underneath.

An index that exceeds its disk quota is not throttled and does not merely refuse writes: the platform makes it answer every request with a refusal, reads included, until usage is back under the limit. So an unbounded Loghound would eventually fill the plan and switch its own dashboard off. Preventing that is the entire point of this mechanism.

• The trim runs before a write, when an index reaches 90% of its disk quota, and brings it back to 80% (both set as quota.disk_high_water and quota.disk_target), so the index never reaches the quota and the blackout never triggers on disk.
• It deletes by date, never by document count: a date cutoff is one cheap operation and it leaves an answer a human understands — “you have data from this date onward” — where “the oldest N documents” would need a sort, a page and a list of identifiers.
• There is a protected recent window — a day by default — that the trim will never touch whatever the arithmetic says.
• No single step may remove more than a bounded slice of the data’s timespan, however far over quota the index is, and there is a cooldown afterwards so the index’s own merges can catch up.

Because of all this, the Storage and bandwidth view presents disk as a fact about how far back your data goes rather than as an alarm. The one thing there that is an alarm is a trim that failed while the index is genuinely near the limit — because then the blackout really is coming.

The same daily run also cleans the local state database under var/, as its third pass:

• Expired lookup caches (location, network and reverse DNS of addresses), once past their lifetime: 30 days for location and network, 7 days for reverse DNS, by default.
• Visits that closed more than a day ago, and beacon reports already merged more than a day ago.
• Rate-limit records older than an hour.

The local database also holds the reader’s position in every log file and the visits still open, so it is not safe to delete by hand: the reader would restart each log at its current end and lose anything not yet read.

When several Loghound installations write into one pair of indexes, each one’s age rule deletes only its own documents, so a 90-day policy on one machine never removes a neighbour’s 365 days. If the run cannot tell whether the pair is shared, it assumes it is. If it is shared and this installation cannot name its own documents, the age rule does not run and the command says why.

Upstream of all this, Loghound can only see as far back as your own log files, and that is often much shorter than you assume.