Index size

How much disk the two indexes use, and which switches to turn off first.

How much disk the two Loghound indexes use per million log lines, and which switches to turn off first when your plan holds less history than you want.

01 · Disk per million log lines

These are estimates from field-count arithmetic rather than measurements of a live index, shown as a breakdown so you can check the reasoning and adjust for your own traffic. Real numbers depend heavily on how many distinct paths and addresses you see.

Per million hit documentsplain combinedrecommended format
docValues~105 MB~140 MB
Inverted index~60 MB~85 MB
Stored fields, excluding the raw line~105 MB~110 MB
The raw line copy~65 MB~165 MB
The catchall~275 MB~285 MB
Everything on~0.6 GB~0.79 GB
Raw copy and catchall off~0.27 GB~0.34 GB

The sessions index is much smaller: at eight to fifteen hits per session, roughly 50–90 MB per million hits. Daily rollups are about 100 KB a year, which is why keeping them forever is the default.

What to turn off, in order

  1. The catchall (ingest.catchall set to false in the configuration). The biggest single win, roughly 45% of the hits index. You lose the free-text search box in Sessions; every facet, filter and chart keeps working.
  2. The raw line copy (ingest.keep_raw set to false). About 11% on plain combined, 21% on the recommended format. You lose the ability to re-derive documents under a future ruleset.
  3. The retention window. The only lever that bounds growth rather than reducing a constant.
  4. Drop the stored query string if you never look at query strings.

Deleted documents do not return disk immediately — the index reclaims on merge. Both indexes are configured so that happens steadily rather than in one stall, and nothing forces a full merge, which would rewrite the whole index and need double the disk while it ran.

Reference

Loghound is open source and MIT licensed. Questions about the Opensolr half — the account, the indexes, the plan — go to opensolr.com/contact; questions about the software itself belong on GitHub.

Loghound Documentation