Two Solr 9 indexes, provisioned and managed by Loghound. Field names are a contract: the parser writes them, the scorer reads them and the panel queries them, so renaming one is a breaking change to three components at once.
The session drill-down, the latency percentiles, and the fingerprint facet the scorer runs
Everything else in the dashboard
The names are generated per installation rather than fixed — see provisioning. The identifiers stored in these fields are on identifiers and labels.
These configsets are pushed when the indexes are provisioned, and upgrading the code does not re-push them. A release that adds a field therefore writes a field the live schema does not declare — and the catch-all in section 04 silently discards it. bin/loghound-schema is what puts that right, and upgrading explains why running it is not optional.
- Not stored by default; retrieve from docValues. The dashboard facets and sorts on nearly every field, so docValues is being paid for regardless. Storing as well writes the value a second time. Only fields a human reads in the drill-down are stored.
- No norms on any string field. Norms exist for relevance scoring by length, and nothing here is ranked on a string field — they are filtered and faceted. At forty-six string fields and ten million documents that would be 460 MB of nothing.
- No term frequencies or positions on string fields. They answer “how often” and “how near”, and an exact match on an address needs neither. Roughly halves the postings size for those fields.
- No stock sprawl. Solr’s default configset ships a catchall plus about thirty dynamic fields and a rule copying every value into it. On a schema with seventy named fields that makes the catchall the largest single component of the index — for a field nothing in Loghound queries. Both are deleted.
Figure 1 — two indexes with mirror-image workloads, which is why they are configured differently. Ten to twenty times fewer documents on the right is what makes “bots by network” one cheap query rather than a scan of the hit stream.
A hash of the installation id, the source file and the byte offset. Deterministic, so replaying a rotated file overwrites instead of duplicating. The installation id is in it for a reason: two machines tailing identically named files would otherwise produce identical ids, and a duplicate key in Solr is a delete-and-add — each would silently have overwritten the other’s traffic, one request at a time, with no error anywhere.
The most expensive docValues field in the schema, and non-negotiable: the distinct count per fingerprint is the product’s headline metric.
Stored only — not indexed, no docValues. Query strings are unbounded attacker-controlled text with a near-unique value per request; indexing them would build a term dictionary the size of the corpus and faceting on it would be meaningless. Search terms are its deliberate opposite.
Absent when the log format does not carry it, never zero — a zero would drag every latency percentile toward the floor.
Present only when the operator logs them, and absent rather than empty string when they are not. An empty string is a real value that facets, sorts and matches, and it would make “no header” indistinguishable from “an empty header” — two rules worth 70 points or more turn on exactly that distinction, and both go silent rather than guessing.
Indexed only, so it answers bounding-box queries but cannot be retrieved or aggregated — which is why the map is country resolution. It is also absent rather than wrong when the service returned only a country-level centroid.
From the geolocation service when it names one, otherwise derived locally from the country when the country has exactly one zone. Absent for a multi-zone country, because guessing one would make the mismatch rule fire on people who did nothing wrong.
Filled only when the log carries a JA4 fingerprint, which needs a TLS-terminating proxy in front exporting it into an X-JA4 header. Defined now so enabling it is a log format change rather than a reindex, and an absent field costs zero bytes.
Capped at fifty. A crawler walking two hundred thousand URLs would otherwise put two hundred thousand terms into one document, which is where a document goes from large to breaking the merge. The true distinct count is carried separately, so nothing is fabricated — but the interface must label the list a sample.
Absent when it could not be computed, and the rule that reads it must not treat absent as one.
Never empty. A clean session carries a code saying nothing fired.
Discriminated by a document-type field rather than being a third index, which would triple the provisioning for a few hundred documents a year. It is recomputed from the sessions on every scorer run, never incremented: a client retries a request whose response was lost, and a retried increment double-counts, whereas recomputing with one facet query is idempotent, exact and self-healing — and it is what makes a shared pair converge rather than fork.
How big the indexes get and what to turn off first: Index size. What the configsets deliberately leave out, and how they commit: What the configsets leave out.