Project structure

What every folder is, how a log line travels through the code, and where to go to change anything.

This page is for a developer who has just cloned the repository. It explains what every folder is, how a log line and a beacon hit travel through the code, how the panel is put together, and which files to open for the most common changes. It stays high level on purpose; the linked pages and SPEC.md go deeper.

Want to work on it with us?

Loghound is open source and contributions are welcome. To become a contributor to this or any other Opensolr open source project, write to support@opensolr.com and tell us what you would like to work on.

01 · The big picture in one minute

Loghound is plain PHP 8.1+ with no Composer packages, no npm and no build step. It is really four programs sharing one library in src/:

The ingest daemon

bin/loghound-tail runs all the time, reads your access logs as they grow, and writes one document per request into the hits index.

The scorer

bin/loghound-score runs every minute, closes finished visits, merges beacon data, scores each visit and writes it into the sessions index.

The beacon

public/b.js runs in visitors' browsers and posts to public/collect.php, which only stages the data locally.

The panel

public/index.php serves the web interface; every card on a page loads its own data from the two indexes.

Everything Loghound learns lives in two Opensolr indexes it provisions for you; small working state (file positions, open visits, staged beacon rows) lives in a local SQLite file.

02 · The repository, top to bottom
bin/

The command-line programs: the daemon, the scorer, retention, the schema check and the setup wizard.

src/

All PHP classes, namespace Loghound\, loaded by src/autoload.php.

public/

The web root: the panel, the beacon collector, the beacon script, CSS, JavaScript and vendored assets.

solr/

The configuration of the two indexes, hits/conf and sessions/conf.

config/

loghound.example.php, the documented template. The live loghound.php is written by setup and never committed.

install/

install.sh, uninstall.sh, the systemd units and timers, web server and PHP-FPM examples.

tests/

A dependency-free test runner, the tests, anonymised log fixtures and the secret scanner.

tools/

panel-preview.php, for looking at the panel with demo data while developing.

var/

Runtime state, never committed: the SQLite database, status files, setup jobs.

docs/

Operator and developer documentation in Markdown: install, beacon, detection, panel, schema, privacy, security, attacks.

SPEC.md

The binding technical specification. Field names, paths and signatures in it are contracts.

README.md, CONTRIBUTING.md, CHANGELOG.md, LICENSE

The front page, the rules for contributions, the release history (Keep a Changelog, semantic versions), and the MIT licence.

03 · The programs in bin/
loghound-tail

Tail, parse, enrich, group into visits, and write batches into the hits index.

When it runs
Always, as loghound-tail.service
loghound-score

Close idle visits, merge beacon data, apply the scoring rules, write visits into the sessions index, rebuild daily rollups.

When it runs
Every 60 seconds, by a timer
loghound-retention

Delete what is older than the retention period, keep the indexes within the plan, purge old local state.

When it runs
Daily at 03:20, by a timer
loghound-schema

Compare the live schemas with this release; --apply uploads them.

When it runs
By hand, after an upgrade
loghound-setup

The interactive setup wizard; it shares its logic with the browser installer.

When it runs
By hand, once

Every command and option is on the command reference; the services and timers on running it.

04 · The library in src/

Top-level classes

Configuration and state

Load, validate and save the configuration (every default is in Config::defaults()); the local SQLite store for file positions, open visits, caches, staged beacon rows and rate limits.

Classes
Config, State
Reading logs

Follow files across rotation; compile an Apache or nginx log format; find the logs and their format; turn a line into a normalised request.

Classes
Tail, LogFormat, LogDetect, Parser
Visits and rules

Group requests into visits; skip traffic that is never recorded; the operator's own attack patterns; where a visit really came from.

Classes
Sessionizer, Exclusions, AttackPatterns, RefererOrigin
Solr and Opensolr

The only way code talks to Solr, with its query safety checks; the Opensolr account API; the Opensolr request log; query shapes; plan quota and size-based retention.

Classes
Solr, Opensolr, OpensolrLog, OpensolrShape, Quota
Beacon

Validates beacon data and merges it into visits.

Classes
Beacon
Web support

Escaping, safe paths, CSRF and authentication; the panel's read cache; asset versions; safe CSV export; the failure report.

Classes
Security, Cache, Assets, Csv, Diagnostics

Subfolders

  • src/Enrich/ — adds context to a request: Geo (country and timezone), Asn (network, netname, reverse DNS), Ua (browser and declared bots).
  • src/Score/ — detection: Signals extracts the evidence, Rules holds the weighted rules and the verdict, Attacks spots hostile requests.
  • src/Panel/ — the web interface: one controller class per view (Overview, Bots, Fingerprints, Sessions, Seo...), plus Layout (routes, navigation, page chrome), Gateway (the panel's only route to Solr), Query, Facets, Sorting, Paging and the long-running Jobs.
  • src/Setup/ — installing: the browser Installer, detection and requirement checks, index provisioning and schema sync, resumable jobs, reset and teardown.
  • src/Auth/ — panel sign-in extras: two-factor codes, the QR code for the authenticator, “stay signed in”.
  • src/Live/ — the live view, reading logs as they are written without touching ingest.
  • src/Geo/Countries.php and src/Beacon/Doc.php — the one country table, and the one description of the beacon shown wherever it is documented.
05 · How data moves through the code

A log line, into the hits index

  1. Tail reads the new line and remembers its position in State.
  2. LogFormat and Parser turn it into a request with named fields; lines that do not parse are sampled to a file.
  3. Enrich\Geo, Enrich\Asn and Score\Attacks add context; Exclusions and AttackPatterns apply the operator's rules.
  4. Sessionizer attaches the request to a visit (same network and browser, 30 minutes of inactivity closes it).
  5. Solr::addDocs() writes the batch into the hits index, after Quota confirms there is room.

A visit, into the sessions index

  1. loghound-score finds visits that went idle.
  2. Beacon::mergeIntoSession() adds what the browser reported.
  3. Score\Signals gathers the evidence and Score\Rules computes the score, the verdict and its reasons.
  4. The visit document, and the daily rollups, are written into the sessions index.

A beacon hit

  1. b.js in the visitor's browser posts to collect.php.
  2. The collector checks the signed token, size, origin and allowed hosts, rate-limits, stages the row in SQLite, and always answers 204. It never touches Solr.
  3. The next scorer run merges the staged row into its visit.

A panel page

  1. public/index.php sets security headers, loads the configuration, sends first-time visitors to the installer, and checks sign-in and CSRF.
  2. The view named in ?v= is looked up in Layout::routes(); its controller draws the page frame without querying Solr.
  3. Each card then asks the server for its own data (?api=); the controller answers through Gateway, Cache and Solr, so one slow card never blocks the others.
  4. On the browser side, public/assets/js/app.js loads the matching module from public/assets/js/views/, which fills its cards with loadCard() from core.js.
06 · Configuration, state and secrets
  • The configuration is a PHP array in config/loghound.php, mode 0640, written by Config::save(). Its sections are documented in config/loghound.example.php and on settings.
  • Secrets — the Opensolr API key, the beacon secret, the IP salt, the panel password hash and the two-factor secret — live in that file; sign-in state lives in files under var/ with mode 0600. None of them is ever committed.
  • Working state is var/state.db (SQLite) and a few status files, also under var/.
07 · The two indexes

solr/hits/conf holds one document per request; solr/sessions/conf one per visit plus daily rollups. Every field is explained on the Solr schema. The schema rules are strict: fields are not stored unless needed, string fields carry no norms, there is no catch-all field, and an undeclared field is silently ignored.

  1. Change solr/hits/conf/schema.xml or solr/sessions/conf/schema.xml.
  2. Run bin/loghound-schema --check to see what differs from the live index.
  3. Run bin/loghound-schema --apply to upload it. Operators do the same after upgrading.
08 · Where to go to change something
Add or change a scoring rule

src/Score/Signals.php (the evidence) and src/Score/Rules.php (weights, reasons); raise Rules::RULE_VERSION when weights change; tests in tests/test_scoring.php for the rule firing and not firing; docs/DETECTION.md.

Add a panel view

A controller in src/Panel/, registered in Layout::routes() and Layout::nav(); a module in public/assets/js/views/ imported by app.js; docs/PANEL.md.

Store a new field

The schema in solr/*/conf/schema.xml; fill it in Parser.php (hits) or bin/loghound-score (sessions); allow it in src/Panel/Query.php; SPEC.md and docs/SCHEMA.md. Renaming or removing a field is discussed first.

Change the beacon

public/b.js, public/collect.php, src/Beacon.php, src/Beacon/Doc.php; docs/BEACON.md. A test fails if the documented options fall behind the script.

Support a new log format

src/LogDetect.php, plus a fixture in tests/fixtures/.

Add a configuration option

Config::defaults() (and validate()), config/loghound.example.php, the settings page in src/Panel/Settings.php, and its documentation.

Add a command or a scheduled job

A script in bin/ following the existing ones, the bin list in composer.json, and for a schedule a unit and timer in install/ plus install.sh.

Change colours or layout of the panel

public/assets/css/panel.css, the only file where colours are defined.

09 · Tests, previews and releases
php tests/run.php # the whole suite, no network needed php tests/run.php tail # only tests whose name matches php tests/scan-secrets.php # make sure no secret is about to be committed php tools/panel-preview.php # a demo config, then: php -S 127.0.0.1:8099 -t public
  • A test file is tests/test_<name>.php returning named closures; the runner finds it by itself.
  • Versions are the headings in CHANGELOG.md and the matching git tags.
10 · The rules every contribution follows

They are written in CONTRIBUTING.md; in short:

  • No dependencies: no Composer packages, no npm, no build step, no CDN.
  • Security first: escape at output, user text to Solr only as bound parameters, allowlisted field names, safe paths, CSRF on every change, no secrets in code.
  • Honesty: an unknown number is absent, never zero; every verdict carries its reasons; limitations are documented.
  • Style: PSR-12, four spaces, single quotes, strict types, and the comment rules in CONTRIBUTING.md.
  • One change per pull request, with its documentation and a changelog entry.

Become a contributor

Read SPEC.md and CONTRIBUTING.md in the repository, then write to support@opensolr.com to ask for contributor access to Loghound or any other Opensolr open source project. Issues and pull requests on GitHub are open to everyone.

Loghound is open source and MIT licensed. Questions about the Opensolr half — the account, the indexes, the plan — go to opensolr.com/contact; questions about the software itself belong on GitHub.

Loghound Documentation