Solr does not run out of memory because it is busy. It runs out because of what it was asked for: a page ten thousand results deep, a facet on a field that has to be un-inverted first, a query that returns every field of a thousand documents. This page is the list of the requests and settings that cost memory, what each one costs, and what to do instead.
A Solr node uses two kinds of memory. Only the left-hand side has a hard ceiling.
The guide, one page per topic
The JVM heap and the memory-mapped index: which one has a hard ceiling.
Every returned document is read, decompressed, cached and sent.
The cost of a page grows with its offset. cursorMark keeps it flat.
docValues and the TextField trap
Facet, sort and group from disk instead of the heap, and the type that cannot have them.
filterCache, queryResultCache and documentCache, sized against your document count.
Faceting, grouping and highlighting
What each one costs and how to ask for less.
Queries that fail on their own
Too many clauses, leading wildcards, regular expressions and other queries that throw.
Indexing without a memory spike
Batch sizes, commits and the searcher that every commit opens.
Never put binary data or base64 into Solr
Pictures, files and any base64 blob have no place in a search index.
Inline base64 images in indexed HTML
Pictures pasted into a CMS editor get indexed inside the page HTML.
N-gram fields belong on short text only
Partial matching that multiplies the index when used on long text.
Suggesters that rebuild on every commit
Autocomplete that reads the whole index on every commit.
Field names generated per site or environment, paid for in every segment.
When one document keeps crashing Solr
One oversized document that fails every indexing run.
CMS integrations that commit every second
Per-second commits from CMS modules and what they cost.
If you connect through a CMS
Most of the traffic we see is generated by a module rather than by hand, which means the fix is usually a setting in the module and not in Solr.
Keep search_api_solr and the Opensolr module current — several releases have fixed query construction bugs that produced far heavier requests than intended. Check the enabled facets and the number of items per page before blaming the index.
The plugin defaults are sane; the memory problems come from templates that request every field to render a result card. Trim the field list in the template.
Bots reach your search page too
A paginator that a crawler can walk will be walked, one deep page at a time. Rate limit the search path, or mark deep pages as not indexable.
What Opensolr does when it happens anyway
Automatic recovery
Every server is checked from outside itself. A failed check is retried ten seconds later, and if it still fails, the services on that server are restarted and the recovery is logged.
Query volume, response times and the slowest queries per index: a deep paging loop or a runaway facet shows up as a shape, not as an outage.
Every Solr error of your index, parsed and searchable, so an exception at three in the morning is still there when you look for it.
Unusual traffic alerts
A daily check of the request log emails the owner and the team when an index sees deep paging, a sudden traffic spike, one address flooding it or a wave of rejected logins, with the request that triggered it.
The short version
| Do | Because |
|---|---|
Keep rows under 100 and always send fl | Every returned document is read, decompressed, cached and serialized. |
Keep start under 50,000, and use cursorMark to export | The cost of a page grows with its offset: 74 ms at the start became 5,988 ms a million results deep (deep paging). |
Put docValues="true" on every field you facet, sort or group by | Otherwise the column is rebuilt in the heap and stays there. |
Never put docValues on a solr.TextField | The core will not load. Use a string copy or solr.SortableTextField. |
| Size caches against your document count | One filterCache entry can be one bit per document in the index. |
| Ask for the facet values you display | facet.limit=-1 counts and returns every distinct value in the field. |
| Batch your indexing and commit on a schedule | A commit per document is a searcher per document. |
| Store only what you return | Stored fields are index size, cache size and bandwidth at the same time. |
Read your own numbers first
The slow queries, the deep pages and the heavy responses of your index are in its analytics. Start there, then pick the page above that matches what you see.