Solr Best Practices

Account Resources

Solr does not run out of memory because it is busy. It runs out because of what it was asked for: a page ten thousand results deep, a facet on a field that has to be un-inverted first, a query that returns every field of a thousand documents. This page is the list of the requests and settings that cost memory, what each one costs, and what to do instead.

ON THE JVM HEAPOFF HEAP, MEMORY MAPPEDFilter cache bitsetsQuery result windowsUn-inverted field dataStored documents in cacheAnalyzers, merges, threadsInverted indexpostings and termsdocValues columnsfaceting, sorting, groupingStored field datathe documents you returnBounded by the heap size.This is the one that runs out.Bounded by free RAM.The operating system manages it.

A Solr node uses two kinds of memory. Only the left-hand side has a hard ceiling.

01 · Where the memory actually goes

Solr runs inside a JVM with a fixed maximum heap. Everything the JVM cannot fit there ends in an OutOfMemoryError, and a Java process that hits one is not reliably recoverable — it has to be restarted. The index files themselves are not the problem: Lucene memory maps them, so the operating system pages them in and out of free RAM on its own and gives that RAM back under pressure.

Four things compete for the heap, and every one of them is something a request asked for:

Filter sets

Every fq Solr keeps is a set of matching document ids. When a filter matches a large part of the index, that set is stored as a bitset — one bit per document in the index.

Result windows

To return page N, Solr builds and ranks a priority queue of start plus rows entries. The queue is proportional to how deep the page is, not to how many rows you asked for.

Un-inverted fields

Faceting, sorting and grouping need the values of a field per document. Without docValues, Solr builds that view in the heap the first time and keeps it there.

Stored documents

The documents you return are read, decompressed and cached. Wide documents with large stored fields multiply by every row and every concurrent request.

Heap sizing itself is a separate subject: see Solr JVM tuning, RAM and memory management. On Opensolr the JVM is ours to size and we do it for you — what stays on your side is everything below.

02 · Ask for fewer rows, and fewer fields

A search result page shows ten or twenty results. A request for a thousand costs a thousand documents read from disk, decompressed, cached and serialized, on every single call. The numbers below are one query against one of our own indexes, changing nothing but rows.

RESPONSE BYTES FOR ONE QUERYrows=10rows=100rows=100010.7 KB103.8 KB1,053.5 KBSame query, same matches, same index. Only the rows parameter changed.

Ten times the rows is ten times the payload, ten times the decompression and ten times the cache pressure.

RequestResponse sizeWhat it costs on the server
rows=1010,766 bytesTen documents materialized. A normal search page.
rows=100103,790 bytesStill reasonable. This is the sensible ceiling for a page.
rows=10001,053,533 bytesA megabyte per request, plus a documentCache entry for every document.

Keep rows under 100 for anything a user looks at, and always send an fl list. A field list is the cheapest optimisation available: a result page that shows a title, a price and a link has no reason to transport the full body text of every document.

GET /solr/mycore/select
  ?q=running shoes
  &rows=20
  &fl=id,title,price,url
Returning fields you do not display is also what you pay for in transfer: Opensolr bills the bytes your index sends back, not the number of queries. The same change fixes both problems — see Solr traffic bandwidth and how to save transfer bandwidth.

03 · Deep paging is the expensive one

Asking for start=500000&rows=10 does not skip half a million documents. Solr has to find, score and rank every one of them to know which ten come next, and it holds a queue of 500,010 entries while it does. Every shard does this independently. The cost grows with the offset, and it grows on both CPU and heap.

QTIME IN MILLISECONDS02000400060005362,8605,98862 ms at every depthstart plus rowscursorMark, same page size010,000100,000500,0001,000,000start offset, rows=1000, sorted by id

Measured on a 5,029,434 document index, Solr 9. Both series use the same page size, the same sort and the same field list. The cursorMark series was measured across 100 consecutive pages, 100,000 documents deep.

Offsetstart plus rowscursorMark
074 ms62 ms
10,000111 ms62 ms
100,000536 ms62 ms
500,0002,860 msunchanged by depth
1,000,0005,988 msunchanged by depth

Two rules follow from that curve. For a user interface, keep start under 50,000 — nobody browses to page five thousand, and the requests that do are almost always crawlers walking your paginator. For exporting or re-processing an entire result set, never page with start at all: use cursorMark, which carries the position inside the sort instead of counting from the beginning.

GET /solr/mycore/select?q=category:shoes&sort=id asc&rows=1000&fl=id,title&cursorMark=*

# the response carries nextCursorMark; send it back as cursorMark for the next page,
# and stop when nextCursorMark comes back identical to the one you sent.
# the sort must end in a unique field, id asc being the usual choice.
This is not theoretical. One index on our platform received 144 identical requests for start=950500 in eight minutes, each taking between 17 and 72 seconds, each one also faceting twelve fields with facet.limit=-1. That was 3,850 CPU seconds on four cores and a load average of 38. The origin was not an attack — it was the site's own application retrying a page deep in a public catalogue after the first request timed out. Opensolr now watches for exactly this pattern and mails the index owner when it appears.

04 · docValues, and the field type that cannot have them

Faceting, sorting, grouping and function queries all need the same thing: the value of a field for a given document. The inverted index answers the opposite question — which documents contain a given value — so Solr has to get there somehow. It has two ways, and they differ by where the data ends up.

Facet, sort orgroup on a fieldField has docValuescolumn stored on diskMemory mapped,nothing on the heapField has noneSolr un-inverts itThe whole columnsits on the heapMeasured on one 5 million document index: 357.9 MB of un-inverted field data, 149 entries.

docValues is a column written at index time and read straight off disk. Without it, Solr rebuilds that column in the heap at query time.

The measurement in that diagram is real. One 5 million document index on our fleet, whose fields were declared without docValues, was holding 357.9 MB of un-inverted field data on the heap across 149 entries, the largest single entry being 161.7 MB for one field on one segment. None of that memory is reclaimable while the searcher is open. The same index answers the first facet request on such a field in 285 ms and the second in 53 ms — the 232 ms difference is the un-inverting, paid again after every commit that opens a new searcher.

Declare docValues on everything you facet, sort or group by

It moves the column out of the heap and onto disk, where the operating system caches it and can also release it. It costs index size, which is far cheaper than heap.

docValues cannot be added in place

Changing the docValues setting of an existing field requires a reindex — Solr will refuse to open an index whose segments disagree. See Cannot change DocValues type.

Do not put it on a full text field

Faceting on analyzed text is almost never what you want anyway: you would be faceting on stems and tokens, not on values.

That last point is also a hard limit, and it is the single most common mistake in a schema. A field whose type is text_general — that is, solr.TextField — cannot carry docValues="true". Solr does not warn about it and does not ignore it: the core refuses to load, with This field type does not support doc values. Copy the text into a string field instead, or use solr.SortableTextField, which is a text field that keeps a docValues copy of the original value alongside the analyzed one.

FIELD DECLARATIONtype="text_general"docValues="true"The core will not loadThis field type does not support doc valuesTHE FIXcopyField the text intoa string fielddocValues works thereand so do facets, sorting and grouping

The declaration on the left does not degrade gracefully. It stops the core from starting.

<!-- WRONG: solr.TextField does not support doc values, the core will not load -->
<field name="brand" type="text_general" indexed="true" stored="true" docValues="true"/>

<!-- RIGHT, option 1: search the analyzed field, facet and sort on a string copy -->
<field     name="brand"       type="text_general" indexed="true"  stored="true"/>
<field     name="brand_facet" type="string"       indexed="true"  stored="false" docValues="true"/>
<copyField source="brand"     dest="brand_facet"/>

<!-- RIGHT, option 2: one field that does both, docValues capped at maxCharsForDocValues -->
<fieldType name="text_sortable" class="solr.SortableTextField" positionIncrementGap="100"
           maxCharsForDocValues="1024">
  <analyzer>
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
</fieldType>
<field name="title" type="text_sortable" indexed="true" stored="true" docValues="true"/>
Highlighting is the exception that does not want docValues. The UnifiedHighlighter, the default in Solr 9, only needs offsets, and the cheap way to give it those is storeOffsetsWithPositions="true" on the stored field. Term vectors work too, but they add a second copy of the term data to the index and cost far more disk and IO than the highlighter needs.
<!-- enough for the default highlighter, without the weight of term vectors -->
<field name="body" type="text_general" indexed="true" stored="true"
       storeOffsetsWithPositions="true"/>

05 · Caches: fewer, smaller, colder

Solr caches are worth having, and they are also the easiest way to lose a heap. Four of them are configurable in solrconfig.xml, and each entry they hold is a live object on the heap for as long as the searcher lives.

THE FOUR CONFIGURABLE CACHESfilterCacheDocument ids matching one filter query.A wide filter is one bit per document.queryResultCacheOrdered id lists per query, sort and page.Every deep page adds another entry.documentCacheStored fields of the documents returned.Large stored fields, large entries.fieldValueCacheUn-inverted multi valued fields.docValues make it unnecessary.In Solr 9 all of them are solr.CaffeineCache. FastLRUCache and LRUCache were removed.

What each cache holds. The sizes are entry counts, not bytes — the bytes depend on your index.

The filterCache is the one that surprises people, because its entries are sized by the index and not by the filter. A filter matching a large share of the documents is kept as a bitset of one bit per document, so on a 10 million document index a single entry can be around 1.25 MB. Configure size="512" and you have authorised roughly 640 MB of heap for that cache alone. On a small index the same setting costs almost nothing. The number to think about is not the cache size, it is the cache size multiplied by your document count.

<query>
  <!-- entry counts, not megabytes. autowarmCount=0 means a new searcher starts cold
       instead of replaying old queries while the old searcher is still holding its heap. -->
  <filterCache      size="512" initialSize="512" autowarmCount="0"/>
  <queryResultCache size="512" initialSize="512" autowarmCount="0"/>
  <documentCache    size="512" initialSize="512" autowarmCount="0"/>

  <!-- how many documents of a result page are worth caching at all -->
  <queryResultWindowSize>20</queryResultWindowSize>
  <queryResultMaxDocsCached>200</queryResultMaxDocsCached>
</query>

Start small and measure, do not start large and hope

Cache hit ratios are visible per index in the Solr admin interface. A cache with a low hit ratio is pure heap cost; a cache with a high hit ratio and evictions is the one worth growing.

Autowarming is not free

While a new searcher warms, the old one is still open and both hold their caches. Aggressive autowarmCount values on an index that commits often are a classic way to run two full cache sets at once.

Cache classes changed in Solr 9

Every cache is now solr.CaffeineCache. A config carried over from Solr 8 that names solr.FastLRUCache or solr.LRUCache will not load — see ClassNotFoundException: solr.FastLRUCache.

06 · Faceting, grouping and highlighting

These three components do more work per request than the search itself, and all three scale with something other than the number of results returned.

Never send facet.limit=-1 on a high cardinality field

It asks Solr to count and return every distinct value in the field. On a field with a million distinct values that is a million counters built and a million entries serialized, for a menu nobody can read. Ask for the twenty you display.

Set facet.mincount=1

Without it, Solr returns values with zero matches, which is both larger and less useful. It is one parameter and it is almost always right.

Prefer the JSON Facet API for anything nested

It is the newer implementation, it streams better on docValues fields and it avoids the multi-pass work the old parameters do for sub-facets.

Grouping is heavier than it looks

Result grouping keeps per-group state for the whole result set. On a large collection, collapse and expand does the same job with far less memory in most cases.

Highlight the field you display, not every field

Each field in hl.fl is re-analyzed per returned document. A wildcard hl.fl on a wide schema multiplies that by every field in it.

Faceting is the most common cause of a slow Drupal search page, because the module facets on whatever the site builder enabled. Slow Solr queries debugging walks through finding which facet is responsible.

07 · Queries that fail on their own

Some requests do not degrade — they throw. These are the ones we see most often in the error logs of indexes that were working fine the day before.

What arrivesWhat happensWhat to do instead
A filter query built from thousands of idsToo many boolean clauses. The Solr default is 1,024; Opensolr ships 6,500, which is generous and still finite.Use a terms query parser, a join, or a field the documents already share. See Too many boolean clauses.
A leading wildcard on a large fieldEvery term in the field dictionary is enumerated before anything is scored.Index a reversed or n-gram copy of the field, or anchor the wildcard at the end.
A sort on an analyzed text fieldEither an error, or a sort on the wrong thing — the first token rather than the value.Sort on a string copy with docValues, or on solr.SortableTextField.
A document with one enormous field valueImmense term, indexing rejected for that document.Truncate or split before indexing. See Document contains immense term.
A commit after every single documentA new searcher per document, cache warming per document, merge pressure and a growing pile of open readers.Batch your updates and let autoSoftCommit handle visibility. One commit per batch, never per document.

08 · Indexing without a memory spike

Send documents in batches, not one at a time

A few hundred to a few thousand documents per request is the working range. One request per document spends most of its time on connection and commit overhead.

Commit on a schedule, not per write

Configure autoCommit with openSearcher=false for durability and autoSoftCommit for visibility. Explicit commits from the client are what turn a bulk load into an hour of searcher churn.

Do not optimize a live index out of habit

A forced merge rewrites the entire index into one segment. It is IO, disk and page cache for the duration, and Solr handles merging by itself. Run it after a bulk load if at all, never on a schedule.

Keep stored fields to what you return

Fields that are only searched should be stored="false". It shrinks the index, the documentCache entries and the response all at once — see how to save disk space for your index.

09 · If you connect through a CMS

Most of the traffic we see is generated by a module rather than by hand, which means the fix is usually a setting in the module and not in Solr.

Drupal

Keep search_api_solr and the Opensolr module current — several releases have fixed query construction bugs that produced far heavier requests than intended. Check the enabled facets and the number of items per page before blaming the index.

WordPress

The plugin defaults are sane; the memory problems come from templates that request every field to render a result card. Trim the field list in the template.

Bots reach your search page too

A paginator that a crawler can walk will be walked, one deep page at a time. Rate limit the search path, or mark deep pages as not indexable.

10 · What Opensolr does when it happens anyway

Even a well-behaved index meets a bad request eventually. Every Opensolr node is watched from outside itself, so a JVM that dies does not stay dead waiting for someone to notice.

WHEN A NODE DIES ANYWAYProbeevery index, alwaysRetry10 seconds laterRestartJava, Jetty, SolrRecoverylogged and timedtypically back up inside a minute

The watchdog probes each index continuously, confirms a failure with a second probe, and restarts the stack over SSH.

Automatic recovery

A failed probe is retried ten seconds later. If it still fails, the watchdog restarts Java, Jetty, Solr and the web layer on that node and logs the recovery. You usually read about it afterwards rather than during.

Real time monitoring and analytics

Query volume, response times and the slowest queries per index. This is where a deep paging loop or a runaway facet shows up as a shape rather than as an outage.

Error audit

Seven days of searchable error history for your index, so an exception at three in the morning is still there when you look for it.

Unusual traffic alerts

Hard queries, deep paging and request floods are detected against your own baseline and mailed to you, with the exact request that triggered them.

Search tuning

Field weights, minimum match and results per page from the control panel, without editing a config file or reloading the core.

11 · The short version

DoBecause
Keep rows under 100 and always send flEvery returned document is read, decompressed, cached and serialized.
Keep start under 50,000, and use cursorMark to exportThe cost of a page grows with its offset. 74 ms became 5,988 ms in the measurement above.
Put docValues="true" on every field you facet, sort or group byOtherwise the column is rebuilt in the heap and stays there.
Never put docValues on a solr.TextFieldThe core will not load. Use a string copy or solr.SortableTextField.
Size caches against your document countOne filterCache entry can be one bit per document in the index.
Ask for the facet values you displayfacet.limit=-1 counts and returns every distinct value in the field.
Batch your indexing and commit on a scheduleA commit per document is a searcher per document.
Store only what you returnStored fields are index size, cache size and bandwidth at the same time.

Not sure which of these applies to your index?

Send us the index name. We can read the request log, the slow queries and the schema, and tell you which change is worth making first.

Ask us to review your indexJVM tuning guide