Opensolr does not bill by the number of requests. It bills the bytes your index sends back. A tight query and a sloppy one cost the same to make, but not the same to answer.
One gigabyte of bandwidth is a million lean responses, or one heavy one. What you ask for decides which.
01 · What is counted
On a pre-paid plan you are billed for the outgoing bytes sent from your Opensolr index to your site or app. Nothing else. Not the number of queries, not the size of your index, not the traffic your own site receives.
Every field you return, every row you fetch and every facet you do not need is bandwidth. Return only the fields the page shows, page results instead of pulling thousands of rows, and use filters so the index does the narrowing. The best practices guide covers each of these.
02 · Why is my bandwidth high
Whoever reaches your search page, human or bot, triggers a query. The response coming back from the index is the part that counts.
Bots and crawlers visit search pages far more than people do, and each visit becomes a query against your index.
Heavy responses multiply that. A search page that returns full documents with every field makes every bot visit expensive.
A sudden overnight spike is almost always a crawler loop or a scripted attack on the search page, not real visitors.
Find the source in your logs before you tune anything. Opensolr logs every request to your index, and you can see all of it in three places below.
04 · Where to see who is using it
Logs and analytics API
Query, facet and analyze your request log by any field. Facet by IP and path and you can see exactly who is eating the bandwidth. API documentation
The request log faceted by IP address and request path.
Index control panel analytics
Traffic over time, spikes, popular queries and the pages behind them, per index, inside your control panel. Real time monitoring and analytics
Traffic and query analytics for one index.
Popular queries and where the traffic comes from.
Tail the live request log
The last 1,000 lines of the live request log, straight from the control panel. Watch traffic as it happens and spot a loop or an attack the moment it starts.
The live request log, as it comes in.
The same log, filtered down to one source.
If the numbers still do not add up, contact us with the index name and we will look at the log with you.
Solr does not run out of memory because it is busy. It runs out because of what it was asked for: a page ten thousand results deep, a facet on a field that has to be un-inverted first, a query that returns every field of a thousand documents. This page is the list of the requests and settings that cost memory, what each one costs, and what to do instead.
A Solr node uses two kinds of memory. Only the left-hand side has a hard ceiling.
01 · Where the memory actually goes
Solr runs inside a JVM with a fixed maximum heap. Everything the JVM cannot fit there ends in an OutOfMemoryError, and a Java process that hits one is not reliably recoverable — it has to be restarted. The index files themselves are not the problem: Lucene memory maps them, so the operating system pages them in and out of free RAM on its own and gives that RAM back under pressure.
Four things compete for the heap, and every one of them is something a request asked for:
Filter sets
Every fq Solr keeps is a set of matching document ids. When a filter matches a large part of the index, that set is stored as a bitset — one bit per document in the index.
Result windows
To return page N, Solr builds and ranks a priority queue of start plus rows entries. The queue is proportional to how deep the page is, not to how many rows you asked for.
Un-inverted fields
Faceting, sorting and grouping need the values of a field per document. Without docValues, Solr builds that view in the heap the first time and keeps it there.
Stored documents
The documents you return are read, decompressed and cached. Wide documents with large stored fields multiply by every row and every concurrent request.
Heap sizing itself is a separate subject: see Solr JVM tuning, RAM and memory management. On Opensolr the JVM is ours to size and we do it for you — what stays on your side is everything below.
02 · Ask for fewer rows, and fewer fields
A search result page shows ten or twenty results. A request for a thousand costs a thousand documents read from disk, decompressed, cached and serialized, on every single call. The numbers below are one query against one of our own indexes, changing nothing but rows.
Ten times the rows is ten times the payload, ten times the decompression and ten times the cache pressure.
Request
Response size
What it costs on the server
rows=10
10,766 bytes
Ten documents materialized. A normal search page.
rows=100
103,790 bytes
Still reasonable. This is the sensible ceiling for a page.
rows=1000
1,053,533 bytes
A megabyte per request, plus a documentCache entry for every document.
Keep rows under 100 for anything a user looks at, and always send an fl list. A field list is the cheapest optimisation available: a result page that shows a title, a price and a link has no reason to transport the full body text of every document.
GET /solr/mycore/select
?q=running shoes
&rows=20
&fl=id,title,price,url
Returning fields you do not display is also what you pay for in transfer: Opensolr bills the bytes your index sends back, not the number of queries. The same change fixes both problems — see Solr traffic bandwidth and how to save transfer bandwidth.
03 · Deep paging is the expensive one
Asking for start=500000&rows=10 does not skip half a million documents. Solr has to find, score and rank every one of them to know which ten come next, and it holds a queue of 500,010 entries while it does. Every shard does this independently. The cost grows with the offset, and it grows on both CPU and heap.
Measured on a 5,029,434 document index, Solr 9. Both series use the same page size, the same sort and the same field list. The cursorMark series was measured across 100 consecutive pages, 100,000 documents deep.
Offset
start plus rows
cursorMark
0
74 ms
62 ms
10,000
111 ms
62 ms
100,000
536 ms
62 ms
500,000
2,860 ms
unchanged by depth
1,000,000
5,988 ms
unchanged by depth
Two rules follow from that curve. For a user interface, keep start under 50,000 — nobody browses to page five thousand, and the requests that do are almost always crawlers walking your paginator. For exporting or re-processing an entire result set, never page with start at all: use cursorMark, which carries the position inside the sort instead of counting from the beginning.
GET /solr/mycore/select?q=category:shoes&sort=id asc&rows=1000&fl=id,title&cursorMark=*
# the response carries nextCursorMark; send it back as cursorMark for the next page,
# and stop when nextCursorMark comes back identical to the one you sent.
# the sort must end in a unique field, id asc being the usual choice.
This is not theoretical. One index on our platform received 144 identical requests for start=950500 in eight minutes, each taking between 17 and 72 seconds, each one also faceting twelve fields with facet.limit=-1. That was 3,850 CPU seconds on four cores and a load average of 38. The origin was not an attack — it was the site's own application retrying a page deep in a public catalogue after the first request timed out. Opensolr now watches for exactly this pattern and mails the index owner when it appears.
04 · docValues, and the field type that cannot have them
Faceting, sorting, grouping and function queries all need the same thing: the value of a field for a given document. The inverted index answers the opposite question — which documents contain a given value — so Solr has to get there somehow. It has two ways, and they differ by where the data ends up.
docValues is a column written at index time and read straight off disk. Without it, Solr rebuilds that column in the heap at query time.
The measurement in that diagram is real. One 5 million document index on our fleet, whose fields were declared without docValues, was holding 357.9 MB of un-inverted field data on the heap across 149 entries, the largest single entry being 161.7 MB for one field on one segment. None of that memory is reclaimable while the searcher is open. The same index answers the first facet request on such a field in 285 ms and the second in 53 ms — the 232 ms difference is the un-inverting, paid again after every commit that opens a new searcher.
Declare docValues on everything you facet, sort or group by
It moves the column out of the heap and onto disk, where the operating system caches it and can also release it. It costs index size, which is far cheaper than heap.
docValues cannot be added in place
Changing the docValues setting of an existing field requires a reindex — Solr will refuse to open an index whose segments disagree. See Cannot change DocValues type.
Do not put it on a full text field
Faceting on analyzed text is almost never what you want anyway: you would be faceting on stems and tokens, not on values.
That last point is also a hard limit, and it is the single most common mistake in a schema. A field whose type is text_general — that is, solr.TextField — cannot carry docValues="true". Solr does not warn about it and does not ignore it: the core refuses to load, with This field type does not support doc values. Copy the text into a string field instead, or use solr.SortableTextField, which is a text field that keeps a docValues copy of the original value alongside the analyzed one.
The declaration on the left does not degrade gracefully. It stops the core from starting.
<!-- WRONG: solr.TextField does not support doc values, the core will not load --><fieldname="brand"type="text_general"indexed="true"stored="true"docValues="true"/><!-- RIGHT, option 1: search the analyzed field, facet and sort on a string copy --><fieldname="brand"type="text_general"indexed="true"stored="true"/><fieldname="brand_facet"type="string"indexed="true"stored="false"docValues="true"/><copyFieldsource="brand"dest="brand_facet"/><!-- RIGHT, option 2: one field that does both, docValues capped at maxCharsForDocValues --><fieldTypename="text_sortable"class="solr.SortableTextField"positionIncrementGap="100"maxCharsForDocValues="1024"><analyzer><tokenizerclass="solr.StandardTokenizerFactory"/><filterclass="solr.LowerCaseFilterFactory"/></analyzer></fieldType><fieldname="title"type="text_sortable"indexed="true"stored="true"docValues="true"/>
Highlighting is the exception that does not want docValues. The UnifiedHighlighter, the default in Solr 9, only needs offsets, and the cheap way to give it those is storeOffsetsWithPositions="true" on the stored field. Term vectors work too, but they add a second copy of the term data to the index and cost far more disk and IO than the highlighter needs.
<!-- enough for the default highlighter, without the weight of term vectors --><fieldname="body"type="text_general"indexed="true"stored="true"storeOffsetsWithPositions="true"/>
05 · Caches: fewer, smaller, colder
Solr caches are worth having, and they are also the easiest way to lose a heap. Four of them are configurable in solrconfig.xml, and each entry they hold is a live object on the heap for as long as the searcher lives.
What each cache holds. The sizes are entry counts, not bytes — the bytes depend on your index.
The filterCache is the one that surprises people, because its entries are sized by the index and not by the filter. A filter matching a large share of the documents is kept as a bitset of one bit per document, so on a 10 million document index a single entry can be around 1.25 MB. Configure size="512" and you have authorised roughly 640 MB of heap for that cache alone. On a small index the same setting costs almost nothing. The number to think about is not the cache size, it is the cache size multiplied by your document count.
<query><!-- entry counts, not megabytes. autowarmCount=0 means a new searcher starts cold instead of replaying old queries while the old searcher is still holding its heap. --><filterCachesize="512"initialSize="512"autowarmCount="0"/><queryResultCachesize="512"initialSize="512"autowarmCount="0"/><documentCachesize="512"initialSize="512"autowarmCount="0"/><!-- how many documents of a result page are worth caching at all --><queryResultWindowSize>20</queryResultWindowSize><queryResultMaxDocsCached>200</queryResultMaxDocsCached></query>
Start small and measure, do not start large and hope
Cache hit ratios are visible per index in the Solr admin interface. A cache with a low hit ratio is pure heap cost; a cache with a high hit ratio and evictions is the one worth growing.
Autowarming is not free
While a new searcher warms, the old one is still open and both hold their caches. Aggressive autowarmCount values on an index that commits often are a classic way to run two full cache sets at once.
Cache classes changed in Solr 9
Every cache is now solr.CaffeineCache. A config carried over from Solr 8 that names solr.FastLRUCache or solr.LRUCache will not load — see ClassNotFoundException: solr.FastLRUCache.
06 · Faceting, grouping and highlighting
These three components do more work per request than the search itself, and all three scale with something other than the number of results returned.
Never send facet.limit=-1 on a high cardinality field
It asks Solr to count and return every distinct value in the field. On a field with a million distinct values that is a million counters built and a million entries serialized, for a menu nobody can read. Ask for the twenty you display.
Set facet.mincount=1
Without it, Solr returns values with zero matches, which is both larger and less useful. It is one parameter and it is almost always right.
Prefer the JSON Facet API for anything nested
It is the newer implementation, it streams better on docValues fields and it avoids the multi-pass work the old parameters do for sub-facets.
Grouping is heavier than it looks
Result grouping keeps per-group state for the whole result set. On a large collection, collapse and expand does the same job with far less memory in most cases.
Highlight the field you display, not every field
Each field in hl.fl is re-analyzed per returned document. A wildcard hl.fl on a wide schema multiplies that by every field in it.
Faceting is the most common cause of a slow Drupal search page, because the module facets on whatever the site builder enabled. Slow Solr queries debugging walks through finding which facet is responsible.
07 · Queries that fail on their own
Some requests do not degrade — they throw. These are the ones we see most often in the error logs of indexes that were working fine the day before.
What arrives
What happens
What to do instead
A filter query built from thousands of ids
Too many boolean clauses. The Solr default is 1,024; Opensolr ships 6,500, which is generous and still finite.
Use a terms query parser, a join, or a field the documents already share. See Too many boolean clauses.
A leading wildcard on a large field
Every term in the field dictionary is enumerated before anything is scored.
Index a reversed or n-gram copy of the field, or anchor the wildcard at the end.
A sort on an analyzed text field
Either an error, or a sort on the wrong thing — the first token rather than the value.
Sort on a string copy with docValues, or on solr.SortableTextField.
A document with one enormous field value
Immense term, indexing rejected for that document.
A new searcher per document, cache warming per document, merge pressure and a growing pile of open readers.
Batch your updates and let autoSoftCommit handle visibility. One commit per batch, never per document.
08 · Indexing without a memory spike
Send documents in batches, not one at a time
A few hundred to a few thousand documents per request is the working range. One request per document spends most of its time on connection and commit overhead.
Commit on a schedule, not per write
Configure autoCommit with openSearcher=false for durability and autoSoftCommit for visibility. Explicit commits from the client are what turn a bulk load into an hour of searcher churn.
Do not optimize a live index out of habit
A forced merge rewrites the entire index into one segment. It is IO, disk and page cache for the duration, and Solr handles merging by itself. Run it after a bulk load if at all, never on a schedule.
Keep stored fields to what you return
Fields that are only searched should be stored="false". It shrinks the index, the documentCache entries and the response all at once — see how to save disk space for your index.
09 · If you connect through a CMS
Most of the traffic we see is generated by a module rather than by hand, which means the fix is usually a setting in the module and not in Solr.
Keep search_api_solr and the Opensolr module current — several releases have fixed query construction bugs that produced far heavier requests than intended. Check the enabled facets and the number of items per page before blaming the index.
The plugin defaults are sane; the memory problems come from templates that request every field to render a result card. Trim the field list in the template.
Bots reach your search page too
A paginator that a crawler can walk will be walked, one deep page at a time. Rate limit the search path, or mark deep pages as not indexable.
10 · What Opensolr does when it happens anyway
Even a well-behaved index meets a bad request eventually. Every Opensolr node is watched from outside itself, so a JVM that dies does not stay dead waiting for someone to notice.
The watchdog probes each index continuously, confirms a failure with a second probe, and restarts the stack over SSH.
Automatic recovery
A failed probe is retried ten seconds later. If it still fails, the watchdog restarts Java, Jetty, Solr and the web layer on that node and logs the recovery. You usually read about it afterwards rather than during.
Query volume, response times and the slowest queries per index. This is where a deep paging loop or a runaway facet shows up as a shape rather than as an outage.
A complete, working Laravel application that puts a Vue search interface in front of an Opensolr index, using Solarium as the PHP Solr client. Clone it and read it, or use it as the starting point for your own.
The sample application: a Vue search interface over an Opensolr index.
01 · What the sample shows
Full-text search with instant results
Type-ahead querying straight against your Solr index.
Solarium inside Laravel
How to configure and use the standard PHP Solr client in a Laravel application.
A Vue front end
A reactive search interface built as Vue components, not as a page reload.
Opensolr connectivity
Connecting to a hosted index, with no server to manage on your side.
