Solr runs out of memory because of what it was asked for. Knowing where each request lands in memory is the first step to asking for less.
Solr runs inside a JVM with a fixed maximum heap. Everything the JVM cannot fit there ends in an OutOfMemoryError, and a Java process that hits one is not reliably recoverable — it has to be restarted. The index files themselves are not the problem: Lucene memory maps them, so the operating system pages them in and out of free RAM on its own and gives that RAM back under pressure.
Four things compete for the heap, and every one of them is something a request asked for:
Filter sets
Every fq Solr keeps is a set of matching document ids. When a filter matches a large part of the index, that set is stored as a bitset — one bit per document in the index.
Result windows
To return page N, Solr builds and ranks a priority queue of start plus rows entries. The queue is proportional to how deep the page is, not to how many rows you asked for.
Un-inverted fields
Faceting, sorting and grouping need the values of a field per document. Without docValues, Solr builds that view in the heap the first time and keeps it there.
Stored documents
The documents you return are read, decompressed and cached. Wide documents with large stored fields multiply by every row and every concurrent request.
Solr best practices
Where Solr memory goes (this page)