Errors

Understanding and Fixing Slow Drupal Solr Queries

Your Drupal site is slow when searching. You check the Solr debug output and discover that the query itself runs in 200-300ms — but faceting takes 1,500-2,000ms or more. The bottleneck is not Solr's search — it is all the extra work Drupal asks Solr to do.

Why Drupal Queries Are Slow

1. The *:* Query Pattern

Drupal often sends {!boost b=boost_document} *:* — meaning "match every single document." Even with filter queries narrowing results, Solr still has to evaluate the entire index first.

2. Too Many Facets

Drupal's Facet API requests facet counts for every active facet field. Each facet must count every term across every matching document. With 10+ facets and facet.limit=-1 (return all values), this is the main performance killer.

3. FCS Method (Slow)

Solr's default FieldCache Scanning (FCS) method scans the entire field value space. Modern alternatives like JSON facets or streaming facets are much faster.

4. Excessive Filters

Dozens of fq parameters — especially negations like (*:* -field:value) and range queries — each add cost.

Fixes in Drupal

A. Reduce Active Facets

Go to Configuration > Search API > View Modes and disable facets that are not visible on the page. Every active facet triggers Solr work, even if Drupal does not render it.

B. Limit Facet Results

In Facet API settings, change facet.limit from -1 (return everything) to a reasonable number:

facet.limit = 50
facet.mincount = 1

C. Use the JSON Facet API

If your Solr version is 7.x or newer, switch Drupal's Search API backend to use the JSON Facet API. It is faster, parallelized, and supports streaming facet counts.

D. Enable Caching

Enable Drupal's Search API query cache and facet cache to avoid recomputing identical facets for repeated queries.

Fixes in Solr

A. Enable DocValues on Facet Fields

Ensure all fields used for faceting have docValues="true" in your schema:

<field name="field_category" type="string" stored="true"
       indexed="true" docValues="true"/>

DocValues make faceting dramatically faster.

B. Use facet.threads

Parallelize facet computation:

facet.threads=4

C. Enable Filter Cache

In solrconfig.xml, ensure you have a properly sized filter cache:

<filterCache class="solr.FastLRUCache" size="512"
             initialSize="512" autowarmCount="128"/>

D. Switch to JSON Facets

Replace classic facet.field parameters with JSON facets:

json.facet={
  categories: { type: "terms", field: "field_category", limit: 50 },
  formats: { type: "terms", field: "field_format", limit: 50 }
}

Optimization Checklist

Before vs After

Before (slow):

q={!boost b=boost_document} *:*&facet.limit=-1&facet.field=field_a&facet.field=field_b...

After (fast):

q=*:*&fq=available:true&facet.limit=50&facet.threads=4&
json.facet={
  categories:{type:terms,field:field_category,limit:50},
  formats:{type:terms,field:field_format,limit:50}
}

Deep Pagination (start=100000 and beyond)

Another silent killer, and a favorite of bots and scrapers: requesting very deep result pages.

Solr cannot jump straight to result number 100,000. To serve start=100000&rows=20, it must collect and sort all 100,020 top documents first, on every single request. The cost grows with the offset, so page 5,000 costs thousands of times more than page 1. A scraper walking your entire index page by page will happily push start= into the hundreds of thousands, turning each request into a multi-second, CPU-burning query.

How to fix it

1. Cap the page depth in your application. Real visitors never browse past page 100. Refuse or clamp anything deeper:

if (page > 100) page = 100;

2. Use cursorMark for exports and crawls. If you legitimately need to walk the whole index (data export, sitemap, migration), use Solr's cursor API. It has constant cost per page, no matter how deep you go:

q=*:*&sort=id asc&rows=100&cursorMark=*

Each response returns a nextCursorMark value. Pass it as cursorMark in the next request. Requirements: a stable sort that includes your unique key field, and no start parameter.

3. Keep rows sane. Large rows values multiply the cost of every request. Fetch what you display.

How you find out

The Opensolr watchdog monitors your request analytics daily and emails you when it detects hard queries or deep pagination patterns on your index, with the offending IPs and offsets included.

The Problem

Your Drupal site shows this warning:

Solr was unable to retrieve additional data about the server.

And in your logs you see cURL error 28 — a timeout while Drupal tries to reach /admin/luke on your Solr server.

What Is Actually Happening

Drupal's search_api_solr module periodically calls Solr's /admin/luke endpoint to discover what fields exist in the index, check capabilities, and verify the server is healthy. The Luke handler is like asking Solr to give you a complete X-ray of the entire index — every field, every term count, everything.

On large indexes, this response can be megabytes of data and take several seconds to generate. If it takes longer than Drupal's PHP timeout, the request fails.

The Fix

Replace Solr's heavy Luke handler with a lightweight Ping handler at the same URL. Drupal only needs a "yes, I am alive" response — it does not actually need the full Luke data dump.

Add this to your solrconfig.xml:

<requestHandler name="/admin/luke" class="solr.PingRequestHandler">
  <lst name="defaults">
    <str name="q">*:*</str>
  </lst>
</requestHandler>

Step-by-Step for Opensolr Users

1. Log in to your Opensolr Dashboard
2. Open your Index Settings for the affected Opensolr Index
3. Go to Config Files Editor and edit solrconfig.xml
4. Find any existing /admin/luke request handler block and replace it with the snippet above
5. Save your changes — Opensolr reloads the config automatically

Verify It Works

curl -sS -u user:pass -w '%{http_code}
' \
  "https://your-cluster.opensolr.com/solr/your_core/admin/luke?wt=json"

You should get an instant 200 response with a small JSON payload.

Alternative: Skip Schema Check in Drupal

If you cannot modify Solr configs, you can tell Drupal to stop calling Luke entirely:

// settings.php
$settings['search_api_solr']['skip_schema_check'] = TRUE;
$settings['search_api_solr']['timeout'] = 3;
$settings['search_api_solr']['connect_timeout'] = 1;

This disables schema introspection. Search and indexing continue to work normally.

Trade-offs

Quick Checklist

• Replace /admin/luke handler with PingRequestHandler in solrconfig.xml
• Save and let Opensolr reload the config
• Verify with a quick cURL call that /admin/luke responds instantly
• Optionally clear Drupal caches with drush cr
• If you cannot edit Solr configs, use skip_schema_check in Drupal settings.php

The Error

You reload your Opensolr Index or try to index documents and Solr throws:

cannot change field "field_name" from index options=DOCS_AND_FREQS_AND_POSITIONS
to inconsistent index options=DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS

This means the field was originally indexed with one set of "index options," and now your schema (or application) is trying to use a different set. Solr will not allow mixing two different formats in the same index.

What Are Index Options? (The Simple Version)

When Solr indexes a word in a field, it can store different levels of detail about that word:

Why This Happens

Schema Change Without Re-indexing

You (or a module like Drupal Search API) changed the field definition — for example, adding storeOffsetsWithPositions="true" for faster highlighting. The new definition requires OFFSETS, but old documents were indexed without them. Solr cannot mix both.

Drupal Search API Solr

This is especially common with Drupal. The Search API Solr module may set storeOffsetsWithPositions="true" on certain fields automatically when you enable highlighting features. The module generates the schema, and if the generated schema differs from what was previously indexed, you get this error.

How to Fix It

Step 1: Decide Which Index Options You Want

• DOCS_AND_FREQS_AND_POSITIONS — standard, good for most use cases
• DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS — needed for fast highlighting (term vectors with offsets)

Step 2: Update Your Schema

Make the field definition consistent. If you want offsets for highlighting:

<field name="your_field" type="text_general" indexed="true"
       stored="true" storeOffsetsWithPositions="true"/>

If you do not need offsets, make sure storeOffsetsWithPositions is false or absent.

Step 3: Reset and Re-Index

After changing the schema, you must clear and rebuild the entire index:

1. Reset your index from the Opensolr control panel (remove all data)
2. Reload the Opensolr Index
3. Re-index all documents from your application

There is no shortcut — old documents with the old format must be replaced.

Quick Reference

Quick Checklist

• Check your schema for storeOffsetsWithPositions on the affected field
• Make the field definition consistent — old and new must match
• Reset your index and re-index all data after the schema change
• If using Drupal, check the Search API Solr module highlighting settings
• Always reload the Opensolr Index after schema changes