How Parallel Solr Indexing Works in the Turbo Indexer

Rounds, chunks and worker processes

How it works

drush ost works in rounds. In each round it gives every worker its own batch, waits until all of them are done, then starts the next round.

One round: drush ost takes workers times batch items from the Search API tracker, gives one batch to each worker, the workers send them to Solr through Search API, then the next round startsSearch APItrackeritems waitingdrush osttakes workers× batch itemsWorker 1one batchWorker 2one batchWorker None batchSolrvia SearchAPIall workers done: next round
One round: drush ost takes workers times batch items from the Search API tracker, gives one batch to each worker, the workers send them to Solr through Search API, then the next round startsSearch API trackeritems waitingdrush ostworkers × batch itemsWorker 1 · one batchWorker 2 · one batchWorker N · one batchSolrvia Search APIall done: next round
One round. The next round starts only when every worker of this one has finished.

One round, step by step

  1. The main process, the one drush ost starts, takes the next items waiting in the Search API tracker for the current index: up to workers × batch items. With the defaults that is 10 × 200 = 2,000. It reads them straight from the tracker table, in item ID order.
  2. It splits them into batches and writes each batch to its own chunk file.
  3. It starts one worker per batch, 50 ms apart. Each worker is a fresh Drush process: it loads its items and hands them to Search API, which sends them to Solr and marks them as indexed. Then the worker exits.
  4. When every worker of the round has exited, it deletes the chunk files and writes one progress line: percent done, items indexed in this run, items per second and the time left.
  5. It starts the next round, until nothing is waiting in this index. Then it moves on to the next index of the server.

What this means for you

  • Workers never compete for items. Each one gets its own list before it starts.
  • Stopping is safe. An item stops waiting only once Search API has indexed it. See Stop and resume.
  • The indexes of a server go one after another, each with all the workers. To run two indexes at the same time, start one session per index: Several servers and indexes.
  • A round lasts as long as its slowest worker. How that shapes your numbers: Workers and batch size.

Getting started

Need help? Contact support or visit our FAQ.

Opensolr Turbo Indexer Documentation