Choosing Workers and Batch Size for Parallel Solr Indexing
What the two numbers change, and how to pick them
Workers and batch size
Two numbers decide how a run uses your machine: --workers (default 10) and
--batch (default 200). One round holds workers × batch items, 2,000 with the defaults.
What each number changes
How many Drupal processes index at the same time. More workers index more items at once, and use more CPU and memory, and put more load on your database and Solr at the same moment.
How many items each worker loads and indexes in one go. Every worker is a new Drush process that loads its whole batch at once, indexes it and exits. Small batches spend more of the time starting processes. Large batches need more memory per worker, and each round waits longer for its slowest worker.
Finding good values
- Start with the defaults.
- Watch the items per second on the progress lines of the log (Background mode), and the machine with
topandfree -h. - Change one number at a time:
drush osstop, thendrush ostwith the new value. The run carries on where it stopped. - Stop adding workers when the rate no longer goes up, or when the machine starts to swap.
drush osstop --server=my_solr_server
drush ost --server=my_solr_server --workers=6 --batch=300Signs you went too far
- The machine swaps (
free -hshows swap in use): fewer workers or a smaller batch. - Workers fail with
Allowed memory size ... exhausted: a smaller batch. - Your site or Solr slows down for visitors: fewer workers.
Several sessions at once share the machine: count the workers of all of them together.
Reference
- Workers and batch size
- Troubleshooting
- Files and logs