Two settings decide how fast the Data Crawler reads your site and how much load it puts on your server. They are under Crawler Settings on the Data Crawler tab, closed by default: click the title to open them.
Parallel Threads
How many pages are fetched at the same time, from 1 to 20 (3 when the plugin is installed). More threads crawl faster. Your plan sets a maximum, shown as Max Threads in the Account Summary of the Settings tab, and the field never goes above it. With many threads, a firewall in front of your site may need the crawler IP whitelisted: the address is shown on the Data Crawler tab.
Request Delay (seconds)
How long Opensolr waits between two requests to your site, from 0.1 to 10 seconds, in steps of 0.1 (2 seconds when the plugin is installed). Lower is faster. Raise it if your server or host limits how many requests it accepts.
Click Save Configuration. The values are used every time a crawl starts.
Set by the plugin
- The crawl starts from the plugin sitemap and stays on the host name of your site.
- Document links are followed only when Include attached files is ticked: What gets crawled.
- Pages are read as your server sends them, without running JavaScript.
The same settings for any site, in the Opensolr Control Panel: Speed and politeness.