Start, Pause, Resume, Stop and Flush a Crawl

The buttons that run the crawl, and the schedule

Run the crawl

The Crawler Setup panel of the WebCrawler tab starts and stops the crawl of your site. A crawl schedule keeps it going on its own; Pause, Resume and Flush to Solr cover the moments in between.

The status

The Status line says what is happening now: Running, Idle (a schedule is on and nothing runs at this moment), Stopped (no schedule, nothing runs), Starting... or Stopping.... The round arrow next to it checks again.

The buttons

Start Crawl Schedule

Starts the crawl now and puts it on a schedule. At regular intervals the schedule calls the crawl again with Resume, so a crawl that stopped picks up where it left off, and pages queued later, by Keep Index Fresh or your CMS, are read. Shown while no schedule is on.

Stop Crawl Schedule

Stops the crawl and removes the schedule. Shown while a schedule is on.

Pause

Shown while the crawl runs. Stops it now and keeps the schedule, so the schedule starts it again, with Resume, on its next run. To stop for good, use Stop Crawl Schedule.

Resume

Shown while nothing runs. Carries on from where the crawl stopped. Without a schedule it runs once and creates none.

Stop Crawl

Shown only while a crawl without a schedule runs, like one started by the Reindex buttons. Stops it.

Flush to Solr

Crawled pages reach your index in batches. This sends the pages waiting in the buffer right away, even while the crawl runs, and shows the progress.

The same panel holds the Reindex buttons and Global Exclusions.

When a crawl does not start

No active URLs found.

Every start URL is Inactive: tick one in Crawl URLs and click Activate.

ERROR_PLEASE_VERIFY_YOUR_URLS

No active start URL is verified yet: Prove you own the site.

The Solr index contains ...

Your index already holds more documents than your plan lets the crawl fill. Remove documents, or move to a larger plan on https://opensolr.com/pricing.

ERROR_THIS_CRAWLER_IS_NOW_STOPPING_PLEASE_WAIT

A stop is still in progress. Try again in a moment.

Every button here is also an API call, for scripts and other systems: Control the crawl by API.

Crawl your site: all pages

Back to Enterprise Site Search