API - Start the Web Crawler

Web Crawler

Starts the crawl of your site into your Opensolr Index, from the start URLs of the index. The same call resumes a crawl that stopped, or starts it over.

Endpointhttps://opensolr.com/solr_manager/api/start_crawl
MethodGET or POST
Authemail and api_key of your account, as GET or POST parameters: Authentication.

01 · Parameters

core_nameRequired

The name of your index. It needs at least one active and verified start URL: Add start URLs.

cleanOptional

no (default) carries on from where the crawl stopped: this is Resume. yes forgets what was crawled and starts again from the start URLs; the documents in the index stay until they are replaced.

modeOptional

How far the crawl goes, 1 to 6, default 1: Crawl modes. crawl_mode works too.

follow_docsOptional

yes also reads the documents the pages link to (PDF, Word, Excel...). Default no: web pages only.

rendererOptional

curl (default) or chrome for sites built with JavaScript: JavaScript sites.

scheduleOptional

yes (default) puts the crawl on a schedule that keeps it going, when there is none yet. no crawls once.

requeueOptional

all puts every crawled URL back in the queue first, or a list of document types such as pdf,docx only those. Not together with clean=yes.

max_threadsOptional

Pages read at the same time. Default and ceiling: your plan.

relaxOptional

Pause between two requests, in microseconds. Anything below 10000 (0.01 s) becomes 10000.

max_pagesOptional

Most pages to index. Default and ceiling: your plan.

max_trafficOptional

Most data to download, in MB. Default and ceiling: your plan.

max_filesizeOptional

Largest page or document read, in bytes. Default and ceiling: your plan.

auth_username, auth_passwordOptional

HTTP Basic Auth of the start URL, more than two characters each. Per site: Rules per site.

02 · Examples

# Start (or resume) the crawl and keep it on a schedule
curl -s "https://opensolr.com/solr_manager/api/start_crawl?email=YOUR_EMAIL&api_key=YOUR_API_KEY&core_name=my_index&mode=2&follow_docs=yes"

# Crawl everything again, once, without a schedule
curl -s "https://opensolr.com/solr_manager/api/start_crawl?email=YOUR_EMAIL&api_key=YOUR_API_KEY&core_name=my_index&clean=yes&schedule=no"

03 · Answer

{
    "status": true,
    "msg": "Web Crawler Started."
}

With requeue, the answer also carries requeued, the number of URLs put back in the queue.

04 · Errors

ERROR_PLEASE_VERIFY_YOUR_URLSHTTP 200

No active start URL is verified: Prove you own the site.

ERROR_CORE_NOT_OWNER_OR_NO_URLS_ERRORHTTP 200

The index is not yours, or has no start URL.

ERROR_THIS_CRAWLER_IS_NOW_STOPPING_PLEASE_WAITHTTP 200

A stop is still on its way. Try again in a moment.

ERROR: The Solr index contains ...HTTP 200

The index already holds more pages than your plan lets the crawl fill.

ERROR_INVALID_REQUEUEHTTP 200

requeue is not all or a list of types.

ERROR_REQUEUE_WITH_CLEANHTTP 200

requeue was sent with clean=yes.

ERROR_AUTHENTICATION_FAILEDHTTP 403

The email and API key do not match.

ERROR_RATE_LIMIT_PER_MINUTEHTTP 429

Too many calls. Wait for the Retry-After seconds: Rate limits.

05 · Related

Pause: pause_crawl. Stop and remove the schedule: stop_crawl. Is it running: crawler_active.

Progress: get_crawl_stats. The pages waiting for the index: flush_crawl_buffer.

Every error code and HTTP status of the API: API errors. Calls per minute and per hour: Rate limits.