Web Crawler

Opensolr Web Crawler — find answers to your questions

API - Get LIVE Web Crawler Stats

Gives you the numbers of the crawl of your Opensolr Index right now: what was indexed, what failed, what was skipped and what is still in the queue.

Endpointhttps://opensolr.com/solr_manager/api/get_crawl_stats
MethodGET
Authemail and api_key of your account, as GET or POST parameters: Authentication.

01 · Parameters

core_nameRequired

The name of your index.

02 · Example

curl -s "https://opensolr.com/solr_manager/api/get_crawl_stats?email=YOUR_EMAIL&api_key=YOUR_API_KEY&core_name=my_index"

03 · Answer

total_ok_pages

Pages read and indexed.

total_pages_in_error, total_4xx, total_5xx

Pages that answered an error.

total_noindex, total_blocked_scope, total_redirect_outside, total_oversize_pages

Pages skipped: marked noindex, outside the crawl mode, redirected away, too large.

remain_todo

URLs still in the queue.

solr_buffer_docs

Pages read and waiting to be sent to the index.

total_downloaded_mb

Data downloaded.

mime_types

Indexed pages per file type.

not_found, error, noindex_pages and the other lists

The first 100 URLs of each kind (sample_limit).

Before the first crawl, msg only says No crawl yet.

04 · Errors

No such indexHTTP 200

The index is not yours.

ERROR_AUTHENTICATION_FAILEDHTTP 403

The email and API key do not match.

ERROR_RATE_LIMIT_PER_MINUTEHTTP 429

Too many calls. Wait for the Retry-After seconds: Rate limits.

05 · Related

The last lines of the crawl log: tail_crawl_log.

Every error code and HTTP status of the API: API errors. Calls per minute and per hour: Rate limits.

Read Full Answer

API - Start the Web Crawler

Starts the crawl of your site into your Opensolr Index, from the start URLs of the index. The same call resumes a crawl that stopped, or starts it over.

Endpointhttps://opensolr.com/solr_manager/api/start_crawl
MethodGET or POST
Authemail and api_key of your account, as GET or POST parameters: Authentication.

01 · Parameters

core_nameRequired

The name of your index. It needs at least one active and verified start URL: Add start URLs.

cleanOptional

no (default) carries on from where the crawl stopped: this is Resume. yes forgets what was crawled and starts again from the start URLs; the documents in the index stay until they are replaced.

modeOptional

How far the crawl goes, 1 to 6, default 1: Crawl modes. crawl_mode works too.

follow_docsOptional

yes also reads the documents the pages link to (PDF, Word, Excel...). Default no: web pages only.

rendererOptional

curl (default) or chrome for sites built with JavaScript: JavaScript sites.

scheduleOptional

yes (default) puts the crawl on a schedule that keeps it going, when there is none yet. no crawls once.

requeueOptional

all puts every crawled URL back in the queue first, or a list of document types such as pdf,docx only those. Not together with clean=yes.

max_threadsOptional

Pages read at the same time. Default and ceiling: your plan.

relaxOptional

Pause between two requests, in microseconds. Anything below 10000 (0.01 s) becomes 10000.

max_pagesOptional

Most pages to index. Default and ceiling: your plan.

max_trafficOptional

Most data to download, in MB. Default and ceiling: your plan.

max_filesizeOptional

Largest page or document read, in bytes. Default and ceiling: your plan.

auth_username, auth_passwordOptional

HTTP Basic Auth of the start URL, more than two characters each. Per site: Rules per site.

02 · Examples

# Start (or resume) the crawl and keep it on a schedule
curl -s "https://opensolr.com/solr_manager/api/start_crawl?email=YOUR_EMAIL&api_key=YOUR_API_KEY&core_name=my_index&mode=2&follow_docs=yes"

# Crawl everything again, once, without a schedule
curl -s "https://opensolr.com/solr_manager/api/start_crawl?email=YOUR_EMAIL&api_key=YOUR_API_KEY&core_name=my_index&clean=yes&schedule=no"

03 · Answer

{
    "status": true,
    "msg": "Web Crawler Started."
}

With requeue, the answer also carries requeued, the number of URLs put back in the queue.

04 · Errors

ERROR_PLEASE_VERIFY_YOUR_URLSHTTP 200

No active start URL is verified: Prove you own the site.

ERROR_CORE_NOT_OWNER_OR_NO_URLS_ERRORHTTP 200

The index is not yours, or has no start URL.

ERROR_THIS_CRAWLER_IS_NOW_STOPPING_PLEASE_WAITHTTP 200

A stop is still on its way. Try again in a moment.

ERROR: The Solr index contains ...HTTP 200

The index already holds more pages than your plan lets the crawl fill.

ERROR_INVALID_REQUEUEHTTP 200

requeue is not all or a list of types.

ERROR_REQUEUE_WITH_CLEANHTTP 200

requeue was sent with clean=yes.

ERROR_AUTHENTICATION_FAILEDHTTP 403

The email and API key do not match.

ERROR_RATE_LIMIT_PER_MINUTEHTTP 429

Too many calls. Wait for the Retry-After seconds: Rate limits.

05 · Related

Pause: pause_crawl. Stop and remove the schedule: stop_crawl. Is it running: crawler_active.

Progress: get_crawl_stats. The pages waiting for the index: flush_crawl_buffer.

Every error code and HTTP status of the API: API errors. Calls per minute and per hour: Rate limits.

Read Full Answer

API - Stop the Web Crawler

Stops the crawl of your Opensolr Index and removes its schedule, so it does not start again on its own.

Endpointhttps://opensolr.com/solr_manager/api/stop_crawl
MethodGET or POST
Authemail and api_key of your account, as GET or POST parameters: Authentication.

01 · Parameters

core_nameRequired

The name of your index.

02 · Example

curl -s "https://opensolr.com/solr_manager/api/stop_crawl?email=YOUR_EMAIL&api_key=YOUR_API_KEY&core_name=my_index"

03 · Answer

{
    "status": true,
    "msg": "CRAWLER IS READY / NOT RUNNING."
}

The schedule is removed in the background, a few moments after the answer. Pages already in the index stay.

04 · Errors

ERROR_THIS_CRAWLER_IS_NOW_STARTING_PLEASE_WAITHTTP 200

A start is still on its way. Try again in a moment.

NOT_OWNER_ERRORHTTP 200

The index is not yours.

ERROR_AUTHENTICATION_FAILEDHTTP 403

The email and API key do not match.

ERROR_RATE_LIMIT_PER_MINUTEHTTP 429

Too many calls. Wait for the Retry-After seconds: Rate limits.

05 · Related

Stop until the next scheduled run only: pause_crawl. Start again: start_crawl. Is there a schedule: crawl_schedule_status.

Every error code and HTTP status of the API: API errors. Calls per minute and per hour: Rate limits.

Read Full Answer

Loading more articles...