API - Get LIVE Web Crawler Stats

Web Crawler

Gives you the numbers of the crawl of your Opensolr Index right now: what was indexed, what failed, what was skipped and what is still in the queue.

Endpointhttps://opensolr.com/solr_manager/api/get_crawl_stats
MethodGET
Authemail and api_key of your account, as GET or POST parameters: Authentication.

01 · Parameters

core_nameRequired

The name of your index.

02 · Example

curl -s "https://opensolr.com/solr_manager/api/get_crawl_stats?email=YOUR_EMAIL&api_key=YOUR_API_KEY&core_name=my_index"

03 · Answer

total_ok_pages

Pages read and indexed.

total_pages_in_error, total_4xx, total_5xx

Pages that answered an error.

total_noindex, total_blocked_scope, total_redirect_outside, total_oversize_pages

Pages skipped: marked noindex, outside the crawl mode, redirected away, too large.

remain_todo

URLs still in the queue.

solr_buffer_docs

Pages read and waiting to be sent to the index.

total_downloaded_mb

Data downloaded.

mime_types

Indexed pages per file type.

not_found, error, noindex_pages and the other lists

The first 100 URLs of each kind (sample_limit).

Before the first crawl, msg only says No crawl yet.

04 · Errors

No such indexHTTP 200

The index is not yours.

ERROR_AUTHENTICATION_FAILEDHTTP 403

The email and API key do not match.

ERROR_RATE_LIMIT_PER_MINUTEHTTP 429

Too many calls. Wait for the Retry-After seconds: Rate limits.

05 · Related

The last lines of the crawl log: tail_crawl_log.

Every error code and HTTP status of the API: API errors. Calls per minute and per hour: Rate limits.