API - Tail the Web Crawler Log

Web Crawler API

Tail the Web Crawler Log

Returns the tail of the crawler log for an index — the same text the “Crawler log” panel shows in your control panel. It is the quickest way to learn why a crawl finished early: a page or traffic limit reached, a blocked scope, fetch errors.

Endpoint

GET https://opensolr.com/solr_manager/api/tail_crawl_log

Parameters

Parameter	Status	Description
`email`	Required	Your Opensolr registration email address
`api_key`	Required	Your Opensolr API key (master key, or a scoped key that allows this endpoint)
`core_name`	Required	The index whose crawler log to read

How It Works

The request is relayed to the server that hosts the index, which returns the last lines of the crawler’s log file for that index as plain text inside the JSON msg. Lines are newline-separated; an idle crawler with a fresh log returns an empty string. For structured numbers (pages indexed, skipped, blocked) use get_crawl_stats; for a continuous live view see the streaming endpoints.

Response

Key	Type	Description
`status`	bool	`true` when the log was read
`msg`	string	Raw log tail, newline-separated

Code Examples

cURL

curl -s "https://opensolr.com/solr_manager/api/tail_crawl_log?email=YOUR_EMAIL&api_key=YOUR_API_KEY&core_name=my_index"

PHP

$q = http_build_query(['email' => 'YOUR_EMAIL', 'api_key' => 'YOUR_API_KEY', 'core_name' => 'my_index']);
$r = json_decode(file_get_contents("https://opensolr.com/solr_manager/api/tail_crawl_log?{$q}"), true);
echo $r['msg'];

Python

import requests

r = requests.get("https://opensolr.com/solr_manager/api/tail_crawl_log",
                 params={"email": "YOUR_EMAIL", "api_key": "YOUR_API_KEY", "core_name": "my_index"}, timeout=30)
print(r.json()["msg"])

Example Response

{"status": true, "msg": "Max Pages Limit Reached: 4 pages indexed of 3 pages limit\n"}

Use Cases

Surface the crawler’s last message in your CMS after a reindex
Alert when a crawl stops on a limit instead of finishing
Debug scope and robots.txt problems without opening the control panel