API - Remove a URL from the Crawler Queue (HMAC signed)

Web Crawler

Makes the crawl read one page again: the URL is taken out of what the crawl has already done and still has to do, so its next run finds it as new and indexes the new version.

Endpointhttps://opensolr.com/solr_manager/api/delete_crawler_url
MethodGET or POST
Authemail and api_key of your account, as GET or POST parameters: Authentication. Plus a signature: see below.

01 · Parameters

core_nameRequired

The name of your index.

urlRequired

The page, exactly as the crawl knows it.

signatureRequired

Hex HMAC-SHA256 of url followed by core_name, keyed with your API key.

02 · Example

URL="https://www.example.com/blog/new-post"
SIG=$(printf '%s' "${URL}my_index" | openssl dgst -sha256 -hmac "YOUR_API_KEY" | sed 's/^.* //')
curl -s "https://opensolr.com/solr_manager/api/delete_crawler_url" \
  --data-urlencode "email=YOUR_EMAIL" --data-urlencode "api_key=YOUR_API_KEY" \
  --data-urlencode "core_name=my_index" --data-urlencode "url=$URL" --data-urlencode "signature=$SIG"

03 · Answer

{
    "status": true,
    "msg": "URL removed from crawler queue.",
    "done_deleted": 1,
    "todo_deleted": 0,
    "url": "https://www.example.com/blog/new-post"
}

The start URLs of the index stay as they are, and so does the document in the index until the page is read again.

04 · Errors

ERROR_MISSING_PARAMSHTTP 200

core_name, url or signature is missing.

ERROR_INVALID_SIGNATUREHTTP 200

The signature does not match.

ERROR_NOT_CORE_OWNERHTTP 200

The index is not yours.

ERROR_AUTHENTICATION_FAILEDHTTP 403

The email and API key do not match.

ERROR_RATE_LIMIT_PER_MINUTEHTTP 429

Too many calls. Wait for the Retry-After seconds: Rate limits.

05 · Related

Every page again: move_done_to_todo. Add a start URL: add_crawl_url_signed.

Every error code and HTTP status of the API: API errors. Calls per minute and per hour: Rate limits.