Documentation > Web Crawler

API - Remove a URL from the Crawler Queue (HMAC signed)

Web Crawler
Web Crawler API

Remove a URL from the Crawler Queue (HMAC signed)

Deletes a URL from the crawler’s seed list and from its internal done and todo queues. Used by the CMS integrations when a page is unpublished or deleted, so the crawler forgets it (and re-crawls it from scratch if it is ever added again).

Endpoint

GET or POST https://opensolr.com/solr_manager/api/delete_crawler_url
HMAC-signed endpoint. Besides email + api_key, this call carries a signature = hex HMAC-SHA256 of the concatenated string shown in the Parameters table, keyed with your api_key. The signature proves the caller holds the key even when the request is relayed by a CMS module or a third-party host.

Parameters

ParameterStatusDescription
emailRequiredYour Opensolr registration email address
api_keyRequiredYour Opensolr API key (master key, or a scoped key that allows this endpoint)
core_nameRequiredThe index whose crawler holds the URL
urlRequiredThe exact URL to remove (URL-encode it in GET requests)
signatureRequiredHex HMAC-SHA256(url . core_name, api_key) — same scheme as add_crawl_url_signed

How It Works

After the signature and index ownership are verified, the URL is removed from the seed list and the crawler on the Solr server is told to drop it from the done and todo queues; the counts of removed queue entries come back in the response. The document that was indexed from that URL is not deleted from Solr by this call — delete it with a Solr delete-by-id/query, or let the next clean crawl rebuild the index.

Response

KeyTypeDescription
statusbooltrue when processed
msgstringURL removed from crawler queue.
done_deleted / todo_deletedintEntries removed from each queue (0 when the URL was not queued)
urlstringThe URL that was removed

Code Examples

$_ cURL

EMAIL=you@example.com; KEY=YOUR_API_KEY; CORE=my_index; URL='https://www.example.com/old-page'
SIG=$(printf '%s' "$URL$CORE" | openssl dgst -sha256 -hmac "$KEY" | awk '{print $NF}')
curl -s -G "https://opensolr.com/solr_manager/api/delete_crawler_url" \
  --data-urlencode "email=$EMAIL" --data-urlencode "api_key=$KEY" \
  --data-urlencode "core_name=$CORE" --data-urlencode "url=$URL" --data-urlencode "signature=$SIG"

PHP PHP

$sig = hash_hmac('sha256', $url . $core, $key);
$q = http_build_query(['email' => $email, 'api_key' => $key, 'core_name' => $core, 'url' => $url, 'signature' => $sig]);
$r = json_decode(file_get_contents("https://opensolr.com/solr_manager/api/delete_crawler_url?{$q}"), true);
echo $r['msg'], ' (done: ', $r['done_deleted'], ', todo: ', $r['todo_deleted'], ")\n";

Py Python

import hmac, hashlib, requests

sig = hmac.new(key.encode(), (url + core).encode(), hashlib.sha256).hexdigest()
r = requests.post("https://opensolr.com/solr_manager/api/delete_crawler_url", data={
    "email": email, "api_key": key, "core_name": core, "url": url, "signature": sig}, timeout=30)
print(r.json())

Example Response

{
  "status": true,
  "msg": "URL removed from crawler queue.",
  "done_deleted": 1,
  "todo_deleted": 0,
  "url": "https://www.example.com/old-page"
}

Use Cases

  • Drop a page from the crawl when it is unpublished in your CMS
  • Force a single page to be re-fetched on the next run (remove, then add it again)
  • Clean up seed URLs that moved to another domain

Related Documentation

Start the Web CrawlerRun the crawler after changing its URLs.Stop the Web CrawlerHalt a running crawl.Reset Index DataWipe the index before a clean re-crawl.Web Crawler GuideScope rules, exclusions and limits.

Need help with the Opensolr API? We are here to help.

Contact Support