Documentation > Web Crawler

API - Add a Crawl URL (HMAC signed)

Web Crawler
Web Crawler API

Add a Crawl URL (HMAC signed)

Adds a seed URL to the Web Crawler of an index and marks it verified in one call. Because the request is signed with your API key, Opensolr does not need the usual site-ownership verification step — this is how the Drupal module and the WordPress plugin register the site they run on.

Endpoint

GET or POST https://opensolr.com/solr_manager/api/add_crawl_url_signed
HMAC-signed endpoint. Besides email + api_key, this call carries a signature = hex HMAC-SHA256 of the concatenated string shown in the Parameters table, keyed with your api_key. The signature proves the caller holds the key even when the request is relayed by a CMS module or a third-party host.

Parameters

ParameterStatusDescription
emailRequiredYour Opensolr registration email address
api_keyRequiredYour Opensolr API key (master key, or a scoped key that allows this endpoint)
core_nameRequiredThe index whose crawler receives the URL
urlRequiredAbsolute URL to crawl from (URL-encode it in GET requests)
signatureRequiredHex HMAC-SHA256(url . core_name, api_key) — the URL immediately followed by the index name

How It Works

The signature is recomputed server-side from url + core_name with your api_key and compared in constant time; a mismatch is rejected before anything is stored. On success the URL is inserted into the index’s seed list as active + verified, or re-activated if it was already there. The crawl itself does not start — call start_crawl next.

Response

KeyTypeDescription
statusbooltrue when the URL is stored
msgstringURL_ADDED_AND_VERIFIED for a new URL, URL_ALREADY_EXISTS_ACTIVATED when it was already registered
on error{"status":false,"msg":"Invalid signature"}, missing parameters, or an ownership error for the index

Code Examples

$_ cURL

EMAIL=you@example.com; KEY=YOUR_API_KEY; CORE=my_index; URL='https://www.example.com/'
SIG=$(printf '%s' "$URL$CORE" | openssl dgst -sha256 -hmac "$KEY" | awk '{print $NF}')
curl -s -G "https://opensolr.com/solr_manager/api/add_crawl_url_signed" \
  --data-urlencode "email=$EMAIL" --data-urlencode "api_key=$KEY" \
  --data-urlencode "core_name=$CORE" --data-urlencode "url=$URL" --data-urlencode "signature=$SIG"

PHP PHP

$email = 'you@example.com'; $key = 'YOUR_API_KEY'; $core = 'my_index'; $url = 'https://www.example.com/';
$sig = hash_hmac('sha256', $url . $core, $key);
$q = http_build_query(['email' => $email, 'api_key' => $key, 'core_name' => $core, 'url' => $url, 'signature' => $sig]);
echo file_get_contents("https://opensolr.com/solr_manager/api/add_crawl_url_signed?{$q}");

Py Python

import hmac, hashlib, requests

email, key, core, url = "you@example.com", "YOUR_API_KEY", "my_index", "https://www.example.com/"
sig = hmac.new(key.encode(), (url + core).encode(), hashlib.sha256).hexdigest()
r = requests.post("https://opensolr.com/solr_manager/api/add_crawl_url_signed", data={
    "email": email, "api_key": key, "core_name": core, "url": url, "signature": sig}, timeout=30)
print(r.json())

Example Response

{"status": true, "msg": "URL_ADDED_AND_VERIFIED"}

Use Cases

  • Register the site a CMS plugin runs on, without asking the site owner to place a verification file
  • Provision search for many client sites from one deployment pipeline
  • Add extra seed URLs (sub-sites, sitemaps) to an existing crawler

Related Documentation

Start the Web CrawlerKick off a crawl with your limits and schedule.Live Crawler StatsPages indexed, queued, skipped and why.Web Crawler GuideEverything the crawler does and how to tune it.Crawler Running?Poll the crawler state.

Need help with the Opensolr API? We are here to help.

Contact Support