Web Crawler Resume: Sitemap Re-Discovery, Empty Queues and Automatic Stale Lock Cleanup
Resume for the Web Crawler has been rebuilt around what it is actually for: finding what is new on your site. It now launches on an empty queue, re-reads every sitemap, and clears up after a crash.
01 Resume works with an empty queue
Clicking Resume now always launches the crawler, even when the queue appears empty. The interface used to refuse in that case — but an empty queue is exactly the scenario where Resume needs to work, because the crawler re-discovers new content by re-reading your sitemaps.
02 Every child sitemap is re-fetched
When you resume a finished crawl, the crawler re-fetches all XML sitemaps — not only the top-level sitemap index, but every child sitemap under it, for example sitemap-products1.xml through sitemap-products22.xml. Any new URLs found there are queued and crawled automatically, so your index stays up to date as your site adds pages, without a full re-crawl.
03 Stale lock files cleaned up automatically
If a previous crawl crashed or was interrupted, leftover lock files could silently stop the next run: the crawler would launch and then do nothing. Resume now detects and removes stale lock files before starting, so scheduled cron runs and manual resumes both work reliably.
Empty queue refused to resume; only the top-level sitemap was re-read; a stale lock file silently killed the next run.
Resume always launches, re-reads every child sitemap, and clears stale locks first.