Web Crawler Keep Index Fresh: Scheduled Re-Crawls and Automatic Removal of Dead Pages
The Web Crawler gains Keep Index Fresh: it automatically re-crawls all previously indexed pages on a configurable schedule, so your index tracks your site instead of drifting away from it.
01 Scheduled re-crawls
Set a schedule of anything between every 7 and every 365 days. On each run the crawler revisits pages it has already indexed to update content, refresh prices and detect broken links.
You will find it in the Crawler Settings panel, under Index Settings.
02 Dead pages leave the index
The crawler now automatically removes documents from the search index when their pages return a non-200 status code during crawling — 404, 500 and the rest. Previously, dead pages could remain in search results indefinitely.