Web Crawler Keep Index Fresh: Scheduled Re-Crawls Every 7 to 365 Days
New: Web Crawler — Keep Index Fresh. Automatically re-crawl all previously indexed pages on a configurable schedule (every 7–365 days) to update content, prices, and detect broken links. Pages that return 404 or 500 are automatically removed from your search index. Available in the Crawler Settings panel under Index Settings.