Keep Your Index Fresh: Scheduled Recrawl

Keep your index fresh

Sites change: prices move, texts are rewritten, pages disappear. Keep Index Fresh reads every page already crawled again, every few days, so your search shows your site as it is today. Pages that now answer with an error, like 404 or 500, are removed from your index.

Switch it on

1. Make sure the crawl of your site runs on a schedule (Start Crawl Schedule), and let the first crawl finish.
2. In the WebCrawler tab, open the crawl settings, or click Edit next to Refresh on the summary line, and open Keep Index Fresh.
3. In Update crawled pages every, type a number of days.
4. Click Save. The summary line then shows your choice, for example Refresh: Every 7d.

Keep Index Fresh does not start a crawl. It puts the crawled pages back in the queue, and the running crawl schedule reads them on its next run. Without a schedule, nothing is read.

How often

From 7 to 365 days. Type 0 and click Save to switch it off. The bigger your index, the longer the shortest interval:

The field shows the shortest interval for your index next to it.

Right now, without waiting

• One click in the Control Panel: Reindex All.
• From your own code: Re-queue crawled URLs, read on the next run of the schedule.
• One page, the moment it changes: Recrawl from your CMS.