How far the crawl goes
The crawl mode decides which links the crawl follows from your start URL: every host of your domain, only the exact host, or only one folder. Each one also has a shallow version that stops one level down. The default, Follow HOST Links, is right for most sites.
Start URL https://www.example.com/blog/. Each mode crawls what is inside its box. A shallow mode keeps the same box and goes one level of links deep.
The six modes
Every host of your domain: www.example.com, shop.example.com, help.example.com.
The default. Only the host of the start URL: www.example.com, never shop.example.com. www.example.com and example.com count as the same host.
Only the addresses under the folder of the start URL, on the same host: www.example.com/blog/ and below, never www.example.com/about.
The scope of Follow DOMAIN Links, one level deep: the start page and the pages it links to, and no further.
The scope of Follow HOST Links, one level deep.
The scope of Follow PATH Links, one level deep.
A sitemap is always followed to the end, in every mode, shallow ones too: start from your sitemap and every page it lists inside the mode is crawled.
Change it
- Open the crawl settings and pick a Crawl Mode. The i next to it sums up the six modes.
- Click Save Settings.
Pages the mode leaves out are counted in Crawl Stats under Out of Scope.
Crawl your site: all pages
- Crawl your site overview
- Add start URLs
- Prove you own the site
- Crawl settings
- How far the crawl goes
- Sites built with JavaScript
- Documents on your site
- Rules per site
- Pages left out everywhere
- Meta robots, nofollow, canonical
- Start, pause, stop, flush
- Crawl Stats
- Reindex
- Keep your index fresh
- Recrawl from your CMS
- What a page needs
- URLs and duplicates