Per-URL Crawl Settings: HTTP Login, Do-Not-Index and Do-Not-Follow Rules

· Web Crawler · New Feature · All updates

The Web Crawler can now be told what to skip, and how to log in. Each starting URL has its own settings behind a cog in the Crawl URLs list: an optional HTTP user name and password, a list of pages that should be crawled but not indexed, and a list of links that should never be followed at all. The same two lists exist index-wide under Global Exclusions.

01 Crawling a site that asks for a password

Staging sites, intranets and members areas sit behind a basic-auth prompt, and until now the crawler simply got a 401 and moved on. Open the cog on that URL, fill in the user name and password, and the credentials go out with every request the crawler makes to that host — including the headless browser, when a page needs JavaScript to render.

Leave both fields empty for a public site. Nothing changes for sites that never asked for a login.

02 Two lists, two different jobs

Both take one regular expression per line, matched against the full URL. The difference between them is the whole point:

Do not followpage Anever fetchedthe crawler stops at the linknothing beyond it is discoveredcalendars, filters, cartsDo not indexpage Apage Bpage A is fetched and walkedbut never stored in your indexprint views, duplicates
Follow decides where the crawler goes. Index decides what ends up in your search results.
Do not index

The page is still fetched and its links are still followed — it just never reaches your index. For print views, session-tagged duplicates, or a section that only exists to link elsewhere.

Do not follow

The link is never queued at all. The crawler stops there. For infinite calendars, faceted filter URLs, search result pages, carts and checkouts.

03 Per URL, or for the whole index

  • The cog on a URL row holds settings for that starting URL and the host it belongs to. It carries an accent colour when that URL has settings of its own, so a crawl skipping half a site always has a visible cause.
  • Global Exclusions in Crawler Setup holds the same two lists for every host this index crawls — the place for rules about content rather than about one site.
  • They combine, they do not compete. A page is skipped if the global list matches it or the per-URL list does.
  • Built-in defaults still apply on top. Stylesheets, scripts, archives, images, fonts, mailto: and tel: links never need a rule of their own.
  • A broken rule is refused, not ignored. Save an expression that does not compile and the dialog names the line. It will not fail silently halfway through a crawl.

Rules match per host, so a rule written for www.example.com applies to every page the crawler reaches on that host, whichever starting URL led there.

Web Crawler documentation Crawler configuration & best practices

More in Web Crawler

View the full changelog