Web Crawler Overview

Crawler Configuration & Best Practices

Once your Opensolr Web Crawler index is created and you have entered a starting URL, there are several things worth knowing about how the crawler operates and how to get the best results.

Mandatory Configuration Set (Required)

Before you start crawling, your Solr index must use the Opensolr Web Crawler configuration set. This config set defines all the dynamic fields, field types, and analyzers that the crawler needs to properly index your content — including vector embeddings, sentiment fields, phonetic matching, autocomplete edge ngrams, and all meta_* dynamic fields.

Without this configuration set, crawling will fail or produce incomplete results.

Download mandatory_web_crawler_config_set_solr_9.zip

How to Apply

1. New indexes created from the Opensolr dashboard — The config set is applied automatically when you create a new Web Crawler index. No manual step needed.
2. Existing indexes — If you have an existing Solr 9.x index and want to enable the Web Crawler on it, upload this config set from the Index Settings panel → Upload Config section. This will overwrite your current schema and solrconfig.xml, so make sure to back up any custom configurations first.
3. Via the Drupal module — The Opensolr Search Drupal module applies this config set automatically during index setup.

Note: This config set is specifically for Solr 9.x. If you are running an older Solr version, contact support@opensolr.com for the correct version.

Content Types the Crawler Handles

The crawler does not just handle plain HTML pages. It can extract text and metadata from:

• HTML pages — The primary content type. Extracts title, description, body text, meta tags, OG tags, author, and more.
• PDF documents — Full text extraction from PDF files linked on your site.
• DOCX / ODT — Microsoft Word and OpenDocument text files.
• XLSX — Excel spreadsheets (extracts cell data as text).
• Images — Extracts EXIF metadata, GPS coordinates, and alt text.

If your site links to downloadable documents, the crawler will follow those links and index the document contents just like it indexes HTML pages.

Renderer Setting: Curl vs Chrome

The crawler offers two rendering modes, selectable from the Renderer dropdown in the Web Crawler settings (or via the renderer API parameter):

Curl (Fast) — Default

Uses a fast HTTP client (curl) to fetch pages. No browser is launched. This is the right choice for the vast majority of websites — server-rendered HTML, static sites, WordPress, Drupal, and any site that returns its content in the initial HTTP response. Pages are fetched in ~0.2 seconds each.

Chrome (JS Rendering)

Every page is fetched with curl first, then rendered through a headless Chromium browser (Playwright). Use this mode for JavaScript-heavy single-page applications built with React, Vue, Angular, Next.js, Gatsby, Nuxt, or similar frameworks — sites where the raw HTML returned by the server is empty or contains only a loading spinner, and the actual content is rendered client-side by JavaScript.

Chrome mode adds ~0.5–1 second per page. The browser automatically blocks ads, analytics, fonts, and social media embeds for faster rendering.

Tips for JS-Heavy Sites

• Make sure your site does not block headless browsers in its server configuration
• If you use client-side routing, ensure that direct URL access (not just SPA navigation) works for each page
• If you are unsure whether your site needs Chrome mode, try Curl first — if the crawled content looks complete, stick with Curl

robots.txt

The crawler fully respects your robots.txt file. Before crawling any page, it checks the robots.txt rules for that domain.

If you want to allow the Opensolr crawler while blocking other bots, you can add specific rules:

User-agent: *
Disallow: /admin/
Disallow: /private/

# Allow Opensolr crawler everywhere
User-agent: OpensolrBot
Allow: /

Common Pitfalls

• If your robots.txt blocks / entirely, the crawler cannot crawl anything
• Some CMS platforms (WordPress, Drupal) have robots.txt rules that may block crawlers from CSS/JS files — this can prevent proper rendering of JS-heavy pages
• Check your robots.txt at https://yoursite.com/robots.txt before starting the crawler

URL Validation & Quality Checks

The crawler is strict about URL quality, similar to how Google's crawler operates:

• Only valid HTTP/HTTPS URLs are followed
• Fragment-only links (#section) are ignored (they point to the same page)
• Duplicate URLs are detected and skipped (normalized by removing trailing slashes, sorting query parameters, etc.)
• Redirect chains are followed (301, 302) up to a reasonable limit
• Broken links (404, 500, etc.) are logged but not indexed
• External links (pointing to a different domain) are not followed by default — the crawler stays within the domain(s) you specify
• Content deduplication — if two different URLs serve the exact same content (detected via MD5 hash), only one copy is indexed

Recrawling & Scheduling

After the initial crawl completes, your pages are indexed and searchable. But websites change — new pages are added, existing pages are updated, some pages are removed.

You can recrawl your site at any time from the Web Crawler section of your Index Control Panel. The crawler will:

• Discover and index new pages
• Update content on pages that changed since the last crawl
• Handle pages that no longer exist

You can also schedule automatic recrawls to keep your index fresh without manual intervention.

Keep Index Fresh — In the Index Settings modal under your Web Crawler panel, the Keep Index Fresh section lets you set an automatic refresh interval (every 7–365 days). When triggered, all previously crawled pages are re-queued for re-crawling. The normal crawl schedule picks them up and re-fetches each page — updating content, prices, and metadata in your search index. Pages that now return 404 or 500 errors are automatically removed from your Solr index, keeping your search results clean and accurate. The minimum interval scales with your index size (e.g. 14 days for 10K+ pages, 30 days for 50K+) to protect server resources.

Crawl Depth & Coverage

The crawler starts from the URL you provide and follows links recursively. The crawl depth determines how many link-hops away from the starting URL the crawler will go:

• Depth 1 — Only the starting page
• Depth 2 — The starting page plus all pages linked from it
• Depth 3 — Two hops away from the starting page
• And so on...

For most websites, the default depth setting covers the entire site. If your site has a deep link structure (many clicks from the homepage to reach some pages), you may need to increase the crawl depth or provide additional starting URLs.

Tips for Better Coverage

• Provide a sitemap URL as the starting point if your site has one (https://yoursite.com/sitemap.xml) — the crawler can parse XML sitemaps and discover all listed URLs directly
• If some pages are not being discovered, it usually means they are not linked from anywhere that the crawler can reach — add internal links or provide the URL directly
• Pages behind login/authentication walls cannot be crawled

Crawl Exclusions and Password-Protected Sites

Two places control what the crawler touches. Both hold the same two rule lists, at different scopes.

Per starting URL — the cog

Every row in Crawl URLs has a cog in the Settings column. It opens the settings that apply to that starting URL and the host it belongs to:

• HTTP Basic Auth — a user name and password for sites behind a basic-auth prompt: staging sites, intranets, members areas. The credentials go out with every request the crawler makes to that host, including the headless browser when a page needs JavaScript to render. Leave both empty for a public site.
• Do not index — one regular expression per line. A matching page is still fetched and its links are still followed, but the page itself never reaches your index.
• Do not follow — one regular expression per line. A matching link is never queued at all. The crawler stops there.

The cog carries an accent colour when that URL has settings of its own, so a crawl skipping half a site always has a visible cause.

Index-wide — Global Exclusions

The Global Exclusions button in Crawler Setup opens the same two lists, applied to every host this index crawls. Use it for rules that are about content rather than about one site: binaries, tracking parameters, login paths.

Which one wins

Neither — they are combined. A page is skipped if the global list matches it or the per-URL list does. On top of both, the crawler always applies its own built-in defaults, so stylesheets, scripts, archives, images, fonts, mailto: and tel: links never need a rule of their own.

Writing the rules

Each line is a regular expression matched against the full URL, delimiters and flags included. A rule that does not compile is refused when you save, with the offending line named — rather than failing silently in the middle of a crawl.

#/print/#i
#[?&]sessionid=#i
#/cart|/checkout|/wp-login#i
#\.(zip|exe|dmg|iso)$#i

Rules match per host, so a rule written for www.example.com applies to every page the crawler reaches on that host, whichever starting URL led there.

What Gets Indexed

For every successfully crawled page, the following data is extracted and indexed:

Query Elevation — Pin & Exclude Search Results

Take full control of what your users see. Query Elevation lets you pin important results to the top or exclude irrelevant ones — directly from the Search UI, with zero code and no reindexing required.

• Pin — Force a specific result to the top of the list for a given search query
• Exclude — Hide a result completely so it never appears for that query
• Pin All / Exclude All — Apply the rule globally, across every search query
• Drag & drop — Reorder pinned results to control exactly which one shows first

Enable it from your index settings panel, then open the Search UI — every result gets an elevation toolbar. Perfect for promoting landing pages, burying outdated content, or curating high-value queries.