Web Crawler

The Opensolr Web Crawler automatically visits your website, reads the content of every page, and adds it to your search index. Think of it as your own personal Google — it crawls your site so your visitors can search everything on it. For a full walkthrough, see the main crawler guide or jump straight to the getting started tutorial.

How the Web Crawler Works

How It Works

The Opensolr Web Crawler works just like Google's crawler, but for your own search index. Here is the simple version:

You give it a starting URL (for example, https://www.example.com).
The crawler visits that page, reads the content, and finds all the links on it.
It follows those links to discover more pages on your site.
For each page, it extracts the title, description, main text, images, and metadata.
All of that content gets added to your Opensolr index, making it searchable.

The result: your visitors can search your entire website using fast, relevant Opensolr-powered search. See the crawler standards reference for technical details on how the crawler respects robots.txt, sitemaps, and other web standards.

Adding URLs to Crawl

Before the crawler can do its job, you need to tell it where to start. You can add one or more URLs.

Open your Index Dashboard and click on the index you want to crawl content into.
Go to the "Web Crawler" tab in the tools section.
Enter a URL in the "Add URL" field. This is your starting point. For most sites, enter your homepage (e.g., https://www.example.com).
Click "Add" — the URL will appear in the crawl queue.

Multiple starting URLs

You can add as many URLs as you want. This is useful if your site has sections that are not linked from the homepage, or if you want to crawl multiple subdomains. For tips on choosing the right starting URLs and scope, see Crawler Configuration Best Practices.

URL Verification (Proving Ownership)

To prevent abuse, Opensolr needs to verify that you own or control the website you want to crawl. There are two ways to prove ownership:

Verification File Upload

Download a small verification file from Opensolr and upload it to the root of your website. The crawler checks for this file to confirm you have access to the server. Similar to how Google Search Console verification works.

HMAC-Signed API

If you prefer automation, you can use the Opensolr API to add URLs programmatically. The API uses HMAC signatures (a tamper-proof digital stamp) to verify that the request comes from your account. No file upload needed.

Per-URL Crawl Settings

Every row in the Crawl URLs list has a cog in the Settings column. It opens the settings that apply to that starting URL and the host it belongs to — useful when one of your sites needs a login, or when a section of it should be crawled but not indexed.

HTTP Basic Auth

A user name and password for sites behind a basic-auth prompt — staging sites, intranets, members areas. The credentials are sent with every request the crawler makes to that host, including the headless browser when the page needs JavaScript to render. Leave both empty for a public site.

Do not index

One regular expression per line. A page whose URL matches any of them is still fetched and its links are still followed, but the page itself never reaches your index. Use it for print views, session-tagged duplicates, or a section that only exists to link elsewhere.

Do not follow

One regular expression per line. A link whose URL matches is never queued at all — the crawler stops there. Use it to keep the crawl out of infinite calendars, faceted filter URLs, search result pages, cart and checkout paths, and anything that would multiply forever.

The difference in one line

Do not index keeps the page out of your search results but keeps walking through it. Do not follow stops the crawler from ever going there. A page that is followed but not indexed still contributes its links; a page that is not followed contributes nothing.

Writing the rules

Each line is a regular expression matched against the full URL. Anything you would write in a normal regex works, including delimiters and flags, and the dialog refuses to save a rule that does not compile — it tells you which line is wrong instead of failing silently mid-crawl.

Rules are matched per host, so a rule you write for www.example.com applies to every page the crawler reaches on that host, whichever starting URL led there.

Global Exclusions

The Global Exclusions button in Crawler Setup opens the same two rule lists, applied to every host this index crawls. Use it for rules that are about content rather than about one site — skipping binaries, tracking parameters, or login paths across the board.

Global rules and per-URL rules are combined, not chosen between: a page is skipped if either list matches it. On top of both, the crawler always applies its own built-in defaults, so stylesheets, scripts, archives, images, fonts, mailto: and tel: links never need a rule of their own.

The button carries an accent colour when the index has global rules, and so does a URL's cog when that URL has its own — without it, a crawl skipping half a site would have no visible cause.

Crawler Settings

The crawler has several settings that let you control exactly how it behaves. Here is what each one does:

CPU Threads (Parallel Requests)

Controls how many pages the crawler fetches at the same time. More threads = faster crawling, but also more load on your website. Start with a lower number (2-4) and increase if your server handles it well.

Crawl Mode: Domain vs. Host

Domain mode crawls all subdomains (www, blog, shop, etc.). Host mode sticks to the exact hostname you provide. Use Host mode if you only want to index one part of your site.

Content Types

HTML Only — crawls regular web pages (HTML). This is the default and covers most websites. HTML + Documents — also extracts text from PDFs, DOCX, XLSX, PPTX, and other document formats found on your site. Great if your site hosts downloadable files that should be searchable.

Renderer: Curl vs. Chrome

Use Curl for regular websites where content is in the HTML source code. Use Chrome if your site uses JavaScript frameworks (React, Vue, Angular) that load content dynamically after the page loads.

Resume vs. Start Over

Resume continues from where the crawler last stopped — it skips pages already crawled and only processes new ones. Start Over clears the crawl history and visits every page fresh, as if crawling for the first time.

Pause Between Requests (Politeness Delay)

Adds a short delay between page requests (from 0.1 seconds to several seconds). This prevents the crawler from overwhelming your web server with too many requests at once. A polite crawler is a good crawler. Increase this value if your server is under heavy load.

HTTP Authentication for Protected Sites

If your website requires a username and password to access (HTTP Basic Auth), you can enter those credentials here. The crawler will use them when visiting your pages. This is useful for crawling staging environments or password-protected areas.

Crawl Controls

Once your URLs and settings are configured, you control the crawler with these buttons:

Start Crawl

Begins crawling your website. The crawler will visit pages, extract content, and queue it for indexing.

Pause Crawl

Temporarily stops crawling. The crawler remembers where it left off so you can resume later.

Stop Crawl

Completely stops the current crawl session. Use this if you need to change settings or start over.

Flush to Solr

Pushes all crawled content from the staging area into your Opensolr index immediately, making it searchable right away.

What does "Flush to Solr" mean?

The crawler first stores extracted content in a staging area. Flushing sends that content into your actual Opensolr index. Think of it as pressing "publish" — the content goes live for search.

Automate crawler controls via API

All of these controls are also available as API endpoints: Start, Stop, Pause, and Flush to Solr. Use them to integrate crawler operations into your CI/CD pipeline or scheduling scripts.

Scheduling Recurring Crawls

You can set the crawler to run automatically on a schedule, so your index always stays up to date without manual work.

Go to the Scheduling section in the Web Crawler tab.
Choose a frequency — daily, weekly, or a custom interval.
Save your schedule — the crawler will automatically start at the scheduled time and crawl your site.

Set it and forget it

Once scheduled, the crawler runs on autopilot. New pages on your site will be discovered and indexed automatically on your chosen schedule.

Keep Your Index Fresh

Websites change over time — pages get updated, new pages are added, old pages are removed. The "Keep Index Fresh" feature automatically re-crawls your site every N days to make sure your search index reflects the current state of your website.

When enabled, the crawler periodically revisits all known pages and updates the index with any changes. Pages that no longer exist are removed from the index.

Reindex All vs. Reindex From Scratch

Reindex All

Re-visits every known URL and updates the content in your index. Does NOT delete anything first. Faster because the crawler already knows what pages exist. Use this when your content has changed and you want to refresh everything.

Reindex From Scratch

Deletes ALL existing crawl data and starts fresh, as if crawling for the very first time. Use this when you have restructured your site, changed domains, or want a completely clean slate.

Choose wisely

Reindex All is usually what you want — it is faster and does not cause any downtime in your search. Reindex From Scratch temporarily removes all content from your index while the crawler works, so your search results may be empty or incomplete until the crawl finishes.

Monitoring Your Crawl

While the crawler is running (or after it finishes), you can monitor its progress in real time:

Pages Crawled — how many pages the crawler has visited so far.
Pages Queued — how many pages are waiting to be crawled.
Errors — pages that could not be crawled (404 not found, 403 forbidden, server errors, etc.).
Crawl Speed — how many pages per second the crawler is processing.

The Crawl Log gives you a detailed breakdown of every URL the crawler visited, what happened (success or error), and the HTTP status code returned. This is invaluable for troubleshooting issues like broken links or pages that block the crawler. You can also retrieve live crawl statistics programmatically via the Get Live Crawler Stats API.

Common status codes in the crawl log

200 = Success (page crawled normally). 301/302 = Page redirected. 403 = Access forbidden. 404 = Page not found. 500 = Server error. If you see many errors, check your website's health or adjust crawler settings.

Searching What You Crawled

Once pages are in the index they are searchable from your hosted search page, from the embed widget, and through the API — and not only by typing. Visitors can also search with a photo: Opensolr works out what is in the picture and your search runs on it, so it covers everything you already crawled without adding a thing, changing a setting, or crawling again.

On a phone that button opens the camera, which is the case it was built for: someone photographs a product, a plant, an animal or a label and lands on the matching pages of your site. Labels, shelf tags and receipts are read, so a photo of a shelf tag with an article number on it lands on that exact product. It also crosses languages — an English photo label finds Romanian pages.

Search by Image on your search page → · image_to_text API endpoint →

Web Crawler - Site Crawling & Indexing