With the Data Crawler, Opensolr reads the pages of your site from a sitemap the extension builds for you, and indexes them in your Opensolr Index. Each page gets everything Opensolr adds to a document, such as its language, the text of its documents and, with AI, its meaning for search by meaning: What Opensolr adds to every document. Your server only serves the pages, the way it does for any visitor.
How it works
- You pick what goes into the sitemap: What gets crawled.
- The sitemap is already in the crawl list of your index: the extension adds it when you connect your account and every time a crawl starts. The extension sitemap
- You start the crawl with Reindex Everything: Start, resume and stop the crawl.
- You follow it in the Crawl Status window.
Good to know
- HTTPS: the extension adds the sitemap only when your site address starts with
https://. - Firewall: if your site has a firewall, whitelist the crawler IP shown on the tab ("If your site has a firewall, whitelist the crawler IP").
- Counters: the tab shows how many addresses your sitemap holds and how many documents of your site are in your index ("N URLs in sitemap, M documents in Solr index").
- Crawl and push together: you can also push pages the moment you save them with Data Ingestion. A page that arrives both ways is one document, because Opensolr knows each page by its address.
Pages of this tab
- What gets crawled: content types and attached files.
- Crawl speed: Parallel Threads and Request Delay.
- Start, resume and stop the crawl: Reindex Everything, Resume Indexing, Stop Crawl.
- Reindex From Scratch or Reset Index: one reads your site again, the other deletes your documents.
- Crawl Status: the live figures of the crawl.
What the crawl reads on every page, and which pages it leaves out: What a page needs to be indexed and robots.txt and meta robots.