The Data Crawler reads the pages listed in the extension sitemap. You decide what goes into that sitemap: the boxes under Content Types for Crawling, at the top of the Data Crawler tab.
Steps
- Open Site Management > Opensolr Search, tab Data Crawler.
- Tick Pages. It shows how many pages of your site can be listed, and Total adds the documents up.
- Tick Include document files (PDF, DOCX, etc.) to index your documents too.
- Click Save Configuration.
- Start a crawl with Reindex Everything so the new list is read: Start, resume and stop the crawl.
What goes into the sitemap
- The standard pages of your site that a visitor can open: not hidden, inside their publish dates, with Include in search left on in the page properties, in every language of your site. Folders, shortcuts, links to other pages, hidden pages, draft versions in a workspace and the search page itself are never listed.
- With Include document files: the document files of your public file storages, PDF, Word, Excel, PowerPoint, OpenDocument text and spreadsheets, plain text and CSV. The crawl also follows links to documents it finds on your pages.
Opensolr reads the text of each document, and a PDF shows up page by page in the results: Documents on your site.
Separate from Data Ingestion
The Data Ingestion tab has its own boxes. What you tick here only decides what the Data Crawler reads.
Data Crawler
- Data Crawler
- What gets crawled
- Crawl speed
- Start, resume, stop
- Reindex or reset
- Crawl status