With Data Ingestion, the extension sends your pages and document files straight to your Opensolr Index, through the Opensolr Data Ingestion API. Opensolr does not have to visit your pages, so there is no crawl to wait for and nothing to whitelist. Pushed documents get the same treatment as crawled pages: What Opensolr adds to every document.
The tab
- Content Types for Ingestion: a Pages box with the number of pages it covers, a Total, and Include document files (PDF, DOCX, etc.). These boxes are separate from the ones of the Data Crawler tab.
- Real-Time Sync: sends each page the moment you save it. Real-Time Sync
- Cron batch size: how many documents each bulk run handles. Ingest All Now
- Save Ingestion Settings, then, under Bulk Ingestion, Ingest All Now and Data Ingestion Status.
What is sent for each page
- The title, the address, a description (the abstract, else the description of the page properties, else the start of its content) and the full text of its content elements: headers, text, bullet lists.
- The first image of the page's Media field, the author, the creation and last update dates, the locale of the page's language and the domain.
- The category paths of the page's system categories for hierarchical filters and your Field Mappings.
- Each language of a page is sent as its own document, with its own address and locale.
- For document files: their address and the title and description from the file metadata. Opensolr fetches the file and reads its text, so the file must be reachable from the internet. Files and documents
Never sent
- Hidden pages, pages outside their publish dates and pages with Include in search switched off.
- Folders, shortcuts, links to other pages and every other page type that is not a standard page; draft versions in a workspace.
- Files of storages that are not public, or that are offline, and files TYPO3 marks as missing.
- The search page itself.
Crawl and push together
You can use Data Ingestion alone, or next to the Data Crawler. Opensolr knows each page by its address, so a page that arrives both ways is one document: the last one to arrive wins.
Pages of this tab
- Real-Time Sync: each save goes to your index at once.
- Ingest All Now: send your whole site in the background.
The same API, for your own code: Push your data.