Data Ingestion API - Push Documents to Solr

Data Ingestion API

The Data Ingestion API is part of the Web Crawler site search product: it lets you push documents into a Web Crawler index from your own application, database, or script. Instead of crawling a website, you send your content as structured JSON — and Opensolr takes care of the rest, including AI enrichment with embeddings, sentiment analysis, and language detection. For the complete API reference with code examples in multiple languages, see the Data Ingestion API guide.

When to Use Data Ingestion vs. Web Crawler

Opensolr gives you two ways to get content into your index. Here is how to pick the right one:

API Endpoint

To push documents, send a POST request to the Data Ingestion endpoint. Here is an example using curl:

Replace YOUR_INDEX_NAME with your Opensolr index name and YOUR_API_KEY with your API key (found in your Opensolr dashboard). Need to index PDFs, Word documents, or other binary files? See Document Extraction (extractOnly) for how to use the rtf=true parameter. The file type is detected from the file itself (PDF, Word, Excel, PowerPoint, OpenDocument, RTF, text, HTML).

Scanned PDFs and pictures inside documents are read too. Every picture inside a PDF is read with OCR in the language of its page, detected automatically, in more than 100 languages. A picture with no text in it, such as a photo or a chart, gets a short description of what it shows, so it can be found by its meaning. A PDF read this way is indexed one Solr document per page (PDFs are indexed page by page); each page holds its own text first, then what was read in its pictures. Each picture read costs 0.5 of an AI request and each picture described another 0.5; anything read before is free (API Quota). See it live on 33 test PDFs.

PDFs Are Indexed Page by Page

A PDF sent with rtf:true is not stored as one big document. Opensolr reads it and writes one Solr document per page. Each page is searched, ranked, highlighted and vectorised on its own, so a query finds the page that answers it instead of a 200-page file that mentions it somewhere. Every other document (Word, Excel, HTML, text, and any document you send with its text already filled in) is still one document, exactly as you sent it.

What each page document holds

Finding the pages of a PDF

The document id you sent is no longer a document in the index: its pages are. To work with one PDF, filter on uri_id_s:

One page of it: fq=uri_id_s:"DOCUMENT_ID" and fq=page_i:7. A search returns pages; the hosted search page shows Page N on each and opens the PDF at that page, and on its Documents tab every PDF is one result, with its best pages under it.

One result per PDF, with its best pages

Since every page is a document, a search can return several pages of the same PDF. To show one result per PDF with its best pages under it, group on uri_id_s. Solr does it natively, two ways.

Collapse and expand. The results list stays flat: one entry per PDF (its best page), and an expanded section holds the other matching pages of each PDF, keyed by uri_id_s, with how many of them matched in its numFound. numFound, paging with start and rows, and facet counts all count PDFs. This is what the hosted search page uses.

On a hybrid query. The same parameters work with {!hybrid}. Give the collapse sort by reference, as above: {!hybrid} picks its candidates with your filter queries but leaves out any filter that refers to a parameter named q…, so it ranks every page and the collapse groups them afterwards. With the sort written inside the filter (sort='score desc'), the candidates would already be collapsed and the expanded section would come back nearly empty.

All the pages of one PDF, in the same order, to list more than five: the same query without the collapse and expand parameters, with fq=uri_id_s:"DOCUMENT_ID" and sort=score desc,page_i asc, paged with start and rows.

Grouping. The response is a list of groups, one per PDF, each with its matching pages in order of relevance and the total number of pages that matched. group.ngroups=true adds the number of PDFs that matched.

With grouping, every document without uri_id_s lands in one shared group whose groupValue is null; collapse with nullPolicy=expand keeps them as separate results instead.

Grouping works on keyword (edismax) queries; on hybrid queries use collapse and expand. To read a PDF from its first page to its last: fq=uri_id_s:"DOCUMENT_ID" with sort=page_i asc. See it on the live OCR demo.

Sending a PDF again

Before a PDF's pages are written, every earlier page of that PDF and any earlier whole-document version of it are removed (id or uri_id_s equal to the document's id). Re-sending a PDF that now has fewer pages leaves no pages behind. A PDF that cannot be read, or has no page with text, is indexed as one document with the text you sent.

What it costs

Each page gets its own vector: one AI request per page vectorised; a page whose content was vectorised before is free. Pictures read by OCR count 0.5 each, as before (API Quota).

Document Structure

Each document you push is a JSON object with fields. Some fields are required, others are optional. Here is the full reference:

Required Fields

Every document must include these fields:

Optional Fields

Add these fields to enrich your documents with extra metadata:

Field Type Suffixes

When you create custom fields, add a suffix to tell Opensolr what type of data the field contains. This ensures proper indexing and enables features like sorting and filtering.

AI Enrichment Pipeline

Every document you push through the Data Ingestion API is automatically enriched by AI before it hits your index. This happens behind the scenes — you do not need to do anything extra.

Here is what each enrichment step does:

• Embeddings — your document's text is converted into a 1024-dimensional numerical vector. This enables vector/semantic search — finding documents by meaning, not just keywords.
• Sentiment Analysis — the text is analyzed to determine its overall tone: positive, negative, or neutral. Useful for filtering reviews, feedback, or social media content.
• Language Detection — the language of the text is automatically identified. This enables language-aware search features and helps with multilingual indexes.

Ingestion Queue Dashboard

When you push documents, they enter an ingestion queue where they are processed one batch at a time. The dashboard shows you exactly what is happening with each batch.

Job States

Each ingestion job goes through these states:

Jobs can also be in a Paused state if you manually pause them from the dashboard.

Available Actions

Progress Monitoring

While a job is processing, the dashboard shows a progress bar and counts: how many documents have been enriched and indexed out of the total. You can watch it in real time. For detailed documentation on all queue actions including pause, resume, retry, and monitoring, see Ingestion Queue Management.

Automatic Deduplication

Opensolr automatically prevents duplicate documents using your uri field. Here is how it works:

A PDF sent with rtf:true follows the same rule one level down: its pages carry ids derived from the document's id and the page number, so re-pushing the PDF replaces its pages (PDFs are indexed page by page).

Step-by-Step: Pushing Your First Document

Here is a quick walkthrough to get your first document into your index via the API:

1. Get your API key — find it in your Opensolr Dashboard under your account settings.
2. Know your index name — this is the name you gave your index when you created it (e.g., my-products).
3. Prepare your document as a JSON object with at least the four required fields: uri, title, description, and text.
4. Send a POST request to https://api.opensolr.com/v1/ingest/YOUR_INDEX_NAME with your API key in the Authorization header and your document in the request body (see the code example above).
5. Check the response — a successful push returns a confirmation with the job ID. You can track the job in the Ingestion Queue Dashboard.
6. Search for your document — once the job completes (usually within seconds), your document is live and searchable in your index.

Related Documentation