The Data Ingestion API: Push 50 Documents per Batch, Extract PDF and Office Files, Upload JSON Payloads, and Global Rate Limits

The Data Ingestion API lets you push documents directly into your Opensolr index over POST, with the same enrichment the crawler applies. It works alongside the Web Crawler to push the content the crawler cannot reach: gated pages, CMS exports and product feeds.

01 Push up to 50 documents per batch

Submit up to 50 documents per batch. Each one gets automatic vector embeddings, sentiment analysis, language detection and all derived search fields. Submitting the same document ID updates the existing record rather than adding another.

02 Document extraction with rtf:true

Add rtf:true to any document and point its uri at a file. Text is extracted automatically and indexed with full enrichment, and regular and RTF documents can be mixed in the same batch.

03 Upload large batches as a JSON file

Instead of embedding your documents in the request body, save the full payload as a .json file and upload it via the payload_file field. The file can contain everything — email, api_key, core_name and documents — or just the documents array, with the authentication sent as separate form fields. This suits large batches generated by CMS exports or data pipelines.

04 URI is mandatory, and it is the document identity

Every document must now include a URI. The document ID is always generated as md5(uri), which makes the URI the single source of truth for document identity: the same URI means the same document, and resubmitting a URI updates the existing one. Duplicate URIs within pending jobs are automatically rejected, to prevent accidental double-indexing.

Every successful response now returns a doc_ids array, giving the auto-generated document ID for each document in the batch. Use those IDs to track, query or update specific documents in your index.

05 Global API rate limits

Global API rate limits are now enforced across all authenticated API endpoints, including search, AI summaries, embeddings and batch operations.