Data Ingestion

Opensolr Data Ingestion — find answers to your questions

API - List the Data Ingestion queue (ingest_queue)

Lists your latest 100 Data Ingestion jobs, newest first, each with its state, counts and the documents it carries.

Endpointhttps://api.opensolr.com/solr_manager/api/ingest_queue
MethodGET or POST
Authemail and api_key of your account, as GET or POST parameters: Authentication. Never rate limited.

01 · Parameters

core_nameOptional

Only the jobs of this index. Without it: every index of your account.

02 · Example

curl -s "https://api.opensolr.com/solr_manager/api/ingest_queue?email=YOUR_EMAIL&api_key=YOUR_API_KEY&core_name=my_index"

03 · Answer

status is true and jobs lists the jobs with the same fields as ingest_status, plus payload, the documents of the job as a JSON string.

With many jobs, the lighter calls are better: ingest_queue_list_jobs pages through the jobs of an index without their documents, ingest_queue_core_progress gives the progress alone.

Jobs that are not running are removed 30 days after they were queued.

04 · Errors

ERROR_QUEUE_UNAVAILABLE_TRY_AGAINHTTP 200

The queue did not answer. Ask again.

WRONG_API_HOSTHTTP 404

Called on opensolr.com. The answer carries the right URL.

ERROR_AUTHENTICATION_FAILEDHTTP 403

The email and API key do not match.

05 · Related

Act on a job: ingest_queue_action. Your indexes with jobs: ingest_queue_list_cores.

Start the waiting jobs now: ingest_run_now. Retry the failed ones: ingest_queue_retry_failed.

Every error code and HTTP status of the API: API errors. Calls per minute and per hour: Rate limits.

Read Full Answer

API - Push documents (ingest)

Sends up to 50 documents to your Opensolr Index in one call. The batch goes into the Data Ingestion queue and is written in the background, each document enriched the same way as a crawled page.

Endpointhttps://api.opensolr.com/solr_manager/api/ingest
MethodPOST with a JSON body (or form fields)
Authemail and api_key of your account, as GET or POST parameters: Authentication.
NeedsAn index in a region with site search: What your index needs.

01 · Parameters

core_nameRequired

The name of your index (index_name works too).

documentsRequired

1 to 50 documents: a JSON array, or a JSON string in a form field.

payload_fileOptional

Instead of documents: a JSON file sent as multipart, holding the documents, or the whole request: Send a JSON file.

02 · A document

uriRequired

The http or https address of the document. It identifies it: the document id is made from it, and sending the same uri again replaces the document.

titleRequired

The title.

descriptionRequired

A short summary.

textRequired unless rtf

The full text.

rtfOptional

true: Opensolr downloads uri and reads the text itself, PDF, Word and more included: Files and documents. A PDF is written one document per page: PDFs page by page. To keep one document per PDF, send its text in text and leave rtf out.

More fields, such as prices, dates and your own fields: Document fields.

03 · Example

curl -s -X POST "https://api.opensolr.com/solr_manager/api/ingest" \
  -H "Content-Type: application/json" \
  -d '{
    "email": "YOUR_EMAIL",
    "api_key": "YOUR_API_KEY",
    "core_name": "my_index",
    "documents": [
      {"uri": "https://www.example.com/products/blue-chair", "title": "Blue chair",
       "description": "A solid oak chair in blue.", "text": "The full product text..."},
      {"uri": "https://www.example.com/manuals/blue-chair.pdf", "title": "Blue chair manual",
       "description": "Assembly and care.", "rtf": true}
    ]
  }'

04 · Answer

{
    "status": true,
    "msg": "QUEUED",
    "job_id": "9f2c41d0b7e84a1c8e3f5a6b7c8d9e0f",
    "total_docs": 2,
    "doc_ids": [
        "...",
        "..."
    ]
}

Keep job_id to follow the job: ingest_status. A document whose uri is already waiting in the queue for this index is skipped; when every document was, the answer is "msg": "All documents already queued" with dupes_skipped.

05 · Errors

VALIDATION_ERRORSHTTP 200

One or more documents break the rules above. errors lists each one by its position.

ERROR_DOCUMENTS_MUST_BE_NON_EMPTY_ARRAYHTTP 200

No documents.

ERROR_BATCH_LIMIT_50_DOCUMENTS_MAXHTTP 200

More than 50 documents.

ERROR_PAYLOAD_TOO_LARGE_MAX_...HTTP 200

The batch is larger than the server accepts. The code ends with that size, for example ERROR_PAYLOAD_TOO_LARGE_MAX_128MB: send fewer documents per call.

ERROR_DISK_QUOTA_EXCEEDEDHTTP 200

The index would go over its disk limit (current_mb, max_mb).

ERROR_BANDWIDTH_LIMIT_EXCEEDEDHTTP 200

The index used its traffic for this month (used_mb, max_mb).

ERROR_CORE_NOT_WEBCRAWLER_ENABLEDHTTP 200

The index is not in a region with site search. The message names the regions that have it.

ERROR_NOT_CORE_OWNERHTTP 200

The index is not yours.

ERROR_QUEUE_UNAVAILABLE_TRY_AGAINHTTP 200

The queue did not answer. Send the batch again.

WRONG_API_HOSTHTTP 404

Called on opensolr.com. The answer carries the right URL.

ERROR_AUTHENTICATION_FAILEDHTTP 403

The email and API key do not match.

ERROR_RATE_LIMIT_PER_MINUTEHTTP 429

Too many calls. Wait for the Retry-After seconds: Rate limits.

All the limits, per document and per plan: Limits and refusals. What the enrichment and the vectors use from your plan: What it uses from your plan.

06 · Related

The queue: ingest_queue. Update and remove documents: Update and remove.

Every error code and HTTP status of the API: API errors. Calls per minute and per hour: Rate limits.

Read Full Answer

API - Send a file for text extraction (file_extract)

Sends Opensolr a file that has no public address, such as a document stored in your CMS, and gets its text back: PDF pages one by one, scanned pages and pictures read by OCR.

Endpointhttps://api.opensolr.com/solr_manager/api/file_extract
MethodPOST, multipart/form-data when a part is sent
Authemail and api_key of your account, as GET or POST parameters: Authentication.

01 · Parameters

index_nameRequired

The index the file is for.

sha1Required

The SHA-1 of the whole file, 40 hex characters.

sizeRequired

The size of the whole file, in bytes. At most the per-document size of your plan.

nameOptional

The file name.

partOptional

The bytes of the file from offset, as a multipart file, at most max_part_bytes.

offsetOptional

Where part starts in the file, in bytes. Default 0.

pagesOptional

1: a PDF comes back as its pages (page, text, is_ocr, lang).

textOptional

0: the state alone, without the text.

02 · How it goes

Ask first with sha1 and size only. A file read before comes back at once, state done, with its text.

Otherwise send the file in parts: each answer says how many bytes were received; send the next part from there.

After the last part the job is queued, then processing, then done. Follow it with file_extract_status.

To index the text, send a document with rtf: true and file set to the job_id to ingest.

03 · Example

SHA=$(shasum -a 1 manual.pdf | cut -d' ' -f1); SIZE=$(wc -c < manual.pdf | tr -d ' ')
curl -s -X POST "https://api.opensolr.com/solr_manager/api/file_extract" \
  -F "email=YOUR_EMAIL" -F "api_key=YOUR_API_KEY" -F "index_name=my_index" \
  -F "sha1=$SHA" -F "size=$SIZE" -F "name=manual.pdf" -F "offset=0" -F "part=@manual.pdf"

04 · Answer

{
    "status": true,
    "job_id": "2fd4e1c67a2d28fced849ee1bb76e7391b93eb12-od",
    "max_part_bytes": 67108864,
    "state": "queued"
}

Reading the text is free. Scanned pages and pictures read by OCR, and pictures described by what they show, count against the AI allowance of the index owner: What it uses from your plan.

05 · Errors

ERROR_INVALID_SHA1HTTP 200

sha1 is not 40 hex characters.

ERROR_INVALID_SIZEHTTP 200

No size.

ERROR_FILE_TOO_LARGEHTTP 200

Larger than your plan allows (max_bytes).

ERROR_PART_TOO_LARGEHTTP 200

The part is larger than max_part_bytes.

ERROR_INVALID_OFFSETHTTP 200

The part does not fit at this offset.

ERROR_UPLOAD_FAILEDHTTP 200

The part did not arrive whole. Send it again.

ERROR_EXTRACTION_QUEUE_FULLHTTP 200

The extraction queue is full. Try again later.

ERROR_EXTRACTION_QUEUE_UNAVAILABLEHTTP 200

The queue did not answer. Try again.

ERROR_NOT_CORE_OWNERHTTP 200

The index is not yours.

WRONG_API_HOSTHTTP 404

Called on opensolr.com. The answer carries the right URL.

ERROR_AUTHENTICATION_FAILEDHTTP 403

The email and API key do not match.

ERROR_RATE_LIMIT_PER_MINUTEHTTP 429

Too many calls. Wait for the Retry-After seconds: Rate limits.

06 · Related

PDFs page by page in your index: PDFs page by page. OCR: Text in scans and pictures.

Every error code and HTTP status of the API: API errors. Calls per minute and per hour: Rate limits.

Read Full Answer

Loading more articles...