Why Did My Data Ingestion Fail? Find the Reason and Fix It

When your JSON documents did not reach your Opensolr Index, the reason is always written down: in the answer to your request, or in the job, document by document. Find it, fix the documents, and send them again.

Refused at once, or failed in the queue

• Refused at once. The answer to your request has "status": false and msg says why: too many documents, a missing field, your plan's disk or bandwidth used up. Nothing was queued. Every answer of this kind: Limits and refusals.
• Failed in the queue. The request was accepted and got a job_id, but some or all of its documents could not be written. The job is Failed (every document failed) or Completed with documents in error (partially failed), and it says why for each document.

See why a job failed

1. Open the Account menu > Data Ingestion.
2. On the row of your index, click the failed or partially failed counter: only those jobs are listed.
3. Click the Job ID. The Error box gives every reason with how many documents it hit. For a job where every document failed, the Result also marks each document with its own reason.
4. For an error Solr gave while writing, click Check Solr Error Log: it opens the Error Log of your index (Error Log).

The same through the API: ingest_status with your job_id gives the state and the error, ingest_queue_job_detail gives the documents you sent and the result (ingest_queue_job_detail). Jobs are kept 30 days.

The common reasons

• unknown field 'director' (in an HTTP 400 Bad Request from Solr)
  The document has a field your index does not know. Give your own fields a name ending with a type suffix of the schema, such as director_s for a value you filter on or year_i for a number, or leave the field out (Document fields).
• rtf extraction: ... or The file could not be read: ...
  With "rtf": true, the file at the document's uri could not be downloaded or read: a wrong address, a site that refused the download, a file over your plan's document size. Open the address in a browser, fix it, and send the document again (Files and documents).
• rtf:true requires a valid uri or rtf: only http/https URLs allowed
  A document sent with "rtf": true needs a full http:// or https:// address in uri.
• The file is still being read (file_extract job ...)
  You sent a file with file_extract and pushed the document before its reading was done. Wait for the state done of that job, then send the document again.
• No file_extract job ... for this account
  The file value is not a job id that file_extract gave to your account. Send the file with file_extract first and use the id it answers.
• Not an object
  An item of your documents list is not a JSON object (a string, a number, a list). Every document is one { ... } with its fields.

These are not failures: a file with no text in it (an empty or unsupported file) is written with an empty text, and a job that waits for meaning vectors goes back to Pending with a note, then carries on where it stopped.

Fix it and send it again

• In the queue: open the job, change the documents under Payload (Documents Sent), click Save Payload, then Retry (The Data Ingestion queue).
• All at once: once the cause is fixed on your side (a field added, a site that lets the file be downloaded), Retry Failed Jobs and Retry Partially Failed put every such job of the index back in the queue.
• From your code: send the fixed documents again. The same uri replaces the same document, so nothing is duplicated (Update and remove documents).