Tika

Opensolr Tika — find answers to your questions

Enable Remote Streaming in Tika Extract with Opensolr

Remote streaming lets Solr fetch a file from a URL and extract its text, instead of you uploading the bytes. Two settings have to be present, and the second one is the one everybody misses.

Remote streaming is disabled by default on Opensolr. Contact us, or write to support@opensolr.com, to have it enabled for your index. Everything below applies to any index, not only to Drupal.

01 · The symptom

You are indexing remote PDFs with stream.url and Solr answers:

Remote Streaming is disabled.

even though enableRemoteStreaming="true" is already set inside <requestDispatcher>. That setting is required, but on its own it is not sufficient.

02 · Both settings must exist

In solrconfig.xml, inside <requestDispatcher>:

<requestDispatcher handleSelect="true">
  <requestParsers enableRemoteStreaming="true" multipartUploadLimitInKB="2048000" />
</requestDispatcher>

And inside the extraction handler itself:

<requestHandler name="/update/extract" class="solr.extraction.ExtractingRequestHandler">
  <lst name="initParams">
    <bool name="allowRemoteStreaming">true</bool>
  </lst>
</requestHandler>

This second block is the part that is usually forgotten. Without it Solr rejects every stream.url request with the error above, and says nothing about why.

03 · Upload, reload, test

Upload the configuration

Edit the files in the Opensolr config editor, or upload the whole config set as a zip from your index control panel.

Reload the index

Open your index in the control panel and use Reload Core in the Tools menu. Nothing changes until the reload.

Send one request

A 200 response, and no Remote Streaming is disabled error, means it is working.

curl "https://YOUR_HOSTNAME.opensolr.com/solr/YOUR_INDEX_NAME/update/extract?stream.url=https://example.com/sample.pdf&literal.id=test-doc&commit=true"

04 · Related reading

Solr reference guide: indexing with Tika

The extraction handler, its parameters and the metadata it produces.

Tika extraction is failing

If the error is about extraction itself rather than about streaming, start there.

Read Full Answer

Tika Extractor doesn't work with my Opensolr Index

Tika extraction failing on your index, from Drupal or from your own code? Four things to check, in the order that resolves most cases.

01 · Try a different Solr version

Before reading logs, create a new Opensolr index on Solr 9, or on 7, 6 or 5. Avoid Solr 8.x: its Jetty version has known issues that show up as Tika failures. Create a new Opensolr index and point your integration at it to confirm.

02 · Make sure Tika is enabled in your region

Tika is not present in every environment. If you suspect it is missing, contact Opensolr support and we will enable it for your index and region.

03 · Confirm the handler in solrconfig.xml

Your solrconfig.xml needs the extraction request handler defined:

<requestHandler name="/extract/tika" class="org.apache.solr.handler.extraction.ExtractingRequestHandler" startup="lazy">
    <lst name="defaults">
    </lst>
    <!-- This path only extracts - never updates -->
    <lst name="invariants">
        <bool name="extractOnly">true</bool>
    </lst>
</requestHandler>

With extractOnly set as an invariant, this path extracts text and never writes documents, so a misconfigured client cannot update your index by accident.

04 · Still failing

Send us the index name and the error

Contact support with the exact message you get back. Most Tika problems are a version mismatch between Solr and the client library, which is quick for us to confirm from our side.

Indexing remote files by URL

If the error mentions remote streaming rather than extraction, that is a different switch: see enabling remote streaming.

Solr and Jetty version compatibility is the first thing to check on any strange extraction error. It is also the cheapest test: a new index on another version takes a minute.
Read Full Answer