Inline Base64 Images in Indexed HTML

Account Resources

Many CMS editors let authors paste a picture straight into the text. The picture then lives inside the HTML as a data URI, and when the rendered page is sent to Solr, the picture is indexed together with the words around it.

How to spot it

A field value containing src="data:image/png;base64, (or image/jpeg, image/gif) followed by tens or hundreds of thousands of characters. A typical page of text is a few thousand characters; a page with two pasted screenshots can be a megabyte.

Why it hurts

The base64 block is binary data: it adds megabytes of unique, unsearchable terms, makes indexing of that one page use a large amount of memory, and bloats every stored copy. See never put binary data or base64 into Solr.

Fix it at the source

Upload pictures as files and insert them by URL. Most editors can be set to refuse pasted images or to convert them to uploaded files automatically. Existing pages keep their inline images until they are edited or converted.

Fix it in your indexing code

Send text, not markup. Strip the HTML before sending the document, or at least remove the data URIs. The pattern below matches the encoded part of any data URI:

data:[a-zA-Z0-9.+-]+/[a-zA-Z0-9.+-]+(;[a-zA-Z0-9=._-]+)*;base64,[A-Za-z0-9+/=]+

If the field must receive HTML

Add HTMLStripCharFilterFactory to the field type, so tags and their attributes (including src) are removed before tokenizing. This cleans what is indexed; a stored copy of the field still keeps the HTML exactly as sent, so removing the data URIs before sending stays the real fix.

<fieldType name="text_html" class="solr.TextField" positionIncrementGap="100">
  <analyzer>
    <charFilter class="solr.HTMLStripCharFilterFactory"/>
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
</fieldType>

Solr best practices

Inline base64 images in indexed HTML (this page)