N-gram Fields Belong on Short Text Only

Account Resources

An n-gram field indexes every substring of every word. It makes partial matching easy, and it multiplies the size of whatever it is applied to. Used on the wrong field, it is the most common cause of slow indexing and memory spikes.

What an n-gram filter does

With NGramFilterFactory minGramSize="2" maxGramSize="25", the word coffee becomes 15 terms (co, cof, coff, … offee, ffee, fee, ee). A 20-letter word becomes 190. A random string, such as a base64 block or a tracking id, becomes thousands of terms that are all unique.

Use it on short fields

Titles, product names, SKUs, part numbers, people and place names: fields of a few words where people type fragments and expect a match.

Never on body text or whole pages

A page of 2,000 words turns into tens of thousands of terms, more with term vectors. Indexing a rendered page as n-grams makes every document dozens of times heavier, and one unusual page (long codes, embedded data) can need more memory than the server has.

Prefer edge n-grams and a small maxGramSize

For "starts with" matching (autocomplete), EdgeNGramFilterFactory produces one term per prefix: about as many terms as letters, not their square. Keep maxGramSize close to what people actually type, 10 to 15, and apply the filter at index time only.

<fieldType name="text_prefix" class="solr.TextField" positionIncrementGap="100">
  <analyzer type="index">
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.LowerCaseFilterFactory"/>
    <filter class="solr.EdgeNGramFilterFactory" minGramSize="2" maxGramSize="15"/>
  </analyzer>
  <analyzer type="query">
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
</fieldType>

Check before you deploy

Paste a real value into the Analysis screen of the Solr admin and count the terms the field produces. More on prefix fields: autocomplete in Solr and EdgeNGram fields.

Solr best practices

N-gram fields belong on short text only (this page)