Dynamic Field Explosion in Solr

Account Resources

Dynamic fields let you add fields without touching the schema. Putting a site, an environment, a language or a customer into the field name turns one index into tens of thousands of fields, and Solr pays for every field in every segment.

Each field costs memory per segment

Field metadata, norms and term dictionaries are kept per field and per segment. 20,000 fields across 40 segments is close to a million entries held in memory before the first query runs, and the number grows with every new site or environment.

Commits and merges get slower

Every flush and every merge walks every field of the segments involved. With thousands of fields, the background work after each batch of updates takes far more CPU than the documents themselves.

Put the variable part in a value

Use one field and one value instead of one field per variant: site:shop_de plus title, filtered with fq=site:shop_de, instead of title_shop_de, title_shop_fr and so on.

{
  "id": "shop_de-1001",
  "site": "shop_de",
  "language": "de",
  "title": "Kaffeebohnen Arabica"
}

Keep test environments out of production indexes

Fields created by dev, staging or test sites stay in the index until their documents are deleted and the segments are merged. Point each environment at its own index.

Count your fields

The Luke handler lists every field the index contains: /solr/YOUR_INDEX/admin/luke?numTerms=0&wt=json. A few hundred fields is normal; tens of thousands means field names are being generated from data.

Solr best practices

Dynamic field explosion (this page)