How Text Analysis Works: Stemming, Synonyms & Accent Folding
Every word of your pages and every word a visitor types is cleaned up the same way before the two are compared. That is why a search finds pages that use another form of the same word.
01 · What you get
| A search for | Also finds |
|---|---|
running shoes | run shoe, Running Shoes |
café menu | cafe menu, CAFE MENU |
Wi-Fi setup | wifi setup, wi fi setup |
iPhone15 case | iphone 15 case |
John's blog | John blog |
São Paulo hotels | sao paulo hotel |
02 · The steps
The fields title, description, text, uri, author and every *_t field go through the same chain:
HTML tags left in the text are removed.
Accents are folded: é becomes e, ß becomes ss.
The text is split into words with Unicode rules, so Chinese, Japanese and Thai split correctly too.
Full-width and half-width Asian characters are made the same, and the English possessive 's is removed.
A second accent pass catches what the first one missed.
Stopwords are dropped: about 170 common English words such as the, and, of, in, for.
Words are split at hyphens, case changes and digits, and the joined form is kept too: Wi-Fi gives wi, fi and wifi.
Everything goes to lower case.
English stemming reduces words to a common stem.
Search only: the synonyms of synonyms.txt are added to the words searched.
Every filter, in order and with its Solr class: how text is analysed.
03 · Stemming, for real
| Word | Stem |
|---|---|
| running, runs, run | run |
| swimming | swim |
| organization, organizations | organ |
| configured, configuring | configur |
| runner, swam, better | unchanged: they do not match run, swim or good |
The stemmer is the English Snowball stemmer. Words listed in protwords.txt are never stemmed or split.
04 · Change the word lists
stopwords.txt, synonyms.txt and protwords.txt are edited in the Configuration tab; saving reloads the index. Synonyms apply when people search, so a new synonym works on the next search. Stopwords and protected words apply to indexing too: documents already in the index change only when they are sent again.
05 · Two more fields
The spell field keeps words almost as they are, for Did you mean. The tags and title_tags fields keep every beginning of a word, for autocomplete. All the fields: schema reference.