Web Crawler: Cleaner Tag Fields for Autocomplete, and Failed Solr Batches Retried
Two crawl-time fixes: the tag fields behind suggestions are cleaned before they are stored, and documents from a failed Solr batch are no longer thrown away.
01 Cleaner tag fields
The tags and title_tags fields used for autocomplete and spellcheck were being stored with raw special characters intact, which could produce noisy or broken suggestions. They are now cleaned as they are generated: special characters stripped, whitespace normalised.
Tags stored verbatim, punctuation and stray whitespace included, so suggestions came back noisy or broken.
Tags cleaned during the crawl, giving cleaner autocomplete and spellcheck results.
02 Failed batch inserts are retried
Solr batch indexing during a crawl is more reliable. When a batch insert to Solr fails — a temporary overload or a timeout, for example — the documents are kept in the local buffer and retried on the next flush cycle, rather than being silently lost.