Web Crawler Fix: Underscores in Product Codes No Longer Replaced With Spaces
Product codes and identifiers are no longer altered before indexing. A step that converted emoji into words was also replacing every underscore in the page text with a space, which quietly rewrote any identifier that used them.
01 What was happening
The crawler converts emoji into their word equivalents before text reaches the index. That conversion was applied across the whole text, and it replaced underscores with spaces everywhere, not only in the emoji names it produced. An identifier such as ABC_123_XL was therefore stored as ABC 123 XL, so searching for the code as written did not find the page.
Before
ABC_123_XL was indexed as ABC 123 XL.
After
ABC_123_XL is indexed exactly as written.
Emoji are still converted to words. The only change is that the rest of the text is left alone.