Web Crawler Dual-Extraction Strategy Captures More Text From JavaScript-Heavy Pages

· Web Crawler · Improvement · All updates

The web crawler now uses a dual-extraction strategy. It runs two independent text extraction engines on a page and keeps whichever captured more real content.

01 Two engines, one result

No single extraction engine wins on every page. One handles a conventional article layout well; another copes better with a page whose content is buried in wrappers. Running both and comparing the output removes the need to guess which one suits a given site.

Page Extraction engine A Extraction engine B Keep the richer text More real content wins
Both engines run on every page; the output with more real content is the one indexed.

02 JavaScript-heavy pages are rendered

Pages with heavy JavaScript, complex layouts, or framework-rendered content — React, Next.js, Angular, Vue — are now detected and rendered automatically, rather than being read as the near-empty HTML shell the server sends.

03 What changes in your index

Richer, more complete text in your Opensolr Index, most noticeably for modern web applications where the visible page and the delivered HTML differ the most.

More in Web Crawler

View the full changelog