Web Crawler: Stop Crawl Button, Instantly Applied Settings, Core-Wide Thread Limits, Cleaner Descriptions and a Buffer Fix
A round of control and correctness work on the Web Crawler: you can now stop a crawl without dismantling its schedule, settings take effect on the crawl that is already running, and the thread limit finally means what it says.
01 Stop Crawl
A new Stop Crawl button in the Web Crawler panel immediately stops all running crawler processes without removing the crawl schedule. The schedule resumes automatically on the next cycle.
02 Settings apply instantly
Crawler settings changes now take effect immediately. When you save a new thread count, crawl mode, renderer or pause setting, the active crawl schedule is automatically updated. There is no need to stop and restart it.
03 max_threads is a core-wide limit
The max_threads setting now controls the total number of concurrent crawler processes across all start URLs for an index, rather than being applied per URL. Setting threads to 1 means exactly one process at a time.
04 Descriptions without CSS or JavaScript in them
Description extraction is smarter: the crawler no longer picks up CSS, JavaScript or theme builder garbage as a page description. Descriptions are now taken in priority order.
- Meta tags. Used first when present.
- JSON-LD structured data. Used next.
- Extracted text. The first two sentences, as a last resort.
05 The Solr buffer no longer sticks after a stopped crawl
The Solr buffer could get permanently stuck after a crawl was stopped mid-run. Documents with oversized embedding payloads were blocking the entire batch. Those payloads are now capped automatically, and each batch flush succeeds independently.