Web Crawler: Choose Curl or Chrome as Your Renderer, With Chrome Pages Now Under a Second
The Web Crawler gains a Renderer setting, so you decide whether pages are fetched fast or executed in a browser. Chrome rendering also got several times quicker, and Crawl Stats can now clear stale error entries.
01 The Renderer setting
| Renderer | What it does |
|---|---|
Curl (Fast) | The default. Fetches pages in around 0.2s each. |
Chrome (JS Rendering) | Runs every page through a headless Chromium browser, for JavaScript SPAs such as React, Vue or Angular where the content is rendered client-side. |
The setting is available both in the UI dropdown and in the REST API, via the renderer parameter, and it persists across cron restarts.
02 Chrome rendering is much faster
Pages in Chrome mode now complete in roughly 0.5–1s instead of 2–25s.
The old approach waited for all network activity to stop before reading the page. On a busy site that never really happens — analytics, trackers and ad pixels keep firing — so the crawler stalled waiting for something that was not coming. It now waits for the DOM, gives JavaScript 500ms to hydrate, and takes the content.
03 Clearing 4xx and 5xx errors
Crawl Stats has a Clear button next to Client Errors and Server Errors. Clicking it deletes those entries from the crawl database.