Pages built with JavaScript
Some sites send an almost empty page and build the content in the browser with JavaScript: React, Vue, Angular and the like. To read what your visitors really see, switch the renderer to Chrome.
The two renderers
The default. Reads each page exactly as your server sends it. Fast and light, and right for most sites, where the text is already in the page.
Opens every page in a real Chrome browser, lets the scripts run, and keeps the browser's version when it carries more content. Slower, so use it when your site needs it.
Do you need Chrome?
Open a page of your site and view its source (the source, not the inspector). If your text is not there, the page is built with JavaScript and needs Chrome. After a crawl, Crawl Stats shows under Dynamic Pages how many pages were rendered with JavaScript.
Switch it
- Open the crawl settings and set Renderer to Chrome (JS Rendering).
- Click Save Settings. If the crawl is running, click Stop Crawl Schedule, then Start Crawl Schedule.
The HTTP Basic Auth of a site, set in its rules, is used in Chrome too.
Pages behind a browser check
Some sites answer a plain request with an error 403 or a "checking your browser" page. With Chrome, Opensolr tries such a page again in the browser and keeps it when the browser reaches the real page. A page that stays behind the check is not indexed. With Curl, these check pages are recognised and never indexed as content.
Crawl your site: all pages
- Crawl your site overview
- Add start URLs
- Prove you own the site
- Crawl settings
- How far the crawl goes
- Sites built with JavaScript
- Documents on your site
- Rules per site
- Pages left out everywhere
- Meta robots, nofollow, canonical
- Start, pause, stop, flush
- Crawl Stats
- Reindex
- Keep your index fresh
- Recrawl from your CMS
- What a page needs
- URLs and duplicates