Let the Web Crawlers Open a Protected URL

Hidden pages that still get indexed

Let the web crawlers in

Protected pages can still be indexed: tick a web crawler and it may open the URL, while everyone else keeps getting 404.

Where: Servers › My Servers › your server › System › Security › Protected URLs
  1. Under Who may open it, find the group Web crawlers. It lists the Opensolr web crawlers with their public addresses.
  2. Tick the crawler that crawls your site, then Protect URL (or Save on an existing row).

Crawlers reach your site over the internet, so the page must be served by the server's own web server with no proxy in front: see Behind a proxy.

Security section

Need more help? Check the error logs or contact support.

Opensolr Server Management Documentation