The Opensolr Web Crawler follows and indexes your entire website's HTML and plain text information, and also creates a sample, embedded search engine, ready for plug-and-play, into your website or application.
It doesn't follow private pages (pages behind firewalls or HTTP Auth, or other Auth mechanisms).
Important: Only works with Solr version 7+
To learn more about what fields are indexed, simply create a new opensolr index, go to Config Files Editor, and select schema.xml.
In order to preserve your Web Crawler's functionality, please do not edit your schema.xml fields, or any other configuration files.
To make sure crawling is done correctly, only use our Solr 7+ servers, and make sure to apply the Solr configuration archive, corresponding to your Solr version, below:
Important: The Out-of-the-box version of the Web Crawler, is only available for small-mid websites, under 500 pages.
For larger websites, please Contact Us, to get a quote.