The Opensolr Web Crawler follows and indexes your entire website's HTML and plain text information, and also creates a sample, embedded search engine, ready for plug-and-play, into your website or application.
It doesn't follow private pages (pages behind firewalls or HTTP Auth, or other Auth mechanisms).
Important: Only works with Solr version 7+
To learn more about what fields are indexed, simply create a new opensolr index, go to Config Files Editor, and select schema.xml.
In order to preserve your Web Crawler's functionality, please do not edit your schema.xml fields, or any other configuration files.
Important: The Out-of-the-box version of the Web Crawler, is only available for small-mid websites, under 10,000 pages!
For larger websites, please Contact Us, to get a quote.