WEB CRAWLER-Search Analytics

Find answers to your questions quickly and easily

Opensolr Analytics — Solr-Level vs Web Crawler Search Anal...

Analytics Guide
Opensolr Analytics — Two Levels of Insight
Opensolr gives you analytics at two levels: Solr-level monitoring for every index (infrastructure, performance, raw queries), and Search Analytics for Web Crawler indexes (what users search, what they click, what returns nothing). Different tools for different questions.
Two Levels of Analytics
Solr-Level Monitoring
Available on ALL Opensolr indexes
Request Volume (daily/hourly charts)
Live Query Stream (real-time feed)
QPS Avg QPS per Server
/q Request Path analysis
Disk and Bandwidth Usage
Deep Query Inspection (click any query)
Answers: “How is my Solr infrastructure performing?”
Search Analytics
Available on Web Crawler indexes
Top Queries (all time + last 24h)
0 No-Results Dashboard (by unique IP)
Click Analytics and CTR tracking
Daily search trends and volume
CSV Export
Elevation Rules management
Answers: “What are my users searching for?”
1 Solr-Level Monitoring — Infrastructure Analytics
This is available on every Opensolr index — whether it's a Web Crawler index, a custom index with your own schema, or a Tika index. It answers infrastructure-level questions: how many requests per second is my index handling? What are the raw queries hitting Solr? How much disk and bandwidth am I using?
You access it from your index Control Panel. There are two interfaces:
Query Analytics (sidebar)
The Query Analytics panel in the sidebar loads the Solr-level analytics dashboard inside the Control Panel. This shows:
  • Request Volume — daily and hourly charts of query traffic over a date range
  • Request Path — breakdown by Solr endpoint (/select, /autocomplete, etc.)
  • Popular Queries — the raw Solr queries ranked by frequency
  • Latest Queries — most recent queries with timestamps
  • QPS per Server — average queries per second on each replica
  • Disk and Bandwidth Usage — storage consumed and data transferred
Live Query Stream (sidebar)
The Live Query Stream panel shows every query hitting your Solr index in real-time. This is the raw firehose:
  • Every request — timestamp, full query string, response time in ms, number of hits, HTTP status
  • Regex filtering — search through thousands of queries by path or parameter patterns
  • Deep inspection — click any query to see every parameter in a clean table, ready to copy and replay
  • Auto-refresh — keeps updating as new queries arrive
Data source: Solr-level analytics are powered by server access logs, parsed daily by the log parser and stored in dedicated analytics Solr shards. The data covers every HTTP request to your Solr index — including API calls, crawler requests, and Search UI queries. This is infrastructure-grade visibility.
2 Search Analytics — User Behavior Intelligence
This is available on Web Crawler indexes. It answers user-facing questions: what are people actually searching for on my site? Which results do they click? What searches return nothing? Where should I improve my content?
You access it from the Query Stats button in the Web Crawler panel, or via the sidebar's Query Analytics (which loads the same dashboard). It has five tabs:
Overview
Total searches, unique queries, zero-result count, and daily search volume chart at a glance.
Queries
Top search terms ranked by frequency. Two sub-views: Last 24 Hours and All Time. Sortable, searchable, with CSV export.
No Results
Every query that returned zero results, tracked by unique IP. Your content gap detector. Full guide →
Click Analytics
Top Clicked docs, CTR by query, Low CTR alert list. IP-deduplicated. Full guide →
Elevation Rules
View and manage all active pin/exclude rules across queries. Elevation guide →
Data source: Search Analytics data is collected by the Opensolr Search UI itself. When a user searches, the query is logged. When they click a result, the click is recorded. When a search returns zero results, it's flagged. All data is stored per-index and is immediately available — no daily processing, no lag.
When to Use Which
Use Solr-Level Monitoring when you need to...
  • See how many requests per second your index handles
  • Debug a slow query by inspecting its full Solr parameters
  • Check disk usage or bandwidth consumption
  • Monitor performance across your cluster replicas
  • Analyze traffic patterns over days or weeks
  • See raw Solr queries from API integrations, crawlers, and Search UI combined
Use Search Analytics when you need to...
  • Know what your end users are searching for
  • Find searches that return zero results and fix content gaps
  • Track which results users actually click
  • Find low-CTR queries where results aren't satisfying users
  • Export search terms to CSV for marketing or SEO analysis
  • Manage query elevation rules from a central dashboard
Aspect Solr-Level Monitoring Search Analytics
Available on All Opensolr indexes Web Crawler indexes
Data source Server access logs, parsed daily Search UI events, recorded in real-time
Tracks Every HTTP request to Solr (API, crawlers, UI, everything) User search queries and result clicks only
Key metrics Request volume, QPS, response time, paths, disk, bandwidth Top queries, zero-result queries, click-through rates
Granularity Raw Solr parameters visible per query Clean search terms (what the user typed)
IP tracking Available in raw logs IP-deduplicated counts (unique users)
Export Charts and table views CSV export of all queries
Best for DevOps, performance monitoring, debugging Product managers, SEO, content strategy
Both levels of analytics are included in every Opensolr plan.
No add-ons. No extra tiers. No per-query charges for analytics. It's all there the moment your index goes live.
Read Full Answer

No-Results Dashboard — Find and Fix Content Gaps

Search Analytics
No-Results Dashboard
Every search that returns zero results is a missed opportunity. The No-Results Dashboard tracks them all — so you can find the gaps and fix them.
How Zero-Result Tracking Works
?
User Searches
“return policy”
→
0
Zero Results
No matching docs
found in the index
→
Dashboard Logs It
Query + unique IP
→
You Fix It
Synonym, content,
or elevation rule
IP-Deduplicated
One user refreshing 50 times = 1 count. Real unique users, not inflated numbers.
Auto-Cleanup
Zero-result entries older than 180 days are automatically cleaned up.
Bulk Management
Select and delete junk or test queries in bulk across all analytics tabs.
What Gets Tracked
The Query
The exact search term the user typed. Stored as-is so you can see exactly what they were looking for — typos, misspellings, and all. This alone tells you what content your site is missing.
Unique IP Count
How many distinct users searched for this term and got nothing. One person refreshing repeatedly counts as one. This tells you whether a gap affects one person or hundreds.
Timestamp
When the zero-result search happened. Spot trends — did a recent content removal cause a spike in zero-result queries? Are seasonal terms appearing that you haven't covered yet?
How to Fix Zero-Result Queries
Add Content
If users are searching for something you genuinely don't cover, create that content on your site and let the next crawl pick it up.
Add Synonyms
If the content exists but users search with different words, add synonyms to your Solr configuration. "return policy" and "refund policy" should match the same pages.
Create Elevation Rules
Pin a relevant result to a query that currently returns nothing. Once elevated, that document will always appear for that search. Query Elevation guide →
Delete Junk Queries
Bot traffic, test searches, or nonsense strings? Select them in bulk and delete. Keeps your dashboard clean and actionable.
Where to Find It
Open your Opensolr Control Panel, go to your index, and click Query Analytics in the sidebar. The dashboard has multiple tabs — click the No Results tab to see all zero-result queries, sorted by unique IP count (highest first). From there you can select queries individually or in bulk, delete junk, and plan your content fixes.
Every zero-result query is a content gap waiting to be filled.
The dashboard shows you exactly what your users want but can't find. Fix the gaps and your search gets better every week.
Read Full Answer

Click Analytics and CTR — Track What Users Actually Click

Search Analytics
Click Analytics and CTR
Knowing what people search for is only half the picture. Click Analytics shows you which results they actually engage with — and which they ignore.
Three Views of Click Data
1 Top Clicked
Which documents get the most clicks across ALL search queries.
/products/boots 847
/about/returns 612
/sale/winter 389
Best for:
Finding your most popular content. These pages matter most to your users.
2 By Query
Click-through rate for each search query individually.
“winter boots”68%
“product manual”31%
“return policy”8%
Best for:
Comparing engagement across queries. High impressions + low CTR = relevance problem.
3 Low CTR Action
High-impression queries where nobody clicks. These need fixing.
“shipping cost” 2% CTR
“store hours” 4% CTR
“warranty info” 5% CTR
Best for:
Finding queries where results appear but don't satisfy users. Pin better docs or improve content.
How Click Tracking Works
Automatic Collection
Every time a user clicks a search result, the click is recorded with the query, the document URL, and the result position. No extra integration code needed — the Opensolr Search UI handles it automatically.
IP-Deduplicated
Click counts are deduplicated by IP address. One user clicking the same result 20 times counts as one click. This gives you accurate engagement metrics, not inflated numbers from bots or repeated clicks.
Rate-Limited
Click recording is rate-limited to prevent abuse. Automated scrapers or bots hammering your search won't pollute your analytics data.
What to Do With Low CTR Queries
A low click-through rate means users see results but don't find them useful. This is a relevance problem, and there are several ways to fix it:
  • Pin a better result — use Query Elevation to force the most relevant page to the top for that query
  • Exclude bad results — if irrelevant pages are showing up, exclude them via elevation rules or URL exclusion in search.xml
  • Improve your content — if the page title and description don't match what users expect, update them on your site and recrawl
  • Add synonyms — if the right page exists but uses different terminology, add synonyms so it matches the query
Where to Find It
Open your Opensolr Control Panel, go to your index, and click Query Analytics in the sidebar. The Click Analytics tab gives you three sub-views: Top Clicked, By Query, and Low CTR. IP data older than 90 days is automatically cleaned up, keeping your dashboard focused on recent behavior.
Search doesn't end when results appear. It ends when users click.
Click Analytics tells you whether your results are actually useful. Fix the low-CTR queries and your search gets measurably better.
Read Full Answer