Web Crawler Dates: Strict UTC Conversion, a Safety Gate, and Extraction From Date-Related CSS Classes

· Web Crawler · Bug Fix · All updates

Two changes to how the web crawler handles dates: offsets are normalised properly before they reach Solr, and pages that publish no structured date at all can now still yield one.

01 Timezone offsets converted to UTC

Date fields carrying a timezone offset, for example +03:00, were causing indexing errors. All dates are now strictly converted to UTC before being sent to Solr, so the offset is resolved rather than passed along.

A final safety gate

Behind the conversion there is now a last check. Any date that does not match the exact Solr format is dropped instead of being sent, so a single malformed date can no longer cause an indexing error for the document.

Extracted date e.g. +03:00 offset Convert to UTC strictly, always Format gate exact Solr format no match: dropped, not an error Date pipeline
Every date is normalised to UTC and then checked against the Solr format; anything that fails is dropped rather than breaking the index.

02 Dates from CSS classes when there is no metadata

Many sites publish no JSON-LD and no date meta tags, so the crawler had nothing structured to read. It now looks for dates inside HTML elements whose CSS classes suggest a date, such as .date, .posted and .info.

This is deliberately narrow. Targeting date-related classes is far safer than scanning all of the page text, which would pick up any number that happens to look like a date.

Web crawler guide

More in Web Crawler

View the full changelog