Real-Time Drupal Content Sync, Per-Document Duplicate Handling and Full UTF-8 Safety

· Data Ingestion · New Feature · All updates

Three changes to data ingestion: Drupal content can be pushed the moment it is saved, a single duplicate no longer sinks a whole batch, and the enrichment pipeline is now safe for any text you throw at it.

01 Real-time content sync from Drupal

Enable one toggle in the Drupal module and every node create, update or delete is pushed instantly to the search index via the Data Ingestion API. Unpublishing a node removes it from search. No cron run is needed for real-time changes.

BEFORE Node saved Wait for cron Index updated AFTER Node saved Ingestion API, immediately Index updated
With the toggle on, a save reaches the index without waiting for the next cron run.

02 Duplicates are skipped, not fatal

The Ingestion API now skips individual duplicate documents instead of rejecting the entire batch. One duplicate URI no longer blocks the other forty-nine documents in a batch of fifty from being indexed.

The Ingestion Queue also auto-refreshes every 20 seconds while jobs are pending.

03 Full CJK and UTF-8 safety

The whole enrichment pipeline is now UTF-8 safe. Japanese, Chinese, emoji and accented characters no longer crash batch processing. All text is sanitised before embedding, before the push to Solr and before JSON encoding, so a single malformed byte in a PDF can never kill an entire batch again.

More in Data Ingestion

View the full changelog