Data Ingestion: Empty API Responses Fixed, Duplicate Documents Eliminated, Richer Enrichment and a Queue Dashboard Built for Scale

A round of work on Data Ingestion: one silent failure fixed, one long-standing duplication problem closed, better enrichment on every document, and a queue dashboard that no longer struggles at volume.

01 Empty API responses fixed

The Data Ingestion API was returning empty responses. A PHP 8 compatibility issue in the API layer caused all ingestion, embedding and AI endpoints to fail silently. Full functionality is restored for ingest, embed, ai_summary and all related API calls.

02 No more duplicate documents across the two ingestion methods

Using both the Web Crawler and Data Ingestion on the same index produced duplicate documents, because a trailing slash was enough to change the computed document ID. URIs are now normalized — the trailing slash is stripped — before the document ID is computed, so both methods produce the same document instead of two.

03 Better enrichment on every document

• Structured text for embeddings. Embeddings now use structured text (text_t) when it is available, giving richer semantic vectors.
• Longer sentiment input. Sentiment analysis now reads the first ten complete sentences instead of just the title and description.
• Document size. The document size field is now computed automatically.

04 A queue dashboard built for volume

The Data Ingestion queue dashboard has been redesigned for scale. Indexes and jobs load progressively on click instead of all at once, and a per-index progress dialog shows real-time status for each index separately. It handles thousands of jobs without performance degradation.