Web Crawler Skips Embedding Generation on Indexes Without AI Vector Access
The Web Crawler now respects the package’s vector access flag, and skips embedding generation on indexes that do not have AI features.
01 Work that was never used
Embeddings were being generated during every crawl regardless of whether the index could use them. On a package without AI features the vectors served no purpose: the work was done, the GPU time was spent, and the crawl took longer for nothing.
The crawler now checks the vector access flag first. Where there is no vector access, the embedding step is skipped entirely, which frees GPU resources and makes indexing faster on those indexes.