Web Crawler Auto-Removes Dead Pages Returning 404 or 500 From the Search Index
Web Crawler now automatically removes documents from the search index when their pages return non-200 status codes (404, 500, etc.) during crawling. Previously, dead pages could remain in search results indefinitely.