fix: use URL-based hash as Pinecone vector ID to prevent silent data loss (#174) - #176
Open
ahmedk20 wants to merge 1 commit into
Open
fix: use URL-based hash as Pinecone vector ID to prevent silent data loss (#174)#176ahmedk20 wants to merge 1 commit into
ahmedk20 wants to merge 1 commit into
Conversation
hash as Pinecone vector ID to prevent silent data loss Articles published on the same date shared the same date-integer ID (from sorting.ordering_date). Upserting to Pinecone with duplicate IDs silently overwrote previous records — only the last article per date survived in the index. Replace the date-integer with an MD5 hash of the article URL. URLs are already deduplicated before this point (seen_urls set), so each hash is guaranteed unique per run. Repeated runs correctly overwrite the same record rather than creating duplicates. Fixes c2siorg#175
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Replaced the date-integer Pinecone vector ID with an MD5 hash of the
article URL to prevent silent data loss during database updates.
Related Issue
Fixes #174
Motivation and Context
Each article's Pinecone vector ID was set from
article["id"], which isa date-based integer produced by
sorting.ordering_date():Multiple articles published on the same date produce the same integer.
When upserted to Pinecone, records with the same ID overwrite each other —
so only the last article processed for a given date survives. All previous
articles for that date are silently deleted with no warning or error.
Demonstration of the bug:
After the fix:
Why MD5 of the URL is safe here:
seen_urlsset before ID generation,so each URL is unique within a single run
meaning Pinecone correctly updates the existing record rather than
creating a duplicate
How Has This Been Tested?
Verified the uniqueness guarantee with a direct simulation:
Types of changes
Checklist