Skip to content

fix: use URL-based hash as Pinecone vector ID to prevent silent data loss (#174) - #176

Open
ahmedk20 wants to merge 1 commit into
c2siorg:mainfrom
ahmedk20:fix/unique-pinecone-vector-ids
Open

fix: use URL-based hash as Pinecone vector ID to prevent silent data loss (#174)#176
ahmedk20 wants to merge 1 commit into
c2siorg:mainfrom
ahmedk20:fix/unique-pinecone-vector-ids

Conversation

@ahmedk20

Copy link
Copy Markdown

Description

Replaced the date-integer Pinecone vector ID with an MD5 hash of the
article URL to prevent silent data loss during database updates.

Related Issue

Fixes #174

Motivation and Context

Each article's Pinecone vector ID was set from article["id"], which is
a date-based integer produced by sorting.ordering_date():

# Before — date integer used as vector ID
record = {
    "id": str(article.get("id", "")),  # e.g. "20240315" for March 15 2024
    ...
}

Multiple articles published on the same date produce the same integer.
When upserted to Pinecone, records with the same ID overwrite each other —
so only the last article processed for a given date survives. All previous
articles for that date are silently deleted with no warning or error.

Demonstration of the bug:

Article 1 → id = "20240315"  ← upserted ✅
Article 2 → id = "20240315"  ← overwrites Article 1 ❌
Article 3 → id = "20240315"  ← overwrites Article 2 ❌
Result: only Article 3 in Pinecone. Articles 1 and 2 lost silently.

After the fix:

Article 1 → id = "d53cf5f1e1f9bebaea2221f08c8558b5"  ← upserted ✅
Article 2 → id = "c2fae69474d6ac9b0b8502d83b073462"  ← upserted ✅
Article 3 → id = "0d5764b1d30f389bb719af111c6770a0"  ← upserted ✅
Result: all 3 articles in Pinecone. No data loss.

Why MD5 of the URL is safe here:

  • URLs are already deduplicated by the seen_urls set before ID generation,
    so each URL is unique within a single run
  • Repeated runs (next scrape) produce the same hash for the same URL,
    meaning Pinecone correctly updates the existing record rather than
    creating a duplicate

How Has This Been Tested?

Verified the uniqueness guarantee with a direct simulation:

=== OLD behaviour (date integer) ===
IDs: ['20240315', '20240315', '20240315']
Unique IDs: 1 — only 1 record survives in Pinecone

=== NEW behaviour (URL hash) ===
IDs: ['d53cf5f1...', 'c2fae694...', '0d5764b1...']
Unique IDs: 3 — all 3 records survive in Pinecone

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

   hash as Pinecone vector ID to prevent silent data loss

   Articles published on the same date shared the same date-integer ID
   (from sorting.ordering_date). Upserting to Pinecone with duplicate IDs
   silently overwrote previous records — only the last article per date
   survived in the index.

   Replace the date-integer with an MD5 hash of the article URL. URLs are
   already deduplicated before this point (seen_urls set), so each hash is
   guaranteed unique per run. Repeated runs correctly overwrite the same
   record rather than creating duplicates.

   Fixes c2siorg#175
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant