fix: move module-level scraping and config inside update_database() (#173) - #174
Open
ahmedk20 wants to merge 1 commit into
Open
fix: move module-level scraping and config inside update_database() (#173)#174ahmedk20 wants to merge 1 commit into
ahmedk20 wants to merge 1 commit into
Conversation
move module-level scraping and config inside update_database()
All news collection (CyberNews, newsBox) and Pinecone client setup
were executing at import time, triggering full web scraping whenever
the module was imported. Moved everything inside update_database() so
scraping only runs when the function is explicitly called.
Fixes c2siorg#173
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Moved all news collection and Pinecone client initialisation out of module
scope and into
update_database(), so scraping only runs when the functionis explicitly called — not on import.
Related Issue
Fixes #173
Motivation and Context
Lines 9–26 of
db_update/Update.pywere executing at module level:This caused three problems:
1. Import side-effect
Any script, tool, or test that imported this module immediately triggered
full web scraping across all 6 categories — even if
update_database()was never called.
2. Import failure
If any single news source timed out or raised an exception during the
module-level scrape, the entire import failed with an unhandled exception,
making the function completely unreachable.
3. Untestable
Unit tests could not import the module without hitting live external URLs,
making isolated testing impossible.
How Has This Been Tested?
Verified that importing the module no longer triggers any scraping:
Before this fix, the same import would block for 30–120 seconds
while scraping all news sources.
update_database()continues to work correctly when called explicitly:Types of changes
Checklist