Skip to content

fix: move module-level scraping and config inside update_database() (#173) - #174

Open
ahmedk20 wants to merge 1 commit into
c2siorg:mainfrom
ahmedk20:fix/update-module-level-scraping
Open

fix: move module-level scraping and config inside update_database() (#173)#174
ahmedk20 wants to merge 1 commit into
c2siorg:mainfrom
ahmedk20:fix/update-module-level-scraping

Conversation

@ahmedk20

Copy link
Copy Markdown

Description

Moved all news collection and Pinecone client initialisation out of module
scope and into update_database(), so scraping only runs when the function
is explicitly called — not on import.

Related Issue

Fixes #173

Motivation and Context

Lines 9–26 of db_update/Update.py were executing at module level:

news = CyberNews()
newsBox["general_news"]       = news.get_news("general")
newsBox["cyber_attack_news"]  = news.get_news("cyberAttack")
newsBox["vulnerability_news"] = news.get_news("vulnerability")
newsBox["malware_news"]       = news.get_news("malware")
newsBox["security_news"]      = news.get_news("security")
newsBox["data_breach_news"]   = news.get_news("dataBreach")

This caused three problems:

1. Import side-effect
Any script, tool, or test that imported this module immediately triggered
full web scraping across all 6 categories — even if update_database()
was never called.

2. Import failure
If any single news source timed out or raised an exception during the
module-level scrape, the entire import failed with an unhandled exception,
making the function completely unreachable.

3. Untestable
Unit tests could not import the module without hitting live external URLs,
making isolated testing impossible.

How Has This Been Tested?

Verified that importing the module no longer triggers any scraping:

>>> import db_update.Update        # completes instantly
Import done — no scraping triggered.
update_database function exists: True

Before this fix, the same import would block for 30–120 seconds
while scraping all news sources.

update_database() continues to work correctly when called explicitly:

python db_update/Update.py
python db_update/Update.py --overwrite

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

      move module-level scraping and config inside update_database()

      All news collection (CyberNews, newsBox) and Pinecone client setup
      were executing at import time, triggering full web scraping whenever
      the module was imported. Moved everything inside update_database() so
      scraping only runs when the function is explicitly called.

      Fixes c2siorg#173
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] db_update/Update.py runs full web scraping at module import time

1 participant