feat(aggregation): materialize topic and peer group list edges in LinkedEdgeGenerator

[clincoln8]

Summary

In Data Commons, MCF ingestion writes 1-to-1 scalar triples (relevantVariable for Topics, member for SVPGs). However, downstream consumers (Mixer's TopicCacheManager and Explore's /api/explore/detect-and-fulfill) require consolidated list predicates (relevantVariableList and memberList) to expand topic hierarchies efficiently without scanning thousands of individual rows.

Changes

1. LinkedEdgeGenerator:
   • Added run_topic_list_edges() which queries raw relevantVariable and member arcs in BigQuery, resolves types via schema-driven typeOf IN ('Topic', 'StatVarPeerGroup') and predicate domain (relevantVariable), and executes STRING_AGG(DISTINCT object_id, ',' ORDER BY object_id).
   • Generates property graph hashed keys (CONCAT(SUBSTR(val, 1, 16), ':', sha256)) and exports raw CSV payloads to Spanner's Node table (Node.value) and relationship links to Edge (Edge.object_id = node_hash), preventing Spanner STRING(1024) column overflows.
   • Materializes with global canonical provenance generated/TopicLists (or dc/base/generated/TopicLists).
   • Converted run_linked_member() and run_topic_list_edges() from fragile ID substring pattern matching (LIKE '%/topic/%') to schema-driven type resolution (typeOf = 'Topic' and predicate domain relevantVariable), natively supporting arbitrary custom taxonomy DCIDs (e.g. UN, WHO, SDG).
2. AggregationOrchestrator:
   • Executes run_topic_list_edges() once globally on the orchestrator thread during postprocessing (in _run_global_calculations()), avoiding thread-pool race conditions and Spanner write-lock contention.
   • Forwarded generate_topic_list_edges CLI argument to OrchestratorConfig.
3. AggregationDeleter:
   • Added dedicated delete_topic_list_edges() method to delete generated/TopicLists only when topic list edge regeneration is active.
   • Restored delete_linked_edges() to strictly per-import deletion (generated/{import_name}) to prevent accidental cross-import data loss.
4. Configuration & Opt-in Toggle:
   • Added generate_topic_list_edges flag (default: False for opt-in safety) across LinkedEdgeConfig, OrchestratorConfig, and main.py CLI (--generate_topic_list_edges).
5. Unit & E2E Tests:
   • Added unit tests covering list edge generation, global orchestrator dispatch, and isolated deleter cleanup in aggregation_test.py, orchestrator_test.py, and deleter_test.py (all 68 unit tests passing).

Testing

• uv run pytest across pipeline/workflow/aggregation-helper (all 68 passed).
• Verified live end-to-end against Cloud Spanner, Mixer, and Explore NL on calinc-dev.

[codacy-production]

Not up to standards ⛔

🔴 Issues 1 high · 1 medium

Alerts:
⚠ 2 issues (≤ 0 issues of at least minor severity)

Results:
2 new issues

Category	Results
ErrorProne	1 high
Complexity	1 medium

View in Codacy

🟢 Metrics 22 complexity

Metric	Results
Complexity	22

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[gemini-code-assist]

Code Review

This pull request adds support for materializing topic list edges (relevantVariableList and memberList) during linked edge generation, controlled by a new generate_topic_list_edges configuration flag. Feedback focuses on maintaining the intended 'opt-in safety' design where this feature is disabled by default; the reviewer recommends changing the fallback defaults in several getattr calls from True to False across linked_edge_generator.py, orchestrator.py, and main.py to prevent the feature from being unexpectedly enabled.

[clincoln8]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces the capability to materialize relevantVariableList on Topics and memberList on SVPGs via a new generate_topic_list_edges configuration option, alongside corresponding updates to the orchestrator, CLI, and tests. It also simplifies the deletion logic in deleter.py by removing the thread pool in favor of sequential execution. Feedback on the changes highlights two critical issues: first, a bug in delete_linked_edges where unconditionally deleting generated/TopicLists leads to accidental global data loss of topic list edges across unrelated imports; second, a performance bottleneck in run_topic_list_edges due to an unfiltered global query on the Edge table instead of scoping the query by import_names.

[clincoln8]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a new global calculation step to materialize relevantVariableList on Topics and memberList on StatVarPeerGroups (SVPGs). It updates the orchestrator, deleter, and linked edge generator to support generating and deleting these topic list edges, and exposes this feature via a new --generate_topic_list_edges CLI flag. Additionally, deletions in deleter.py are simplified to run sequentially. Feedback on the changes points out that the import_names parameter in run_topic_list_edges is unused and should be removed to clean up the method signature.

[clincoln8]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces the ability to generate and materialize consolidated topic lists (relevantVariableList and memberList edges) as a global calculation step, including configuration options, orchestrator integration, and corresponding deletion logic in Spanner. It also refactors AggregationDeleter to run deletions sequentially rather than in parallel. The review feedback suggests expanding the fallback namespace checks for topics and peer groups to support direct prefixes like topic/% and svpg/% for custom Data Commons installations.

[clincoln8]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for materializing consolidated topic list edges (relevantVariableList and memberList) as a global calculation step. It updates the LinkedEdgeGenerator to aggregate and export these list edges to Spanner, adds corresponding deletion logic in AggregationDeleter (while also refactoring the deletion process to run sequentially), and exposes this feature via orchestrator configuration and CLI arguments. The feedback recommends ensuring that the generate_topic_list_edges setting is checked within individual calculation steps in addition to the global orchestrator configuration, preventing configuration settings in YAML files from being silently ignored during both generation and deletion phases.