Skip to content

[DCP Ingestion] Add preprocessing stage to ingestion helper #670

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
gmechali wants to merge 3 commits into
base: master
Choose a base branch
from
Open

[DCP Ingestion] Add preprocessing stage to ingestion helper #670

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
38e2d5f
Add preproc stage to ingestion workflow ingstion helper
gmechali Jul 20, 2026
55f052d
Merge branch 'master' into preprocStage
gmechali Jul 20, 2026
651eddb
checking in spanner hange for preproc
gmechali Jul 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion pipeline/workflow/ingestion-helper/clients/spanner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ class IngestionState(str, Enum):
class IngestionStage(str, Enum):
DATAFLOW = "dataflow"
POSTPROCESSING = "postprocessing"
PREPROCESSING = "preprocessing"


class SpannerClient:
Expand Down Expand Up @@ -323,7 +324,7 @@ def update_ingestion_history(self,
Args:
workflow_id: The ID of the workflow.
status: The status of the ingestion stage (PENDING, RUNNING, SUCCESS, FAILURE).
stage: The stage of the ingestion (dataflow, postprocessing, etc.).
stage: The stage of the ingestion (preprocessing, dataflow, postprocessing, etc.).
job_id: The Dataflow job ID.
ingested_imports: List of ingested import names.
metrics: A dictionary containing metrics about the ingestion.
Expand Down