Skip to content

[FEATURE] Added BigQuery Sink - #24

Open
aditypan wants to merge 4 commits into
glassflow:mainfrom
aditypan:feat/bigquery_sink
Open

aditypan wants to merge 4 commits into
glassflow:mainfrom
aditypan:feat/bigquery_sink

Conversation

@aditypan

@aditypan aditypan commented Mar 25, 2026 •

Copy link
Copy Markdown
Contributor
  • Added a new file bigquery_sink under sinks
  • BigQuerySink inherits BaseSink and excepts parameters like project_id , dataset , table name and gcp credentials file path.
  • BigQuerySink can publish single and bulk records. It validates the configs, checks whether table exists in the provided project or not. If doesn't exist Glassgen helps in creating the table and load the data.
  • Made it available in sink init file
  • Added test case

Example for biquery sink:

config = {
    "schema": {
        "name": "$name",
        "email": "$email",
        "country": "$country",
        "id": "$uuid",
        "emails": "$array(email,2)",
    },
    "generator": {
        "rps": 0,
        "num_records": -1
    }
}

generate(config, sink={
        "project_id": "gcp-project-name",
        "dataset": "dataset_name",
        "table": "table_name",
        "credentials_path": "/fake/path/credentials.json",
    })

@vercel

vercel Bot commented Mar 25, 2026

Copy link
Copy Markdown

@aditypan is attempting to deploy a commit to the GlassFlow Team on Vercel.

A member of the Team first needs to authorize it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants