Skip to content

ETL-782: Introduce PGBouncer for PostgreSQL connection pooling - #69

Merged
sudopower merged 2 commits into
release-516from
kiran/etl-782-introduce-pg-bouncer-in-helm-chart
Apr 14, 2026
Merged

sudopower merged 2 commits into
release-516from
kiran/etl-782-introduce-pg-bouncer-in-helm-chart

Conversation

@sudopower

Copy link
Copy Markdown
Member

Summary

Introduces PGBouncer as an optional connection pooler for the internal PostgreSQL deployment. Addresses connection exhaustion when many pipeline component replicas each maintain their own connection pool to Postgres (particularly relevant with schema evolution, where every ingestor/sink connects to Postgres to fetch schemas).

  • New chart charts/pgbouncer/ (bitnami/pgbouncer, v0.1.0) — transaction pooling by default, fully configurable
  • postgresql chart bumped to v0.1.9 — when global.pgbouncer.enabled=true, connection-url in the glassflow-postgresql secret points to PGBouncer; original Postgres URL stored in direct-connection-url
  • glassflow-etl chart — adds pgbouncer subchart dependency and pgbouncer: config section; DB migration init container automatically uses direct-connection-url (bypasses PGBouncer) when enabled
  • release.yaml — CI pre-packages local subcharts before helm dependency update so new subchart versions resolve in the same PR as the parent chart

How it works

PGBouncer is transparent to all components:

  • Operator and pipeline components read glassflow-postgresql/connection-url as before — it now points to PGBouncer when enabled
  • Migrations connect directly to Postgres via direct-connection-url
  • No changes to operator chart, component images, or schema evolution code

Usage

# Enable PGBouncer (only supported with internal PostgreSQL)
helm upgrade glassflow glassflow-etl/ --set global.pgbouncer.enabled=true

# Tune pool settings
helm upgrade glassflow glassflow-etl/ \
  --set global.pgbouncer.enabled=true \
  --set pgbouncer.defaultPoolSize=30 \
  --set pgbouncer.maxClientConn=200

For external PostgreSQL users: manage your own connection pooler and point global.postgres.connection_url to it directly.

Backward compatibility

global.pgbouncer.enabled defaults to false. Zero impact on existing deployments.

Closes ETL-782

The pre-install/pre-upgrade hook was unconditionally generating a new
UUID on every helm upgrade, overwriting the existing installation ID in
the secret and sending a spurious helm_install event. The fix reads the
existing secret upfront so the ID is reused on upgrades and only
generated fresh on first install.
@sudopower
sudopower merged commit e5765f2 into release-516 Apr 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant