feat(openbao): add OpenBao role as external KMS for Percona pg_tde

[alemacci]

Summary

This PR introduces a new openbao module/role to Pigsty, adding OpenBao (Apache-2.0 licensed, open-source fork of HashiCorp Vault) as an external Key Management Service (KMS) that backs PostgreSQL Transparent Data Encryption via Percona's pg_tde extension.

With pg_tde, PostgreSQL encrypts heap data and WAL at rest, but the master/principal key must never live on the database host.
This role deploys and manages an OpenBao cluster so that PostgreSQL fetches its principal key over TLS at startup and on key rotation, keeping encrypted data (on PGSQL nodes) and key material (in the KMS) properly separated, which is what most at-rest encryption compliance regimes actually require.

Opening this PR directly given the scope of testing already completed, rather than starting with an issue as the contributing guide suggests. Happy to discuss architecture, naming, or alternatives here if preferred.
The branch can be adjusted based on feedback.

What's included

• roles/openbao - deploy an OpenBao cluster:
  • install the openbao package
  • create config/data/log/TLS/seal directories
  • issue and distribute TLS server certificates via Pigsty's CA
  • generate and distribute a cluster-wide static unseal key
  • render server config and systemd unit
  • launch the service and open firewall ports
  • initialize the cluster, persisting the root token / recovery keys to the admin node
  • provision a pg_tde KV v2 mount, policy, and long-lived token, persisted to the admin node
  • register /v1/sys/metrics as a VictoriaMetrics scrape target
• roles/openbao_remove - clean removal of an OpenBao cluster
• openbao.yml / openbao-rm.yml - top-level playbooks
• roles/pgsql/tasks/tde.yml - installs the pg_tde key-provider token issued by OpenBao on the PGSQL node before Patroni starts
• roles/pgsql/tasks/patroni.yml - a new task sets pg_tde GUCs after running prerequisites, then restarts Patroni
• conf/pgtde.yml - ready-to-use single-node config template wiring a Percona PostgreSQL cluster (with pg_tde preloaded) to a single OpenBao instance
• conf/ha/pgtde.yml - 3-node HA variant: 3-node etcd, 3-node OpenBao cluster, and HAProxy in front of OpenBao, for production-like setups where the KMS itself must not be a SPOF
• roles/ca/tasks/main.yml - adds a dedicated directory to hold OpenBao's TLS material, isolated from other CA consumers
• roles/node_id - registers the openbao package across all supported distro/arch package lists (currently only EL-distros are tested)
• deploy.yml - wires the openbao role into the main deploy playbook
• Grafana dashboard (files/grafana/infra/openbao-overview.json), updates to the shared files/grafana/pigsty.json overview dashboard, and VictoriaMetrics alerting rules for OpenBao cluster monitoring (files/victoria/rules/openbao.yml)

Why

Percona's PostgreSQL distribution ships pg_tde for at-rest encryption, but it requires an external key management backend to hold the principal key. Pigsty currently has no first-class way to stand up and manage such a KMS. OpenBao is a natural fit: it's open source, Apache-2.0 licensed (no HashiCorp BSL concerns), API-compatible with Vault, prometheus-style metrics and fits Pigsty's existing patterns for CA/cert issuance, systemd-managed services, and Ansible-driven provisioning.

Beyond tooling fit, this addresses a gap that TDE alone does not close.

Storing the principal key on the same host as the data it encrypts means a single host compromise defeats the encryption entirely, the key and the ciphertext fail together.

An external, dedicated KMS is what turns "encryption" into an actual security control:

• Separation of duties. Key material is never co-located with encrypted data -> separate host, network path, credentials, and audit trail. Compromising the database node no longer implies compromising the key.
• Availability of the key layer. A single-node key provider is a new SPOF: unreachable KMS means unreadable data even
  though the data itself is intact. The 3-node cluster with quorum-based consensus removes this failure mode.
• Controlled key lifecycle. Seal key material lives outside the OpenBao data path, so data-directory resets don't affect it, and
  cluster rebuilds reuse the existing seal key unless an operator explicitly removes it. Rotation and rebuild are deliberate actions, not incidental side effects of routine maintenance.
• Reproducible, auditable deployment. The whole setup (TLS, file permissions, systemd hardening, key provisioning) is declarative Ansible, reviewable in version control rather than manual host state.

This lines up with how at-rest encryption is actually evaluated under common compliance frameworks: GDPR Art. 32 names encryption as an example technical measure assessed against risk and state of the art, and sector-specific cloud/infrastructure security baselines typically scale encryption and key-management requirements with data sensitivity: external key management with enforced separation of duties is what distinguishes a defensible control from encryption-as-checkbox.

Implementation notes

A few non-obvious sequencing/compatibility decisions worth flagging for review:

GUCs must stay off during bootstrap, then get enabled on-the-fly.
default_table_access_method, pg_tde.enforce_encryption, and pg_tde.wal_encrypt cannot be turned on before the pg_tde extension exists and the key provider chain has run (pg_tde_add_global_key_provider_vault_v2 → pg_tde_create_key_using_global_key_provider → pg_tde_set_default_key_using_global_key_provider).
Enabling them earlier blocks cluster bootstrap outright. The new apply pgtde GUCs and restart task in roles/pgsql/tasks/patroni.yml sets these GUCs after the prerequisites have run, not before.
This ordering is intentional, not an oversight.

pg-init intentionally does not CREATE EXTENSION pg_tde on template1.
Instead, pg_tde is expected to be declared through each database's own extensions: list, and it must be first in that list and mandatory wherever pg_tde is in use. If it's missing or ordered after other extensions, default_table_access_method = tde_heap combined with pg_tde.enforce_encryption = on blocks any CREATE TABLE, including ones issued during database provisioning itself, not just by the hosted application later.

pgBackRest parameters were adjusted for pg_tde compatibility
as Percona's documentation:

archive-async=n
archive-header-check=n
checksum-page=n

Without these, backups are not reliably compatible with the Percona pg_tde kernel.

Tags / usage

The role is fully tag-driven (openbao_install, openbao_config, openbao_cert, openbao_seal, openbao_launch, openbao_init, openbao_provision, openbao_register, ...), consistent with the tag hierarchy used by other Pigsty roles (pgsql, etcd, minio, etc.).

Quickstart (single-node):

curl https://repo.pigsty.io/get | bash
./configure -c pgtde
./deploy.yml

Quickstart (3-node HA, OpenBao and etcd both clustered):

curl https://repo.pigsty.io/get | bash
./configure -c ha/pgtde
./deploy.yml

Testing

• Deployed a 3-nodes OpenBao cluster and verified init/unseal/provision tasks
• Deployed a custom Percona PostgreSQL cluster with pg_tde using the conf/pgtde.yml template and confirmed encrypted tables/WAL
• Verified openbao-rm.yml cleanly removes the cluster
• Verified Grafana dashboard renders OpenBao metrics correctly

Related

• Docs: https://pigsty.io/docs/conf/pgtde
• pg_tde: https://docs.percona.com/pg-tde/
• OpenBao: https://openbao.org/docs/