feat(connectivity): add Connectivity module bound to the stack ledger#492
feat(connectivity): add Connectivity module bound to the stack ledger#492Dav-14 wants to merge 27 commits into
Conversation
Introduce a new GatewayGRPCAPI custom resource that allows modules to expose gRPC services through the gateway, following the same pattern as GatewayHTTPAPI. gRPC routing uses fully-qualified protobuf service names (e.g. formance.ledger.v1.LedgerService) instead of HTTP path prefixes. Traffic is served on the same port (8080) with h2c protocol enabled conditionally when gRPC APIs are registered. Each GatewayGRPCAPI creates a dedicated <name>-grpc Kubernetes Service pointing to the module's gRPC port, avoiding conflicts with existing HTTP services. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The deletion step was timing out on K8s 1.34 because the script timeout (2m) was being consumed by kubectl wait --for=delete. Increase the script timeout to 5m and the wait to 3m for more headroom. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract the operator image build from Earthly into a standard Dockerfile so it can be used by both Earthly (via FROM DOCKERFILE) and the new Pulumi deployment app. The Pulumi app (deployment/operator/) builds the operator image, applies CRDs, and deploys the operator via its Helm chart. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The Helm release creates the namespace itself via createNamespace. Defaults to formance-system. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The Docker build context may not include .git, causing go build to fail with "error obtaining VCS status". Disable VCS stamping since version info is already passed via ldflags. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
`ledgerV3TLSName` returned `<stack>-tls`, the same Secret name the gateway ingress already uses for its own TLS certificate (`internal/resources/gateways/ingress.go` names it `<gateway>-tls`, and the gateway is named after the stack). When gateway ingress TLS is enabled, this duplicates an existing mechanism: two independent cert-manager clients target the *same* Secret — - the ledger v3 `Certificate` (self-signed, per-stack Issuer), and - the gateway ingress (ingress-shim, public ClusterIssuer) Each observes the other's issuance as `IncorrectIssuer` and reissues, producing an unbounded reissue race: the keypair rotates every few seconds, the ledger raft mTLS cert reloader repeatedly fails with `tls: private key does not match public key`, and the raft cluster never forms quorum (stuck Degraded). Rename the raft TLS Secret to `<stack>-ledger-v3-tls` so it no longer collides with the gateway ingress Secret. The gateway consumes this backend certificate by the `GatewayBackendTLSSecretLabel` label (not by name), so a dedicated name is safe. `ledgerV3TLSName` is the single source used by the Certificate/secretName, the ledger pod mount, the gateway backend ref and the preview cleanup, so the rename propagates consistently. This also matches the name already expected in `v3_spec_test.go` (`stack0-ledger-v3-tls`). Co-authored-by: David Ragot <david@formance.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. 🗂️ Base branches to auto review (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
🛑 Changes requested — automated reviewThe prior review cycle resolved the two previously identified major issues (missing CRD in the kustomize resources list and incorrect ledger version resolution for versionsFromFile stacks) as well as the minor version-resolution concern. One new concrete major issue has been identified in this cycle: the Connectivity reconciler does not register a watch on the ledger Credentials resource it creates, so if the reconciler returns early while waiting for Credentials readiness (phase=Ready), no event will trigger a requeue and the reconciliation can stall indefinitely until an unrelated watched object changes. This needs to be addressed before merging. |
NumaryBot
left a comment
NumaryBot
left a comment
There was a problem hiding this comment.
NumaryBot posted 2 new inline findings.
Summary: #492 (comment)
NumaryBot
left a comment
NumaryBot
left a comment
There was a problem hiding this comment.
NumaryBot review complete: no remaining inline findings.
Resolved 2 stale NumaryBot review threads (1 fixed, 1 outdated).
Summary: #492 (comment)
Introduce a new `formance.com/v1beta1 Connectivity` stack module, mirroring the Ledger v3 delegation pattern. The module does not run the workload itself. It: - detects, at controller start-up, whether the connectivity operator (`connectivity.formance.com` Connectivity CRD) is installed and reachable with the required RBAC — the same capability + API-group probe the Ledger v3 module uses (CRD served-version check + SelfSubjectAccessReview per verb). When absent, the module reports the capability as unavailable and stays pending instead of failing the controller. - gates on the stack's ledger being v3 and ready (connectivity ingests into the Ledger v3 gRPC endpoint). - provisions a `connectivity.formance.com/v1alpha1 Connectivity` resource bound to that ledger: `ledgerAddress` = the ledger v3 gRPC service and `ledgerTLS` = the ledger backend TLS secret. The connection details are taken from `ledgers.V3GRPCBackendRef`, the single source of truth already used to reach the ledger over gRPC, so connectivity and the gateway stay in sync. - reflects the delegated resource's readiness back onto the module status. Includes the module type, reconciler + capability detection, unit tests for the capability-gating paths, and the generated CRD/RBAC/deepcopy + helm CRD.
Address review feedback and the Dirty check: - Resolve the ledger version with core.ResolveModuleVersion so the v3 gate also works for stacks using spec.versionsFromFile (previously the version fell back to empty and Connectivity stayed stuck on LedgerNotV3). - Register bases/formance.com_connectivities.yaml in config/crd/kustomization so non-Helm (kustomize) installs create the Connectivity CRD. - Regenerate CRD reference docs + helm CRD (just pre-commit).
f5bf061 to
1f6c39e
Compare
NumaryBot
left a comment
NumaryBot
left a comment
There was a problem hiding this comment.
NumaryBot posted 1 new inline finding.
Summary: #492 (comment)
Document the current end-to-end process for adding a stack module: the module CR type (incl. the mandatory formance.com/kind=module label), the reconciler + init registration, all.go + config/crd/kustomization registration, capability detection for delegating modules, version resolution, codegen, and the deployment gotchas (reconcileStrategy: Revision, the startup-only capability probe, and versionsFromFile requirements).
NumaryBot
left a comment
NumaryBot
left a comment
There was a problem hiding this comment.
NumaryBot posted 1 new inline finding.
Summary: #492 (comment)
The delegated connectivity.formance.com Connectivity was created without spec.image, so the connectivity operator fell back to its built-in ghcr.io/formancehq/connectivity-core:latest default — bypassing the stack's registry rewrite (e.g. ghcr.io -> registry.v2.formance.dev) and pull secrets, which makes it unpullable on rewritten registries. Resolve the connectivity-core image via registries.GetFormanceImage (using the Connectivity module version) so it honours the stack registry settings, and set spec.image + spec.imagePullSecrets on the delegated resource.
NumaryBot
left a comment
NumaryBot
left a comment
There was a problem hiding this comment.
NumaryBot review complete: no remaining inline findings.
Resolved 1 stale NumaryBot review thread (1 fixed, 0 outdated).
Summary: #492 (comment)
- Always enable the connectivity-api companion on the delegated Connectivity (spec.api.enabled=true), resolving the connectivity-api image through the registry translation so it honours the stack's registry rewrite + pull secrets (not the connectivity operator's ghcr.io/...:latest default). - Register a GatewayHTTPAPI for the module routing /api/connectivity to the connectivity-api Service (<stack>-api:8080) the connectivity operator provisions, and own it so changes reconcile.
…e auth The connectivity module deployed the delegated Connectivity CR but never wired connectivity-core's authentication to the stack's Ledger v3 gRPC endpoint, so connectivity-core sent no token and the ledger rejected every call with 'requires scope ledger:LedgerWrite'. Provision a god-mode ledger.formance.com/Credentials (cluster-scoped, owned by the Stack) selecting the stack's ledger Cluster: the ledger operator generates the Ed25519 keypair, registers the public key on the ledger, and distributes the private seed as a Secret in the stack namespace. Once Ready, wire the Connectivity CR's spec.auth (keyId + secretKeyRef->seed.hex) so the connectivity operator passes --auth-key-id/--auth-key-file and connectivity-core signs its gRPC tokens with the registered key. Regenerates RBAC for the new resource.
NumaryBot
left a comment
NumaryBot
left a comment
There was a problem hiding this comment.
NumaryBot posted 1 new inline finding.
Summary: #492 (comment)
| setCondition(connectivity, metav1.ConditionFalse, "LedgerCredentialsFailed", err.Error()) | ||
| return err | ||
| } | ||
| if !credReady { |
There was a problem hiding this comment.
🟠 [major] No watch on ledger Credentials — reconciler can stall indefinitely
When the Connectivity reconciler creates the ledger Credentials object and then returns a pending/application error waiting for status.phase=Ready, it will not be requeued automatically because the Credentials resource is not registered as a watched object. The reconciler only watches GatewayHTTPAPI, the delegated Connectivity, settings, and Ledger. On a normal install the controller can remain stuck in LedgerCredentialsPending until some unrelated watched object triggers a reconcile.
Suggestion: Add a Watch for the ledger Credentials resource in the controller setup so that a status update on that object (e.g. phase→Ready) triggers a reconcile of the owning Connectivity. Alternatively, use a periodic requeue (e.g. ctrl.Result{RequeueAfter: ...}) whenever the reconciler returns while waiting for Credentials readiness.
4 participants
What
Adds a new
formance.com/v1beta1 Connectivitystack module that binds connectivity to the stack's ledger, mirroring the Ledger v3 delegation pattern.The module doesn't run the workload itself — it delegates to the connectivity operator (
connectivity.formance.com) and reflects its readiness, exactly like theLedgermodule delegates to the ledger operator (ledger.formance.com Cluster).Behaviour
Capability + API-group detection (same mechanism as Ledger v3): at controller start-up it lists CRDs, checks the
connectivity.formance.com/ConnectivityCRD is present with a served version, and runs aSelfSubjectAccessReviewfor each required verb. If the group/CRD/RBAC is missing it reports the capability as unavailable and stays pending — it never fails controller setup.Gates on Ledger v3: connectivity ingests into the Ledger v3 gRPC endpoint, so it only provisions once the stack's
Ledgermodule is v3 and ready.Binds to the stack ledger: provisions a
connectivity.formance.com/v1alpha1 ConnectivitywithledgerAddress= the ledger v3 gRPC service, andledgerTLS= the ledger backend TLS secret (CA + SNI).The connection details come from
ledgers.V3GRPCBackendRef— the single source of truth already used to reach the ledger over gRPC — so connectivity and the gateway stay in sync (no duplicated address/secret naming).Reflects the delegated resource's
status.phaseback onto the module's Ready condition.Files
api/formance.com/v1beta1/connectivity_types.go— new module CR (Connectivity, labelledformance.com/kind=module).internal/resources/connectivities/init.go— reconciler, capability detection, ledger-v3 gate, delegated-resource bind.internal/resources/connectivities/init_test.go— unit tests for the capability-gating paths (discovery failure, inaccessible resource, missing RBAC, capability-unavailable reconcile).internal/resources/ledgers/exports.go— exportsIsV3+V3GRPCBackendRefso the ledger v3 gRPC connection stays the single source of truth.internal/resources/all.go— registers the module.Test
go build ./...✅go vet ./internal/resources/connectivities/...✅go test ./internal/resources/connectivities/...✅helm template ./helm/crdsrenders ✅Follow-ups (out of scope)