fix(certificatee): remove legacy duplicate certificate files from the sync loop - #42
Merged
Merged
Conversation
… sync loop Every wildcard domain we checked on canary carries two HAProxy storage files for the same SAN: a pre-migration underscore-sanitized name (__devnet_rpcpool_com.pem) alongside the current dot-preserving name (_.devnet.rpcpool.com.pem). Both get synced independently every cycle, and the leftover legacy file appears to be what makes runtime cert pushes to its sibling take a few extra cycles to converge. Add a cleanup pass to processHAProxyEndpoint that removes the legacy duplicate once its current-format sibling has synced cleanly this cycle (no error, no skip, no fresh runtime push), and only when the group resolves unambiguously to exactly one keeper. Deletion always passes skip_reload=true so it never forces an HAProxy reload. Also fixes DeleteCertificate: it was calling a configuration/version endpoint and passing a version query param that this Data Plane API endpoint doesn't accept, and it never set skip_reload, so calling it as-is would have forced a reload on every cleanup.
Code Coverage ReportTotal Coverage: Coverage by function |
4 tasks
ananthb
added a commit
that referenced
this pull request
Aug 13, 2026
…oop (#43) v0.5.5 (#42) deleted a legacy duplicate's storage file every cycle, but HAProxy's runtime cert listing never drops it without a reload, so ListCertificateRefs kept rediscovering the same stale ref and the ordinary per-ref sync recreated it via EnsureStorageCertificate's 404-fallback-to-create. Confirmed directly against a real Data Plane API (shared_lb_fra_staging / haproxy-fra1): storage delete succeeds and the file is gone from disk, but the runtime listing keeps reporting the old content until an unrelated reload happens; a runtime-level delete is not an option either, since HAProxy refuses to remove a certificate still referenced by a bind (500 "in use"). Restructure processHAProxyEndpoint into three phases: fetch every ref's live metadata first, classify legacy duplicates from that complete picture, then sync everything that isn't a confirmed duplicate. Confirmed duplicates are deleted once (never recreated, since they no longer go through the normal per-ref sync at all) as long as their keeper stays stable; if the keeper isn't stable yet, the duplicate falls back to being synced normally so nothing goes unrenewed while cleanup waits for a calmer cycle.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AI description, click for details
Summary
While checking on the SAN-match (#38) and storage-persist (#37) fixes in staging/canary, found that HAProxy's
certs/directory carries two files per wildcard domain: a pre-migration underscore-sanitized name (__devnet_rpcpool_com.pem) alongside the current dot-preserving name (_.devnet.rpcpool.com.pem), both with the identical live SAN. Confirmed via the canary-AP dataplane API directly: all 19 wildcard domains on that node have this exact duplicate pair, both marked"status": "Used".certificatee already discovers and syncs both names independently (correctly — it can't assume the legacy one is dead just because a newer one exists). This duplication is the most likely explanation for a runtime-cert-push convergence delay observed on canary (
*.devnet.rpcpool.comneeded 3 identical-mismatch retries over ~1hr before HAProxy reflected the pushed serial).processHAProxyEndpointthat removes the legacy duplicate once its current-format sibling has synced cleanly this cycle (no error, no skip, no fresh runtime push) — never in the same cycle we just changed something.DeleteCertificatenow passesskip_reload=true, so cleanup never forces an HAProxy reload. It also drops aconfiguration/versionround-trip andversionquery param this Data Plane API endpoint doesn't accept for this route (confirmed against the API's own OpenAPI spec) — as written before this change, calling it would have forced a reload on every delete.certificatee_legacy_certificates_removed_total/..._removal_failures_totalmetrics for observability, documented in the README metrics table.Cleanup failures are logged/metered, never treated as sync failures — this is best-effort housekeeping, not certificate delivery.
Test plan
go test ./pkg/haproxy/... ./cmd/certificatee/... ./pkg/certmetrics/...— all passgo vet+gofmtclean on changed filesDeleteCertificatesendingskip_reload=truewith noversionparamcerts/directoriesNo Linear ticket — this is a follow-up from manually investigating staging/canary cert-sync health.
🤖 Generated with Claude Code