fix(helm-chart/infisical-standalone-postgres): remove updatedAt: {{ now }} from Deployment + pod template metadata#6088
Open
lifeofgurpreet wants to merge 1 commit intoInfisical:mainfrom
Conversation
…mplate metadata
The infisical-standalone chart (both the legacy `infisical` chart and the
current `infisical-standalone-postgres` chart) renders a fresh timestamp
into two annotation fields on every `helm template`:
metadata:
annotations:
updatedAt: {{ now | date "2006-01-01 MST 15:04:05" | quote }} # line 7
spec:
template:
metadata:
annotations:
updatedAt: {{ now | date "2006-01-01 MST 15:04:05" | quote }} # line 23
Because the second annotation is on `spec.template.metadata`, it's hashed
into the Deployment's `pod-template-hash` label. Every Helm re-render
produces a different timestamp, a different hash, and a new ReplicaSet.
Under any GitOps controller that reconciles on a schedule (ArgoCD, Flux,
Helmfile + cron), this creates an endless rolling-update loop with no
actual change.
## Observed impact (production)
- Chart version 1.8.0
- Deployment revision advanced from 52 to 68 in 17 days on a single-
replica prod deployment
- 11+ ReplicaSets created in 8 hours on 2026-04-17
- User-visible latency during the ~30s readiness gap of every rotation
(Node.js bootstrap + the chart-hardcoded `readinessProbe` with the
Kubernetes-default 1s `timeoutSeconds`)
- Readiness probe flapped because the aggressive 1s timeout + CPU
throttling during rollout caused frequent probe timeouts, making the
rollout even more disruptive
## Format-string note
The reference constant in the chart template is `"2006-01-01 MST 15:04:05"`.
Go's canonical time format uses `"2006-01-02 15:04:05"` — the second `01`
in the current format was almost certainly meant to be `02` (day). It
happens to collide with the month placeholder, so both dash-separated
positions substitute to the current month, yielding strings like
`"2026-04-04 UTC ..."` rather than `"2026-04-17 UTC ..."`. Not the cause
of the churn, but worth noting: this specific format string is
ambiguously broken regardless of the churn issue.
## Fix
Remove both `updatedAt` lines entirely. The chart still supports user-
supplied annotations via `deploymentAnnotations` and `podAnnotations`
values — those keep working. Users who truly need a timestamp can set
it via those value paths (it will still churn on re-render, but that's
now an explicit opt-in, not a chart default).
Alternative considered: gate the line behind a value like
`emitUpdatedAtAnnotation: false`. Rejected in favour of outright removal
because (a) there is no observable operational value — the Deployment's
`.status` and `kubectl rollout history` already track revision history,
and (b) leaving it behind a gate preserves the footgun in the repo.
## Validation
$ cd helm-charts/infisical-standalone-postgres
$ helm dependency build
$ helm template t . > /tmp/r1.yaml
$ sleep 2 && helm template t . > /tmp/r2.yaml
$ diff /tmp/r1.yaml /tmp/r2.yaml
141c141
< postgres-password: "emQ3bXRZeGd6Ug==" # Bitnami PG subchart secret
---
> postgres-password: "ZE9KMjUxbEZKUg=="
(only the Bitnami PG subchart's random password differs between
renders, as expected — that field is not on any pod-hash-sensitive
surface. The Deployment's pod template is now byte-identical between
renders.)
## Related downstream work
The operators who discovered this (bbi-infrastructure) shipped:
- ArgoCD ignoreDifferences on the two annotation paths with
RespectIgnoreDifferences=true + ServerSideApply=true (PR Infisical#3170)
- HA fix: replicas=2 + topologySpreadConstraints + PDB selector fix
(PR Infisical#3173)
- A CI guard that detects this class of bug in any chart referenced
by their GitOps repo (PR Infisical#3179)
All of those can be simplified or removed once this fix lands and their
Application pins a chart version that includes it.
## Upstream issue
Infisical#6087
Collaborator
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #6087
Summary
helm-charts/infisical-standalone-postgres/templates/infisical.yamlrenders a fresh timestamp into two annotation fields on every render:The second annotation is on
spec.template.metadata, which Kubernetes hashes into the Deployment'spod-template-hashlabel. Every Helm re-render produces a different timestamp → different hash → new ReplicaSet. Under any GitOps controller that reconciles on a schedule (ArgoCD, Flux, Helmfile + cron), this creates an endless rolling-update loop.Observed impact (production, 2026-04-17)
Format-string note (unrelated, but worth fixing at the same time)
The chart uses
"2006-01-01 MST 15:04:05". Go's canonical reference is"2006-01-02 15:04:05"— the second01was likely meant to be02(day). It happens to collide with the month placeholder, so both dash-separated positions substitute to the current month, yielding strings like"2026-04-04 UTC ..."rather than"2026-04-17 UTC ...". This line is ambiguously broken regardless of the churn issue.Fix
Remove both
updatedAtlines entirely. The chart still supports user-supplied annotations viadeploymentAnnotationsandpodAnnotationsvalues — those keep working. Users who genuinely need a render timestamp can set one via those value paths (it will still churn on re-render, but that's now an explicit opt-in, not a chart default).Alternative considered: gate the line behind a value like
emitUpdatedAtAnnotation: false. Rejected in favour of outright removal because there is no observable operational value (the Deployment's.status+kubectl rollout historyalready track revision history) and leaving it behind a gate preserves the footgun for anyone who flips it on later.Validation
Only the Bitnami PostgreSQL subchart's random password differs between renders (that field is not on any pod-hash-sensitive surface). The Infisical Deployment's pod template is now byte-identical between renders.
Related downstream work
Operators who discovered this (
bbi-infrastructure) shipped:ignoreDifferenceson the two annotation paths withRespectIgnoreDifferences=true+ServerSideApply=trueAll of those become unnecessary once this upstream fix lands and users pin to a chart version that includes it.
Checklist
updatedAttemplating in pod template causes continuous rolling updates under GitOps controllers #6087)helm dependency build+helm templatesucceeddeploymentAnnotations/podAnnotationsstill respectedversion:bump left for maintainer preferenceHappy to iterate on the commit message, format, or approach.