Skip to content

fix(helm-chart/infisical-standalone-postgres): remove updatedAt: {{ now }} from Deployment + pod template metadata#6088

Open
lifeofgurpreet wants to merge 1 commit intoInfisical:mainfrom
lifeofgurpreet:fix/remove-updatedAt-annotation-from-deployment
Open

fix(helm-chart/infisical-standalone-postgres): remove updatedAt: {{ now }} from Deployment + pod template metadata#6088
lifeofgurpreet wants to merge 1 commit intoInfisical:mainfrom
lifeofgurpreet:fix/remove-updatedAt-annotation-from-deployment

Conversation

@lifeofgurpreet
Copy link
Copy Markdown

Fixes #6087

Summary

helm-charts/infisical-standalone-postgres/templates/infisical.yaml renders a fresh timestamp into two annotation fields on every render:

# line 7 — Deployment metadata
updatedAt: {{ now | date "2006-01-01 MST 15:04:05" | quote }}
# line 23 — pod template metadata
updatedAt: {{ now | date "2006-01-01 MST 15:04:05" | quote }}

The second annotation is on spec.template.metadata, which Kubernetes hashes into the Deployment's pod-template-hash label. Every Helm re-render produces a different timestamp → different hash → new ReplicaSet. Under any GitOps controller that reconciles on a schedule (ArgoCD, Flux, Helmfile + cron), this creates an endless rolling-update loop.

Observed impact (production, 2026-04-17)

  • Chart version 1.8.0, single-replica Deployment
  • Revision advanced from 52 to 68 in 17 days (eleven-plus ReplicaSets in an 8-hour window)
  • User-visible latency during the readiness gap of every rotation (Node.js bootstrap + the chart's readinessProbe using the Kubernetes-default 1s timeoutSeconds)

Format-string note (unrelated, but worth fixing at the same time)

The chart uses "2006-01-01 MST 15:04:05". Go's canonical reference is "2006-01-02 15:04:05" — the second 01 was likely meant to be 02 (day). It happens to collide with the month placeholder, so both dash-separated positions substitute to the current month, yielding strings like "2026-04-04 UTC ..." rather than "2026-04-17 UTC ...". This line is ambiguously broken regardless of the churn issue.

Fix

Remove both updatedAt lines entirely. The chart still supports user-supplied annotations via deploymentAnnotations and podAnnotations values — those keep working. Users who genuinely need a render timestamp can set one via those value paths (it will still churn on re-render, but that's now an explicit opt-in, not a chart default).

Alternative considered: gate the line behind a value like emitUpdatedAtAnnotation: false. Rejected in favour of outright removal because there is no observable operational value (the Deployment's .status + kubectl rollout history already track revision history) and leaving it behind a gate preserves the footgun for anyone who flips it on later.

Validation

$ cd helm-charts/infisical-standalone-postgres
$ helm dependency build
$ helm template t . > /tmp/r1.yaml
$ sleep 2 && helm template t . > /tmp/r2.yaml
$ diff /tmp/r1.yaml /tmp/r2.yaml
141c141
<   postgres-password: "emQ3bXRZeGd6Ug=="
---
>   postgres-password: "ZE9KMjUxbEZKUg=="

Only the Bitnami PostgreSQL subchart's random password differs between renders (that field is not on any pod-hash-sensitive surface). The Infisical Deployment's pod template is now byte-identical between renders.

Related downstream work

Operators who discovered this (bbi-infrastructure) shipped:

  • ArgoCD ignoreDifferences on the two annotation paths with RespectIgnoreDifferences=true + ServerSideApply=true
  • HA fix: replicas=2 + topologySpreadConstraints + PDB selector fix
  • CI guard that detects this class of bug in any chart referenced by the repo

All of those become unnecessary once this upstream fix lands and users pin to a chart version that includes it.

Checklist

Happy to iterate on the commit message, format, or approach.

…mplate metadata

The infisical-standalone chart (both the legacy `infisical` chart and the
current `infisical-standalone-postgres` chart) renders a fresh timestamp
into two annotation fields on every `helm template`:

  metadata:
    annotations:
      updatedAt: {{ now | date "2006-01-01 MST 15:04:05" | quote }}  # line 7
  spec:
    template:
      metadata:
        annotations:
          updatedAt: {{ now | date "2006-01-01 MST 15:04:05" | quote }}  # line 23

Because the second annotation is on `spec.template.metadata`, it's hashed
into the Deployment's `pod-template-hash` label. Every Helm re-render
produces a different timestamp, a different hash, and a new ReplicaSet.

Under any GitOps controller that reconciles on a schedule (ArgoCD, Flux,
Helmfile + cron), this creates an endless rolling-update loop with no
actual change.

## Observed impact (production)

- Chart version 1.8.0
- Deployment revision advanced from 52 to 68 in 17 days on a single-
  replica prod deployment
- 11+ ReplicaSets created in 8 hours on 2026-04-17
- User-visible latency during the ~30s readiness gap of every rotation
  (Node.js bootstrap + the chart-hardcoded `readinessProbe` with the
  Kubernetes-default 1s `timeoutSeconds`)
- Readiness probe flapped because the aggressive 1s timeout + CPU
  throttling during rollout caused frequent probe timeouts, making the
  rollout even more disruptive

## Format-string note

The reference constant in the chart template is `"2006-01-01 MST 15:04:05"`.
Go's canonical time format uses `"2006-01-02 15:04:05"` — the second `01`
in the current format was almost certainly meant to be `02` (day). It
happens to collide with the month placeholder, so both dash-separated
positions substitute to the current month, yielding strings like
`"2026-04-04 UTC ..."` rather than `"2026-04-17 UTC ..."`. Not the cause
of the churn, but worth noting: this specific format string is
ambiguously broken regardless of the churn issue.

## Fix

Remove both `updatedAt` lines entirely. The chart still supports user-
supplied annotations via `deploymentAnnotations` and `podAnnotations`
values — those keep working. Users who truly need a timestamp can set
it via those value paths (it will still churn on re-render, but that's
now an explicit opt-in, not a chart default).

Alternative considered: gate the line behind a value like
`emitUpdatedAtAnnotation: false`. Rejected in favour of outright removal
because (a) there is no observable operational value — the Deployment's
`.status` and `kubectl rollout history` already track revision history,
and (b) leaving it behind a gate preserves the footgun in the repo.

## Validation

    $ cd helm-charts/infisical-standalone-postgres
    $ helm dependency build
    $ helm template t . > /tmp/r1.yaml
    $ sleep 2 && helm template t . > /tmp/r2.yaml
    $ diff /tmp/r1.yaml /tmp/r2.yaml
    141c141
    <   postgres-password: "emQ3bXRZeGd6Ug=="  # Bitnami PG subchart secret
    ---
    >   postgres-password: "ZE9KMjUxbEZKUg=="
    (only the Bitnami PG subchart's random password differs between
    renders, as expected — that field is not on any pod-hash-sensitive
    surface. The Deployment's pod template is now byte-identical between
    renders.)

## Related downstream work

The operators who discovered this (bbi-infrastructure) shipped:
- ArgoCD ignoreDifferences on the two annotation paths with
  RespectIgnoreDifferences=true + ServerSideApply=true (PR Infisical#3170)
- HA fix: replicas=2 + topologySpreadConstraints + PDB selector fix
  (PR Infisical#3173)
- A CI guard that detects this class of bug in any chart referenced
  by their GitOps repo (PR Infisical#3179)

All of those can be simplified or removed once this fix lands and their
Application pins a chart version that includes it.

## Upstream issue

Infisical#6087
Copy link
Copy Markdown

@claude claude bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@maidul98
Copy link
Copy Markdown
Collaborator

Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

helm chart: updatedAt templating in pod template causes continuous rolling updates under GitOps controllers

2 participants