Skip to content

fix(helm,cli,agents): agent pods run without DNS (CE-21) - #148

Merged
devopssquaddev merged 10 commits into
mainfrom
fix/agents-dns-lockdown
Nov 18, 2026
Merged

devopssquaddev merged 10 commits into
mainfrom
fix/agents-dns-lockdown

Conversation

@devopssquaddev

@devopssquaddev devopssquaddev commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

Merge after #144 and #145 (the tes2 proof ran with both).

Agent pods could send data out through DNS. Every direct connection was blocked, but kube-dns forwards any name upstream, so data written into a query label reached an attacker's nameserver. Proven on demohetzner: a never-seen <random>.example.com came back with example.com's own SOA. The fail-closed matrix has shown egress-dns as GAP since v2.5.1-ce.

CE-21 gives agent pods no DNS: dnsPolicy: None, a nameserver nobody answers (127.0.0.1), hostAliases for the few platform names they call, and no port 53 in their egress policy. Runtimes read /etc/hosts natively. There's no Zelkor resolver and no controller.

How the host entries get there

  • Platform: security.dnsLockdown.enabled (default true) runs a plain per-revision Job (zelkor-mcp image, mcp/ops/agent_hosts.py). It looks up each platform Service's ClusterIP, following ExternalName targets, and writes {release}-agent-hosts, one key per hostname. Its RBAC is namespaced and write access is limited to that ConfigMap. It's not a Helm hook, so installs don't wait on it.
  • Kubernetes ≥ 1.36: a per-release MutatingAdmissionPolicy and binding inject dnsPolicy, dnsConfig and hostAliases from the map into every new agent pod of this platform. There's one JSON patch per entry, because a CEL-built list of objects panics the 1.36.4 API server. A missing map denies the pod.
  • Older clusters: zelkor deploy / deploy -f wait for the map and pass platform.hostAliases.
  • Plain Helm on an older cluster: the agent keeps DNS and is annotated zelkor.io/dns-lockdown: inactive, so FinServe-style and GitOps installs don't break.

Port 53

  • Locked pods carry the label zelkor.io/dns-lockdown: "on".
  • Port 53 is allowed only by agent-dns-egress and aegra-dns-egress, which select pods without that label: older agents, inactive ones, and the platform's own Aegra.
  • A render test checks that every policy allowing port 53 excludes locked pods.

Operations

  • zelkor verify <platform> --hosts checks the host map against the live Services, each pod's entries against the map, and each sidecar HTTP API name against the platform's routes. Unlocked agents show as WARN.
  • The sidecar's /ready fails while its LLM or MCP gateway is unreachable, with one upstream_unreachable ERROR naming the upstream, IP and fix.

Tests

  • About 35 new tests: writer, platform and agent renders, a port-53 guard, CLI, sidecar probe, and S-90.
  • The config-matrix model gains security.dnsLockdown.enabled in the 3-way isolation set.
  • S-90 is mapped to the egress-dns fail-closed row.

Proof on tes2 (kind, Kubernetes 1.36.4, all first-party images built from this branch as :ce21)

  • S-90 PASS on 4 locked agents: rt-pydantic via zelkor deploy, FinServe coder/desk/quant via the admission policy.
  • Regression: platform 1281 passed / 90 skipped; FinServe 34 / 2.
  • Drift: recreating the gateway Service made verify --hosts FAIL and the sidecar go unready. A platform upgrade rewrote the map and deleting the pods restored the entries; verify then reported OK.
  • S-90 also caught locked agents with Valkey access getting port 53 through aegra-egress; that's fixed in this branch.

Proof on demohetzner (k3s 1.35, zelkor deploy fallback; platform upgraded with --reuse-values, only zelkor-mcp on :ce21)

  • The 13 existing older-chart agents stayed Ready and kept DNS.
  • rt-pydantic redeployed → locked; S-90 PASS; verify --hosts OK.
  • A real finserve-alpha run (Zitadel JWT, public front door, LLM + MCP postgres tools) exited 0 with no sidecar ERROR.
  • Found and fixed: --reuse-values carried no security.dnsLockdown (now read nil-safe, defaults on).

Not in this slice: platform Aegra (Mode A graphs), runtime: app pods and sandbox workers.

MINOR release: new chart contract (security.dnsLockdown, platform.hostAliases, admission policy objects).

🤖 Generated with Claude Code

devopssquaddev and others added 10 commits November 18, 2026 15:37
mcp/ops/agent_hosts.py resolves the Services agents call to their final
ClusterIPs (following in-cluster ExternalName targets, skipping outside
and headless ones) and writes one ConfigMap key per hostname, short and
FQDN. It retries until the required gateway names resolve and talks to
the API server with the Job's service-account token (no kubectl image).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…agent egress

security.dnsLockdown.enabled (default true) adds a plain per-revision Job
(zelkor-mcp image, namespaced RBAC scoped to its ConfigMap) that writes
{release}-agent-hosts. On clusters that serve MutatingAdmissionPolicy the
chart also renders a per-release policy and binding: new agent pods of
this platform (zelkor.io/platform-release) get dnsPolicy None, a dead
nameserver and hostAliases from the map; no map denies the pod. The
platform agent-egress policy drops port 53. Agents not yet redeployed
keep DNS through their own chart's rule (policies are additive).

Config matrix: dnsLockdown joins the model and the 3-way isolation set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l them

charts/zelkor-agent: security.dnsLockdown.enabled (default true) and
platform.hostAliases. When zelkor deploy supplies host entries, or the
platform admission policy can inject them (K8s >= 1.36 with
platform.releaseName), the pod gets dnsPolicy None, nameserver
127.0.0.1, its hostAliases and the labels zelkor.io/dns-lockdown=on and
zelkor.io/platform-release; its own egress policy drops port 53. A plain
Helm install on an older cluster keeps DNS and is annotated
zelkor.io/dns-lockdown=inactive for zelkor verify, so GitOps and
FinServe-style installs do not break.

Platform: a separate agent-dns-egress policy allows port 53 only for
agent pods without dns-lockdown=on, so older and inactive agents keep
DNS regardless of their own chart's NetworkPolicy setting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ks them

The platform publishes AGENT_HOSTS_CONFIGMAP in its discovery contract.
deploy and deploy -f wait (ZELKOR_HOST_MAP_WAIT_SEC, 120 s) for that map,
group hostnames per IP and pass them as platform.hostAliases; a platform
without the key is skipped. zelkor verify <platform> --hosts reports map
drift against live Services, pods whose hostAliases differ from the map,
sidecar HTTP API names with no platform route (the stale api-echo case),
and agents still on DNS as WARN.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eachable

CE-21 agent pods resolve names from hostAliases; a recreated Service
leaves a stale entry. The sidecar now TCP-checks its LLM and MCP
upstreams (native resolution, cached 10 s) as part of /ready. On a
transition to unreachable it logs one ERROR upstream_unreachable with
the upstream, host, IP and a hint to run zelkor deploy (or delete the
pod on K8s >= 1.36); recovery logs one INFO upstream_reachable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For every running agent pod with zelkor.io/dns-lockdown=on, the probe runs
in its sidecar: a never-seen outside name and github.com must not
resolve, kube-dns must not answer over UDP or TCP 53, and the gateway
name must still resolve from /etc/hosts (positive control). The
fail-closed matrix maps S-90 to egress-dns, so that row can turn PASS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
S-90 on tes2 (K8s 1.36.4): kube-dns still answered finserve-coder. Agent
pods carry app.kubernetes.io/component=aegra, so a locked agent with
Valkey access also matches aegra-egress, which allowed port 53. With
lockdown on, that rule moves to aegra-dns-egress, which selects only
Aegra-labelled pods without zelkor.io/dns-lockdown=on (the platform Aegra
and unlocked agents). A render test now asserts every policy that
allows port 53 excludes locked pods.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… too

Aegra-runtime agents have no sidecar and no ZELKOR_LLM_UPSTREAM; take
the host from OPENAI_BASE_URL or MCP_URL, and fail clearly if none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The JSON formatter emits only allowlisted context keys, so the sidecar's
upstream_unreachable line on tes2 showed the event but not which
upstream, IP or fix hint. Allow upstream, host, ip, hint, configmap,
hostnames and missing (names and addresses, never credentials); the
sidecar reports the exception class under the existing exc_type key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…des render

helm upgrade --reuse-values from a release that predates CE-21 carries
no security.dnsLockdown (new chart defaults are not merged), and the
templates failed on a nil value (demohetzner 2.5.5 upgrade). Two helpers
read it with lockdown defaulting on and the writer deadline to 900 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@devopssquaddev

Copy link
Copy Markdown
Owner Author

demohetzner proof (k3s 1.35, zelkor deploy fallback): platform upgraded with --reuse-values (needed the nil-safe security.dnsLockdown read, now on the branch); writer Job 11 s; the 13 existing older-chart agents stayed Ready and kept DNS; rt-pydantic redeployed with the CE-21 CLI → locked, S-90 PASS, verify --hosts OK; a real finserve-alpha run through the public front door (LLM + MCP postgres tools) exited 0 with no sidecar ERROR.

Both labs are proven. Ready for review once #144 and #145 merge (tes2 ran with them).

@cloudflare-workers-and-pages

Copy link
Copy Markdown

🚀 Deploying Preview to Cloudflare 🚀

Preview Deployments by commit

Status Deployment URL Commit Updated (UTC) See this deployment's details
  • Build: Success ✅

View logs ↗
No Preview URL. Enable ↗ 0aee0e1 2026-11-18T16:19:16.173Z Visit the dashboard ↗

@devopssquaddev
devopssquaddev marked this pull request as ready for review November 18, 2026 17:57
@devopssquaddev
devopssquaddev merged commit e5631c0 into main Nov 18, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant