Repository navigation
fix(helm,cli,agents): agent pods run without DNS (CE-21) - #148
Merged
Merged
Conversation
mcp/ops/agent_hosts.py resolves the Services agents call to their final ClusterIPs (following in-cluster ExternalName targets, skipping outside and headless ones) and writes one ConfigMap key per hostname, short and FQDN. It retries until the required gateway names resolve and talks to the API server with the Job's service-account token (no kubectl image). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…agent egress
security.dnsLockdown.enabled (default true) adds a plain per-revision Job
(zelkor-mcp image, namespaced RBAC scoped to its ConfigMap) that writes
{release}-agent-hosts. On clusters that serve MutatingAdmissionPolicy the
chart also renders a per-release policy and binding: new agent pods of
this platform (zelkor.io/platform-release) get dnsPolicy None, a dead
nameserver and hostAliases from the map; no map denies the pod. The
platform agent-egress policy drops port 53. Agents not yet redeployed
keep DNS through their own chart's rule (policies are additive).
Config matrix: dnsLockdown joins the model and the 3-way isolation set.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l them charts/zelkor-agent: security.dnsLockdown.enabled (default true) and platform.hostAliases. When zelkor deploy supplies host entries, or the platform admission policy can inject them (K8s >= 1.36 with platform.releaseName), the pod gets dnsPolicy None, nameserver 127.0.0.1, its hostAliases and the labels zelkor.io/dns-lockdown=on and zelkor.io/platform-release; its own egress policy drops port 53. A plain Helm install on an older cluster keeps DNS and is annotated zelkor.io/dns-lockdown=inactive for zelkor verify, so GitOps and FinServe-style installs do not break. Platform: a separate agent-dns-egress policy allows port 53 only for agent pods without dns-lockdown=on, so older and inactive agents keep DNS regardless of their own chart's NetworkPolicy setting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ks them The platform publishes AGENT_HOSTS_CONFIGMAP in its discovery contract. deploy and deploy -f wait (ZELKOR_HOST_MAP_WAIT_SEC, 120 s) for that map, group hostnames per IP and pass them as platform.hostAliases; a platform without the key is skipped. zelkor verify <platform> --hosts reports map drift against live Services, pods whose hostAliases differ from the map, sidecar HTTP API names with no platform route (the stale api-echo case), and agents still on DNS as WARN. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eachable CE-21 agent pods resolve names from hostAliases; a recreated Service leaves a stale entry. The sidecar now TCP-checks its LLM and MCP upstreams (native resolution, cached 10 s) as part of /ready. On a transition to unreachable it logs one ERROR upstream_unreachable with the upstream, host, IP and a hint to run zelkor deploy (or delete the pod on K8s >= 1.36); recovery logs one INFO upstream_reachable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For every running agent pod with zelkor.io/dns-lockdown=on, the probe runs in its sidecar: a never-seen outside name and github.com must not resolve, kube-dns must not answer over UDP or TCP 53, and the gateway name must still resolve from /etc/hosts (positive control). The fail-closed matrix maps S-90 to egress-dns, so that row can turn PASS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
S-90 on tes2 (K8s 1.36.4): kube-dns still answered finserve-coder. Agent pods carry app.kubernetes.io/component=aegra, so a locked agent with Valkey access also matches aegra-egress, which allowed port 53. With lockdown on, that rule moves to aegra-dns-egress, which selects only Aegra-labelled pods without zelkor.io/dns-lockdown=on (the platform Aegra and unlocked agents). A render test now asserts every policy that allows port 53 excludes locked pods. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… too Aegra-runtime agents have no sidecar and no ZELKOR_LLM_UPSTREAM; take the host from OPENAI_BASE_URL or MCP_URL, and fail clearly if none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The JSON formatter emits only allowlisted context keys, so the sidecar's upstream_unreachable line on tes2 showed the event but not which upstream, IP or fix hint. Allow upstream, host, ip, hint, configmap, hostnames and missing (names and addresses, never credentials); the sidecar reports the exception class under the existing exc_type key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…des render helm upgrade --reuse-values from a release that predates CE-21 carries no security.dnsLockdown (new chart defaults are not merged), and the templates failed on a nil value (demohetzner 2.5.5 upgrade). Two helpers read it with lockdown defaulting on and the writer deadline to 900 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner
Author
|
demohetzner proof (k3s 1.35, Both labs are proven. Ready for review once #144 and #145 merge (tes2 ran with them). |
🚀 Deploying Preview to Cloudflare 🚀Preview Deployments by commit
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge after #144 and #145 (the tes2 proof ran with both).
Agent pods could send data out through DNS. Every direct connection was blocked, but kube-dns forwards any name upstream, so data written into a query label reached an attacker's nameserver. Proven on demohetzner: a never-seen
<random>.example.comcame back with example.com's own SOA. The fail-closed matrix has shownegress-dnsas GAP sincev2.5.1-ce.CE-21 gives agent pods no DNS:
dnsPolicy: None, a nameserver nobody answers (127.0.0.1),hostAliasesfor the few platform names they call, and no port 53 in their egress policy. Runtimes read/etc/hostsnatively. There's no Zelkor resolver and no controller.How the host entries get there
security.dnsLockdown.enabled(defaulttrue) runs a plain per-revision Job (zelkor-mcpimage,mcp/ops/agent_hosts.py). It looks up each platform Service's ClusterIP, following ExternalName targets, and writes{release}-agent-hosts, one key per hostname. Its RBAC is namespaced and write access is limited to that ConfigMap. It's not a Helm hook, so installs don't wait on it.MutatingAdmissionPolicyand binding injectdnsPolicy,dnsConfigandhostAliasesfrom the map into every new agent pod of this platform. There's one JSON patch per entry, because a CEL-built list of objects panics the 1.36.4 API server. A missing map denies the pod.zelkor deploy/deploy -fwait for the map and passplatform.hostAliases.zelkor.io/dns-lockdown: inactive, so FinServe-style and GitOps installs don't break.Port 53
zelkor.io/dns-lockdown: "on".agent-dns-egressandaegra-dns-egress, which select pods without that label: older agents, inactive ones, and the platform's own Aegra.Operations
zelkor verify <platform> --hostschecks the host map against the live Services, each pod's entries against the map, and each sidecar HTTP API name against the platform's routes. Unlocked agents show asWARN./readyfails while its LLM or MCP gateway is unreachable, with oneupstream_unreachableERROR naming the upstream, IP and fix.Tests
security.dnsLockdown.enabledin the 3-way isolation set.egress-dnsfail-closed row.Proof on tes2 (kind, Kubernetes 1.36.4, all first-party images built from this branch as
:ce21)rt-pydanticviazelkor deploy, FinServecoder/desk/quantvia the admission policy.verify --hostsFAIL and the sidecar go unready. A platform upgrade rewrote the map and deleting the pods restored the entries;verifythen reported OK.aegra-egress; that's fixed in this branch.Proof on demohetzner (k3s 1.35,
zelkor deployfallback; platform upgraded with--reuse-values, onlyzelkor-mcpon:ce21)rt-pydanticredeployed → locked; S-90 PASS;verify --hostsOK.finserve-alpharun (Zitadel JWT, public front door, LLM + MCP postgres tools) exited 0 with no sidecar ERROR.--reuse-valuescarried nosecurity.dnsLockdown(now read nil-safe, defaults on).Not in this slice: platform Aegra (Mode A graphs),
runtime: apppods and sandbox workers.MINOR release: new chart contract (
security.dnsLockdown,platform.hostAliases, admission policy objects).🤖 Generated with Claude Code