Skip to content

feat(extract): add Kubernetes/ArgoCD/Helm-values YAML extractor#2043

Open
stevenpollack wants to merge 1 commit into
Graphify-Labs:v8from
stevenpollack:feat/k8s-extractor
Open

feat(extract): add Kubernetes/ArgoCD/Helm-values YAML extractor#2043
stevenpollack wants to merge 1 commit into
Graphify-Labs:v8from
stevenpollack:feat/k8s-extractor

Conversation

@stevenpollack

Copy link
Copy Markdown

Problem

graphify has no YAML extractor. Kubernetes manifests, ArgoCD GitOps config and Helm values all fall through to the LLM document path — expensive, and what comes back is inferred prose relationships rather than a real deployment graph.

On a Helm + ArgoCD monorepo the practical effect is that the deployment topology is unqueryable. I hit this trying to answer "how does this service reach prod?" — every file holding the answer was invisible to the graph.

What this adds

extract_k8s() parses the YAML that genuinely is YAML and emits:

contains · in_namespace · references · deploys_to · deploys_chart · generates_from · produces

Three decisions worth reviewing:

1. Global <kind>.<name> addressing. extract_terraform scopes IDs by directory because Terraform resources are module-scoped. Kubernetes isn't: an ApplicationSet's project: platform means the AppProject named platform wherever it is declared. Global scoping is what lets those cross-file edges survive the per-file extraction merge. Pinned by test_cross_file_reference_resolves_by_global_address.

2. References matched by key NAME, not fixed spec paths. An ApplicationSet nests an entire Application spec under spec.template.spec. A depth-agnostic recursive walk keyed on field names picks those up with zero ApplicationSet-specific code — that's most of why this is ~290 lines and not 600.

3. Detection by CONTENT, not extension. .yaml stays in DOC_EXTENSIONS; only files carrying both apiVersion: and kind: are rerouted to the AST path. Ordinary YAML (CI configs, docs data, front-matter) keeps going to the LLM path exactly as before. This mirrors the existing is_package_manifest_path() precedent (#1377).

Security note: the sniff runs after _is_sensitive() in detect(), so a secrets.yaml is dropped before it can reach the AST path. I verified that ordering rather than assuming it.

Deliberately out of scope

Helm templates (charts/*/templates/*.yaml). Go template directives make them invalid YAML — measured on a real chart, 0 of 46 template files parse, while 100% of the manifests and values files do. They degrade to an empty result plus an error string rather than raising.

Documented in the module docstring, so the graph doesn't overstate what it knows:

  • A git generator yields generates_from to the glob it scans, not the files that glob matches. Resolving it would mean globbing the filesystem from inside a single-file extractor, breaking per-file caching and determinism.
  • Registry-style YAML with neither apiVersion nor kind (an ArgoCD fleet entry is often just app: <name>) is indistinguishable from arbitrary config, so it stays a document.
  • Same-named objects across clusters/namespaces collapse into one node.

Validation

Real Helm + ArgoCD monorepo, 31 manifests: 0 parse errors, 94-node graph end-to-end through detectextractbuild_from_json.

The generates_from edges alone answer "which services can reach prod at all":

ApplicationSet.fleet-apps-dev             -> path.gitops/apps/*.yaml
ApplicationSet.fleet-apps-dev             -> path.gitops/apps-dev/*.yaml
ApplicationSet.fleet-apps-prod            -> path.gitops/apps/*.yaml
ApplicationSet.fleet-apps-staging         -> path.gitops/apps/*.yaml
ApplicationSet.fleet-apps-under-migration -> path.gitops/apps-under-migration/*.yaml

Two ApplicationSets read different registries — which is exactly why one service deploys to dev but never to prod. That was previously invisible.

Dependency

pyyaml as the optional [k8s] extra, matching the sql / terraform / pascal pattern. Without it, affected files hit the standard missing-extra warning (_EXTRA_FOR_EXTENSION) instead of failing the run.

Tests

14 new tests in tests/test_k8s.py (+ a registry-identity test following the test_<lang>_migrated convention), covering each relation, ApplicationSet nesting, multi-document files, real line numbers, graceful Helm-template failure, and both directions of the detection sniff. Guarded by pytest.importorskip("yaml").

Full suite: 3424 passed. Four failures in test_ollama_retry_cap.py (ModuleNotFoundError: No module named 'openai') reproduce identically on v8 without this change — pre-existing and unrelated.

graphify had no YAML extractor, so k8s manifests, ArgoCD GitOps config and
Helm values all fell to the LLM document path — expensive, and it produced
inferred prose links rather than a real deployment graph. On a Helm+ArgoCD
monorepo that meant the entire deploy topology was effectively unqueryable.

extract_k8s() parses the YAML that is actually valid YAML and emits:
  contains / in_namespace / references / deploys_to / deploys_chart /
  generates_from / produces

Objects are addressed <kind>.<name> GLOBALLY rather than per-file or per-
directory, mirroring how Kubernetes resolves a reference: an ApplicationSet's
'project: platform' means the AppProject named platform wherever it is
declared. That scoping is what lets cross-file edges survive the per-file
merge.

References are matched by key NAME during a recursive walk rather than by
fixed spec paths, so an ApplicationSet's nested spec.template.spec picks up
the same refs as a bare Application with no ApplicationSet-specific code.

Detection is by CONTENT (apiVersion + kind), never by extension: .yaml stays
a DOC extension so ordinary YAML keeps going to the LLM path. The sniff runs
after _is_sensitive() in detect(), so a secrets.yaml is dropped first.
Mirrors the existing is_package_manifest_path() precedent (Graphify-Labs#1377).

Helm templates are out of scope — Go template directives make them invalid
YAML; they degrade to an empty result with an error rather than raising.
pyyaml ships as the optional [k8s] extra, matching sql/terraform/pascal.

Validated on a production Helm+ArgoCD monorepo: 31 manifests, 0 parse
errors, 94-node graph. The generates_from edges alone show which registry
each environment fans out over — i.e. which services can reach prod at all.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant