feat(extract): add Kubernetes/ArgoCD/Helm-values YAML extractor#2043
Open
stevenpollack wants to merge 1 commit into
Open
feat(extract): add Kubernetes/ArgoCD/Helm-values YAML extractor#2043stevenpollack wants to merge 1 commit into
stevenpollack wants to merge 1 commit into
Conversation
graphify had no YAML extractor, so k8s manifests, ArgoCD GitOps config and Helm values all fell to the LLM document path — expensive, and it produced inferred prose links rather than a real deployment graph. On a Helm+ArgoCD monorepo that meant the entire deploy topology was effectively unqueryable. extract_k8s() parses the YAML that is actually valid YAML and emits: contains / in_namespace / references / deploys_to / deploys_chart / generates_from / produces Objects are addressed <kind>.<name> GLOBALLY rather than per-file or per- directory, mirroring how Kubernetes resolves a reference: an ApplicationSet's 'project: platform' means the AppProject named platform wherever it is declared. That scoping is what lets cross-file edges survive the per-file merge. References are matched by key NAME during a recursive walk rather than by fixed spec paths, so an ApplicationSet's nested spec.template.spec picks up the same refs as a bare Application with no ApplicationSet-specific code. Detection is by CONTENT (apiVersion + kind), never by extension: .yaml stays a DOC extension so ordinary YAML keeps going to the LLM path. The sniff runs after _is_sensitive() in detect(), so a secrets.yaml is dropped first. Mirrors the existing is_package_manifest_path() precedent (Graphify-Labs#1377). Helm templates are out of scope — Go template directives make them invalid YAML; they degrade to an empty result with an error rather than raising. pyyaml ships as the optional [k8s] extra, matching sql/terraform/pascal. Validated on a production Helm+ArgoCD monorepo: 31 manifests, 0 parse errors, 94-node graph. The generates_from edges alone show which registry each environment fans out over — i.e. which services can reach prod at all. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
graphify has no YAML extractor. Kubernetes manifests, ArgoCD GitOps config and Helm values all fall through to the LLM document path — expensive, and what comes back is inferred prose relationships rather than a real deployment graph.
On a Helm + ArgoCD monorepo the practical effect is that the deployment topology is unqueryable. I hit this trying to answer "how does this service reach prod?" — every file holding the answer was invisible to the graph.
What this adds
extract_k8s()parses the YAML that genuinely is YAML and emits:contains·in_namespace·references·deploys_to·deploys_chart·generates_from·producesThree decisions worth reviewing:
1. Global
<kind>.<name>addressing.extract_terraformscopes IDs by directory because Terraform resources are module-scoped. Kubernetes isn't: an ApplicationSet'sproject: platformmeans the AppProject namedplatformwherever it is declared. Global scoping is what lets those cross-file edges survive the per-file extraction merge. Pinned bytest_cross_file_reference_resolves_by_global_address.2. References matched by key NAME, not fixed spec paths. An ApplicationSet nests an entire Application spec under
spec.template.spec. A depth-agnostic recursive walk keyed on field names picks those up with zero ApplicationSet-specific code — that's most of why this is ~290 lines and not 600.3. Detection by CONTENT, not extension.
.yamlstays inDOC_EXTENSIONS; only files carrying bothapiVersion:andkind:are rerouted to the AST path. Ordinary YAML (CI configs, docs data, front-matter) keeps going to the LLM path exactly as before. This mirrors the existingis_package_manifest_path()precedent (#1377).Security note: the sniff runs after
_is_sensitive()indetect(), so asecrets.yamlis dropped before it can reach the AST path. I verified that ordering rather than assuming it.Deliberately out of scope
Helm templates (
charts/*/templates/*.yaml). Go template directives make them invalid YAML — measured on a real chart, 0 of 46 template files parse, while 100% of the manifests and values files do. They degrade to an empty result plus an error string rather than raising.Documented in the module docstring, so the graph doesn't overstate what it knows:
generates_fromto the glob it scans, not the files that glob matches. Resolving it would mean globbing the filesystem from inside a single-file extractor, breaking per-file caching and determinism.apiVersionnorkind(an ArgoCD fleet entry is often justapp: <name>) is indistinguishable from arbitrary config, so it stays a document.Validation
Real Helm + ArgoCD monorepo, 31 manifests: 0 parse errors, 94-node graph end-to-end through
detect→extract→build_from_json.The
generates_fromedges alone answer "which services can reach prod at all":Two ApplicationSets read different registries — which is exactly why one service deploys to dev but never to prod. That was previously invisible.
Dependency
pyyamlas the optional[k8s]extra, matching thesql/terraform/pascalpattern. Without it, affected files hit the standard missing-extra warning (_EXTRA_FOR_EXTENSION) instead of failing the run.Tests
14 new tests in
tests/test_k8s.py(+ a registry-identity test following thetest_<lang>_migratedconvention), covering each relation, ApplicationSet nesting, multi-document files, real line numbers, graceful Helm-template failure, and both directions of the detection sniff. Guarded bypytest.importorskip("yaml").Full suite: 3424 passed. Four failures in
test_ollama_retry_cap.py(ModuleNotFoundError: No module named 'openai') reproduce identically onv8without this change — pre-existing and unrelated.