-
Notifications
You must be signed in to change notification settings - Fork 126
Add component detectors for Docker Compose, Helm, and Kubernetes #1759
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from 25 commits
Commits
Show all changes
63 commits
Select commit
Hold shift + click to select a range
1a3f651
Add Docker Compose, Helm, and Kubernetes component detectors
jpinz 077d1d0
Add tests for Docker Compose, Dockerfile, Helm, and Kubernetes compon…
jpinz fd74cb5
Add `#nullable disable` to the new files
jpinz d5c91ed
Fix helm file patterns, and update tests
jpinz a111856
Add tests for Docker, Helm, and Kubernetes images with tag and digest
jpinz 70776f8
Add support for additional Docker Compose file patterns and new tests…
jpinz a099d56
Refactor DockerCompose and Kubernetes component detectors to address …
jpinz 0bbc3db
Refine HelmComponentDetector to process only relevant values files an…
jpinz 5c4e487
Add default to switch cases
jpinz b5d3fbf
Enhance KubernetesComponentDetector to skip non-manifest files and im…
jpinz 5158b75
Move DockerCompose ordering in DetectorClass enum
jpinz a26f434
Merge branch 'main' into jupinzer/add_container_detectors
jpinz 0873316
Fix build issues
jpinz ea3b0a4
Skip image references with unresolved variables in DockerCompose, Hel…
Copilot 5984b1c
Remove accidentally committed .nuget/nuget.exe and add to .gitignore
Copilot 9d56f38
Fix failing tests
jpinz 8bce22b
Merge branch 'main' into jupinzer/add_container_detectors
jpinz ad592b2
Add docs for the new detectors
jpinz 9ae1dd4
Add a new Containers DetectorCategory that enabled Dockerfile, Docker…
jpinz 70762f1
Use shared HasUnresolvedVariables in Dockerfile detector, add Helm co…
Copilot d220257
Fix AllowedDetectorIds+Categories intersection bug in DetectorRestric…
Copilot dff7d5f
Rename test variables for clarity in DetectorRestrictionServiceTests
Copilot 569babe
Refactor DockerfileComponentDetector to use DockerReferenceUtility fo…
jpinz f651b13
Revert Containers DetectorCategory support
jpinz 7f08f43
Fix Helm detector file ordering via two-pass OnPrepareDetectionAsync
Copilot b77c789
Refactor image reference handling in component detectors to use TryPa…
jpinz 70ada53
Remove duplicate search patterns for Helm charts in HelmComponentDete…
jpinz e5ffbb5
Add "PodTemplate" to KubernetesKinds in KubernetesComponentDetector
jpinz c0f586c
Remove `#nullable enable` from files added in this PR
jpinz 5d270f5
Add test for skipping values.yaml in different directory from Chart.yaml
jpinz df6c196
enable nullable on the new files and fix issues with it
jpinz ce8e98b
Clarify that values files won't be processed in the helm detector if …
jpinz 5b148b7
Refactor Categories property to use nameof for DetectorClass in multi…
jpinz 84b4383
Add `PodTemplate` to the docs for kubernetes
jpinz 6478e2e
Optimize the KubernetesComponentDetector.OnFileFoundAsync method
jpinz a3fa9b8
Add verification tests for DockerCompose, Helm, and Kubernetes.
jpinz 146759c
Merge branch 'main' into jupinzer/add_container_detectors
jpinz 7fc72c1
Add test for Kubernetes Pod with image tag and digest
jpinz 9b8e1b7
Merge branch 'main' into jupinzer/add_container_detectors
jpinz bac8dc8
Address copilot PR comments
jpinz e0e81f9
Merge branch 'main' into jupinzer/add_container_detectors
jpinz ccfdd1b
Address Copilot comment
jpinz 1cc4383
Fix IDE0057 error
jpinz 6b7d6fe
Merge branch 'main' into jupinzer/add_container_detectors
jpinz 035bdc9
Refactor ExtractImageReferences and TryRegisterStructuredImageReferen…
jpinz 2f0d13c
Address PR comments
jpinz 7307efc
Address copilot comments on the HelmComponentDetector
jpinz 5c56424
Merge branch 'main' into jupinzer/add_container_detectors
jpinz 2e5d0f5
Update apiVersion check in KubernetesComponentDetector to be case-ins…
jpinz 8ab1513
Rework the HelmComponentDetector to find Chart/values yaml files usin…
jpinz 77a8af2
Address copilot comment about Kubernetes logger, swapped it to a debu…
jpinz ade40f8
Merge branch 'main' into jupinzer/add_container_detectors
jpinz b52999a
Refactor component comparison logic to exclude experimental detectors…
jpinz d20304a
Revert integration test changes
jpinz 0ef45f1
All 3 new detectors now are marked as experimental.
jpinz d438845
Fix typo in Kubernetes detector
jpinz f66238d
Remove the experimental detector reference as defaultoff takes preced…
jpinz b0a4661
Remove new detectors from VerificationTest.ps1
jpinz f102880
Remove new detectors from workflow snapshots.
jpinz 4bc3c0a
Mark the Kubernetes Detector as IDefaultOffComponentDetector
jpinz 78aacbf
Address PR comments.
jpinz 40942a9
Update Helm detection documentation
jpinz 930d0b2
Address PR comments
jpinz File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -54,3 +54,4 @@ launchSettings.json | |
|
|
||
| # Dev nupkgs | ||
| dev-source | ||
| .nuget/ | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| # Docker Compose Detection | ||
|
|
||
| ## Requirements | ||
|
|
||
| Docker Compose detection depends on the following to successfully run: | ||
|
|
||
| - One or more Docker Compose files matching the patterns: `docker-compose.yml`, `docker-compose.yaml`, `docker-compose.*.yml`, `docker-compose.*.yaml`, `compose.yml`, `compose.yaml`, `compose.*.yml`, `compose.*.yaml` | ||
|
|
||
|
jpinz marked this conversation as resolved.
|
||
| The `DockerComposeComponentDetector` is a **DefaultOff** detector and must be explicitly enabled via the `--DetectorArgs` parameter. | ||
|
|
||
| ## Detection strategy | ||
|
|
||
| The Docker Compose detector parses YAML compose files to extract Docker image references from service definitions. | ||
|
|
||
| ### Service Image Detection | ||
|
|
||
| The detector looks for the `services` section and extracts the `image` field from each service: | ||
|
|
||
| ```yaml | ||
| services: | ||
| web: | ||
| image: nginx:1.21 | ||
| db: | ||
| image: postgres:14 | ||
| ``` | ||
|
|
||
| Services that only define a `build` directive without an `image` field are skipped, as they do not reference external Docker images. | ||
|
|
||
| ### Full Registry References | ||
|
|
||
| The detector supports full registry image references: | ||
|
|
||
| ```yaml | ||
| services: | ||
| app: | ||
| image: ghcr.io/myorg/myapp:v2.0 | ||
| ``` | ||
|
|
||
| ### Variable Resolution | ||
|
|
||
| Images containing unresolved variables (e.g., `${TAG}` or `{{ .Values.tag }}`) are skipped to avoid reporting incomplete or incorrect references. The detector checks for `$`, `{`, or `}` characters in image references. | ||
|
jpinz marked this conversation as resolved.
Outdated
|
||
|
|
||
| ## Known limitations | ||
|
|
||
| - **DefaultOff Status**: This detector must be explicitly enabled using `--DetectorArgs DockerCompose=EnableIfDefaultOff` | ||
| - **Variable Resolution**: Image references containing unresolved environment variables or template expressions are not reported, which may lead to under-reporting in compose files that heavily use variable substitution | ||
| - **Build-Only Services**: Services that only specify a `build` directive without an `image` field are not reported | ||
| - **No Dependency Graph**: All detected images are registered as independent components without parent-child relationships | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| # Helm Detection | ||
|
|
||
| ## Requirements | ||
|
|
||
| Helm detection depends on the following to successfully run: | ||
|
|
||
| - One or more Helm values files matching the patterns: `*values*.yaml`, `*values*.yml` | ||
| - Chart metadata files (`Chart.yaml`, `Chart.yml`, `chart.yaml`, `chart.yml`) are matched for file discovery but only values files are parsed for image references | ||
|
jpinz marked this conversation as resolved.
Outdated
|
||
|
|
||
|
jpinz marked this conversation as resolved.
|
||
| The `HelmComponentDetector` is a **DefaultOff** detector and must be explicitly enabled via the `--DetectorArgs` parameter. | ||
|
|
||
| ## Detection strategy | ||
|
|
||
| The Helm detector parses Helm values YAML files to extract Docker image references. It recursively walks the YAML tree looking for `image` keys. | ||
|
|
||
| ### Direct Image Strings | ||
|
|
||
| The detector recognizes image references specified as simple strings: | ||
|
|
||
| ```yaml | ||
| image: nginx:1.21 | ||
| ``` | ||
|
|
||
| ### Structured Image Objects | ||
|
|
||
| The detector also supports the common Helm chart pattern of structured image definitions: | ||
|
|
||
| ```yaml | ||
| image: | ||
| registry: ghcr.io | ||
| repository: org/myimage | ||
| tag: v1.0 | ||
| ``` | ||
|
|
||
| The `registry` and `tag` fields are optional. When present, the detector reconstructs the full image reference. The `digest` field is also supported. | ||
|
|
||
| ### Recursive Search | ||
|
|
||
| The detector recursively traverses all nested mappings and sequences in the values file, detecting image references at any depth in the YAML structure. | ||
|
|
||
| ### Variable Resolution | ||
|
|
||
| Images containing unresolved variables (e.g., `{{ .Values.tag }}`) are skipped to avoid reporting incomplete or incorrect references. The detector checks for `$`, `{`, or `}` characters in image references. | ||
|
|
||
| ## Known limitations | ||
|
|
||
| - **DefaultOff Status**: This detector must be explicitly enabled using `--DetectorArgs Helm=EnableIfDefaultOff` | ||
| - **Values Files Only**: Only files with `values` in the name are parsed for image references. Chart.yaml files are matched but not processed | ||
| - **Variable Resolution**: Image references containing unresolved Helm template expressions are not reported | ||
| - **No Dependency Graph**: All detected images are registered as independent components without parent-child relationships | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| # Kubernetes Detection | ||
|
|
||
| ## Requirements | ||
|
|
||
| Kubernetes detection depends on the following to successfully run: | ||
|
|
||
| - One or more Kubernetes manifest files matching the patterns: `*.yaml`, `*.yml` | ||
| - Manifests must contain both `apiVersion` and `kind` fields to be recognized as Kubernetes resources | ||
|
|
||
| The `KubernetesComponentDetector` is a **DefaultOff** detector and must be explicitly enabled via the `--DetectorArgs` parameter. | ||
|
|
||
| ## Detection strategy | ||
|
|
||
| The Kubernetes detector parses Kubernetes manifest YAML files to extract Docker image references from container specifications. | ||
|
|
||
| ### Supported Resource Kinds | ||
|
|
||
| The detector recognizes the following Kubernetes resource kinds: | ||
|
|
||
| - `Pod` | ||
| - `Deployment` | ||
| - `StatefulSet` | ||
| - `DaemonSet` | ||
| - `ReplicaSet` | ||
| - `Job` | ||
| - `CronJob` | ||
| - `ReplicationController` | ||
|
|
||
|
jpinz marked this conversation as resolved.
|
||
| Files with an unrecognized `kind` or missing `apiVersion`/`kind` fields are skipped. | ||
|
|
||
| ### Container Image Detection | ||
|
|
||
| The detector extracts image references from all container types within pod specifications: | ||
|
|
||
| - **containers**: Main application containers | ||
| - **initContainers**: Initialization containers that run before app containers | ||
| - **ephemeralContainers**: Ephemeral debugging containers | ||
|
|
||
| ### Pod Spec Locations | ||
|
|
||
| The detector handles different pod spec locations depending on the resource kind: | ||
|
|
||
| - **Pod**: `spec.containers` | ||
| - **Deployment, StatefulSet, DaemonSet, ReplicaSet, ReplicationController**: `spec.template.spec.containers` | ||
| - **Job**: `spec.template.spec.containers` | ||
| - **CronJob**: `spec.jobTemplate.spec.template.spec.containers` | ||
|
|
||
| ### Variable Resolution | ||
|
|
||
| Images containing unresolved variables (e.g., `${TAG}`) are skipped to avoid reporting incomplete or incorrect references. The detector checks for `$`, `{`, or `}` characters in image references. | ||
|
|
||
| ## Known limitations | ||
|
|
||
| - **DefaultOff Status**: This detector must be explicitly enabled using `--DetectorArgs Kubernetes=EnableIfDefaultOff` | ||
| - **Broad File Matching**: The `*.yaml` and `*.yml` search patterns match all YAML files, so the detector relies on content-based filtering (`apiVersion` and `kind` fields) to identify Kubernetes manifests | ||
| - **Variable Resolution**: Image references containing unresolved template variables are not reported | ||
| - **Limited Resource Kinds**: Only the eight resource kinds listed above are supported. Custom resources (CRDs) or other workload types are not detected | ||
|
jpinz marked this conversation as resolved.
Outdated
jpinz marked this conversation as resolved.
Outdated
|
||
| - **No Dependency Graph**: All detected images are registered as independent components without parent-child relationships | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
145 changes: 145 additions & 0 deletions
145
src/Microsoft.ComponentDetection.Detectors/dockercompose/DockerComposeComponentDetector.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,145 @@ | ||
| #nullable disable | ||
|
jpinz marked this conversation as resolved.
Outdated
|
||
| namespace Microsoft.ComponentDetection.Detectors.DockerCompose; | ||
|
|
||
| using System; | ||
| using System.Collections.Generic; | ||
| using System.IO; | ||
|
jpinz marked this conversation as resolved.
|
||
| using System.Threading; | ||
| using System.Threading.Tasks; | ||
| using Microsoft.ComponentDetection.Common; | ||
| using Microsoft.ComponentDetection.Contracts; | ||
| using Microsoft.ComponentDetection.Contracts.Internal; | ||
| using Microsoft.ComponentDetection.Contracts.TypedComponent; | ||
|
jpinz marked this conversation as resolved.
|
||
| using Microsoft.Extensions.Logging; | ||
| using YamlDotNet.RepresentationModel; | ||
|
|
||
| public class DockerComposeComponentDetector : FileComponentDetector, IDefaultOffComponentDetector | ||
| { | ||
| public DockerComposeComponentDetector( | ||
| IComponentStreamEnumerableFactory componentStreamEnumerableFactory, | ||
| IObservableDirectoryWalkerFactory walkerFactory, | ||
| ILogger<DockerComposeComponentDetector> logger) | ||
| { | ||
| this.ComponentStreamEnumerableFactory = componentStreamEnumerableFactory; | ||
| this.Scanner = walkerFactory; | ||
| this.Logger = logger; | ||
| } | ||
|
|
||
| public override string Id => "DockerCompose"; | ||
|
|
||
| public override IList<string> SearchPatterns { get; } = | ||
| [ | ||
| "docker-compose.yml", "docker-compose.yaml", | ||
| "docker-compose.*.yml", "docker-compose.*.yaml", | ||
| "compose.yml", "compose.yaml", | ||
|
jpinz marked this conversation as resolved.
|
||
| "compose.*.yml", "compose.*.yaml", | ||
|
jpinz marked this conversation as resolved.
|
||
| ]; | ||
|
jpinz marked this conversation as resolved.
|
||
|
|
||
| public override IEnumerable<ComponentType> SupportedComponentTypes => [ComponentType.DockerReference]; | ||
|
|
||
| public override int Version => 1; | ||
|
|
||
| public override IEnumerable<string> Categories => [Enum.GetName(typeof(DetectorClass), DetectorClass.DockerCompose)]; | ||
|
jpinz marked this conversation as resolved.
Outdated
|
||
|
|
||
| protected override async Task OnFileFoundAsync(ProcessRequest processRequest, IDictionary<string, string> detectorArgs, CancellationToken cancellationToken = default) | ||
| { | ||
| var singleFileComponentRecorder = processRequest.SingleFileComponentRecorder; | ||
| var file = processRequest.ComponentStream; | ||
|
|
||
| try | ||
| { | ||
| this.Logger.LogInformation("Discovered Docker Compose file: {Location}", file.Location); | ||
|
|
||
| string contents; | ||
| using (var reader = new StreamReader(file.Stream)) | ||
| { | ||
| contents = await reader.ReadToEndAsync(cancellationToken); | ||
| } | ||
|
|
||
| var yaml = new YamlStream(); | ||
| yaml.Load(new StringReader(contents)); | ||
|
|
||
| if (yaml.Documents.Count == 0) | ||
| { | ||
| return; | ||
| } | ||
|
|
||
| foreach (var document in yaml.Documents) | ||
| { | ||
| if (document.RootNode is YamlMappingNode rootMapping) | ||
| { | ||
| this.ExtractImageReferences(rootMapping, singleFileComponentRecorder, file.Location); | ||
| } | ||
| } | ||
| } | ||
| catch (Exception e) | ||
| { | ||
| this.Logger.LogError(e, "Failed to parse Docker Compose file: {Location}", file.Location); | ||
| } | ||
| } | ||
|
|
||
| private static YamlMappingNode GetMappingChild(YamlMappingNode parent, string key) | ||
| { | ||
| foreach (var entry in parent.Children) | ||
| { | ||
| if (entry.Key is YamlScalarNode scalarKey && string.Equals(scalarKey.Value, key, StringComparison.OrdinalIgnoreCase)) | ||
| { | ||
| return entry.Value as YamlMappingNode; | ||
| } | ||
| } | ||
|
|
||
| return null; | ||
| } | ||
|
|
||
| private void ExtractImageReferences(YamlMappingNode rootMapping, ISingleFileComponentRecorder recorder, string fileLocation) | ||
| { | ||
|
jpinz marked this conversation as resolved.
Outdated
|
||
| var services = GetMappingChild(rootMapping, "services"); | ||
| if (services == null) | ||
| { | ||
| return; | ||
| } | ||
|
|
||
| foreach (var serviceEntry in services.Children) | ||
| { | ||
| if (serviceEntry.Value is not YamlMappingNode serviceMapping) | ||
| { | ||
| continue; | ||
| } | ||
|
|
||
| // Extract direct image: references | ||
| foreach (var entry in serviceMapping.Children) | ||
| { | ||
| var key = (entry.Key as YamlScalarNode)?.Value; | ||
| if (string.Equals(key, "image", StringComparison.OrdinalIgnoreCase)) | ||
| { | ||
| var imageRef = (entry.Value as YamlScalarNode)?.Value; | ||
| if (!string.IsNullOrWhiteSpace(imageRef)) | ||
| { | ||
| this.TryRegisterImageReference(imageRef, recorder, fileLocation); | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| private void TryRegisterImageReference(string imageReference, ISingleFileComponentRecorder recorder, string fileLocation) | ||
|
jpinz marked this conversation as resolved.
Outdated
|
||
| { | ||
| if (DockerReferenceUtility.HasUnresolvedVariables(imageReference)) | ||
| { | ||
| return; | ||
| } | ||
|
jpinz marked this conversation as resolved.
Outdated
|
||
|
|
||
| try | ||
| { | ||
| var dockerRef = DockerReferenceUtility.ParseFamiliarName(imageReference); | ||
| if (dockerRef != null) | ||
| { | ||
| recorder.RegisterUsage(new DetectedComponent(dockerRef.ToTypedDockerReferenceComponent())); | ||
| } | ||
| } | ||
| catch (Exception e) | ||
| { | ||
| this.Logger.LogWarning(e, "Failed to parse image reference '{ImageReference}' in {Location}", imageReference, fileLocation); | ||
| } | ||
| } | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.