diff --git a/docs/BUILD-FILE.md b/docs/BUILD-FILE.md index 5ff455106..4735691c1 100644 --- a/docs/BUILD-FILE.md +++ b/docs/BUILD-FILE.md @@ -7,6 +7,7 @@ This documents the melange build file structure, fields, when, and why to use va The following are the high level sections for the build file, with detailed descriptions for each of them, and their fields in the sections following. ## Required + ### package Package metadata about this package, name, version, etc. @@ -20,6 +21,7 @@ The following are the high level sections for the build file, with detailed desc Ordered list of pipelines that produce this package ## Optional + ### subpackages List of subpackages that this package also produces. For example, docs. @@ -49,20 +51,26 @@ The following are the high level sections for the build file, with detailed desc Details about the particular package that will be used to find and use it. ### name + Unique name for the package. Convention is to use the same name as the YAML file without extension. This is what people will search for, so it's a good idea to keep it consistent with how the package is named in other distributions. for example: + ```yaml name: python-3.10 ``` ### version + Version of the package. For example: + ```yaml version: 3.10.12 ``` ### epoch + Monotonically increasing value (starting at 0) indicating same version of the package, but with changes (security patches for example) applied to it. + ```yaml epoch: 0 ``` @@ -72,40 +80,51 @@ form: `--r.apk` for our example above, this would be: `python-3.10-3.10.12-r0.apk`. ### description + Human readable description of the package. Make this meaningful, as this information shows up when searching for the package with apk, for example: + ```yaml description: "the Python programming language" ``` ### url [optional] + The URL to the packages homepage. ### commit [optional] + The git commit of the package build configuration TODO(vaikas): is the 'is package build configuration' this file? TODO(vaikas): why would I use this? I did not see an example use. ### target-architecture [optional] + List of architectures for which this package should be built for. Valid architectures are: `386`, `amd64`, `arm/v6`, `arm/v7`, `arm64`, `ppc64le`, -`s390x`, `x86_64`, `aarch64`, special `all` that builds it for all of them. -Leaving this out defaults to `all`. +`s390x`, `x86_64`, `aarch64`, special `all` that builds it for all of them, +and special `noarch` for architecture-independent packages. A `noarch` +package is built once using the host's native architecture; `noarch` must be +the only entry in the list when used. Leaving this out defaults to `all`. TODO(vaikas): rekor-cli.yaml sets this to all? So is that not the default? TODO(vaikas): Saw something about riscv64. Does all include that? ### copyright + List of copyrights for this package. Each entry defines the scope (paths, and which license applies to it) and may include the following fields: #### license + The license for either the package or part of the package (if there are multiple entries). It is important to note that only packages with OSI-approved licenses can be included in Wolfi. You can check the relevant package info in the licenses page at [opensource.org](https://opensource.org/licenses/). Supports variable substitution, which is useful for producing unique license references across version streamed packages. #### paths [optional] + File globs (relative to the package root) that this license applies to. Defaults to `*` (the whole package). Use this when different parts of the package ship under different licenses: + ```yaml copyright: - license: Apache-2.0 @@ -117,8 +136,10 @@ copyright: ``` #### attestation [optional] + Free-form attribution text appended to the package's copyright notice (for example, the upstream `NOTICE` contents): + ```yaml copyright: - license: Apache-2.0 @@ -128,23 +149,27 @@ copyright: ``` #### license-path [optional] + Path (relative to the build workspace) to a file containing the license text to embed in the SBOM. Required for non-SPDX `license` values. Supports `${{package.*}}` and `${{vars.*}}` substitution. License must stay within the workspace. #### detection-override [optional] + Overrides the result of automatic license detection for the file referenced by `license-path`. Use this when the on-disk text is misidentified by the classifier but the declared `license` is known to be correct. For example, saying that this entire package has license `PSF-2.0` + ```yaml copyright: - license: PSF-2.0 ``` Another example using `license-path` with a versioned directory: + ```yaml copyright: - license: CustomLicense @@ -154,6 +179,7 @@ copyright: Variables in `license` are also substituted, which is helpful when the SPDX identifier itself is derived from a `var-transforms` rule (e.g. selecting `GPL-2.0-only` vs `GPL-3.0-only` based on the upstream version): + ```yaml vars: license-version: "2.4" @@ -164,10 +190,12 @@ copyright: ``` ### dependencies + List of packages that this package depends on at runtime, but not during build time. These will get installed by apk as system dependencies when the package is installed. For example, saying that a package depends on `openssl`, `socat`, and `curl` at runtime: + ```yaml dependencies: runtime: @@ -177,6 +205,7 @@ dependencies: ``` #### provides + Provides allows you to create "aliases" for a package. If your `package.name` is for example `php-8.1`, but you want somebody be able to get this package by `php`, you could provide a section like this: @@ -192,6 +221,7 @@ provide a floating version, so that when the package gets upgraded, the user will get the latest one. For that melange provides a `${{package.full-version}}` variable. It gets expanded to `${{package.version}}-r${{package-epoch}}`. So for the example above, you could do this + ```yaml dependencies: provides: @@ -203,6 +233,7 @@ using our php example, there are 8.1.X and 8.2.X streams, so the condensed example here: `php-8.1.yaml`: + ```yaml package: name: php-8.1 @@ -214,6 +245,7 @@ package: ``` `php-8.2.yaml`: + ```yaml package: name: php-8.2 @@ -232,6 +264,7 @@ they again explicitly asked for the 8.2 version. Now if they just ask for php no other additional constraints defined. ### options + Options that describe the package functionality. Currently there are three options, and these are used by SCA tools to control their behaviour. @@ -275,6 +308,7 @@ options: ``` ### scriptlets + List of executable scripts that run at various stages of the package lifecycle, triggered by configurable events. These are useful to handle tasks that only happen during install, uninstall, upgrade. The life-cycle events are: @@ -310,6 +344,7 @@ TODO(vaikas): What does it mean to monitor, when new files are added/removed to those directories? Something else?? ### timeout + Optional timeout duration for the build. Specifies the maximum amount of time the build is allowed to take before timing out. The value is specified in seconds as an integer. ```yaml @@ -318,6 +353,7 @@ package: ``` ### resources + Optional resource specifications for the build. Used by external schedulers (like elastic build) to provision appropriately-sized build pods/VMs. For local builds with the QEMU runner, these can be used as resource limits via CLI flags. **Resource Fields:** @@ -328,11 +364,13 @@ Optional resource specifications for the build. Used by external schedulers (lik - `disk`: Disk space in Kubernetes format (e.g., `"50Gi"`, `"100Gi"`, `"1Ti"`) **Value Formats:** + - CPU values are typically whole numbers as strings: `"1"`, `"2"`, `"4"`, `"8"`, etc. - Memory and disk use Kubernetes resource quantities: `Mi` (mebibytes), `Gi` (gibibytes), `Ti` (tebibytes) - All fields are optional and interpretation depends on the scheduler/runner **How resources are interpreted:** + - **External schedulers**: Use these values to provision build pods/VMs - **QEMU runner** (via CLI flags like `--cpu`, `--memory`): Treats values as **maximum limits** - CPU: Defaults to all available cores, capped at the specified value if lower @@ -348,9 +386,11 @@ package: ``` ### test-resources + Optional resource specifications for test execution. Used by external schedulers to provision test pods/VMs with different resource constraints than the build phase. **When to use test-resources:** + - Tests require significantly different resources than builds - Integration tests need more CPU/memory than unit tests - Tests can run with fewer resources to optimize costs @@ -361,10 +401,12 @@ Optional resource specifications for test execution. Used by external schedulers The `test-resources` field is primarily **informational** for external schedulers: **For external schedulers** (reading the YAML): + - Use `test-resources` if specified, otherwise fall back to `resources` - This determines test pod/VM sizing **For local testing with `melange test`:** + - The `test-resources` field in the YAML is **NOT automatically used** by melange - Resources must be explicitly specified via CLI flags: `--cpu`, `--memory`, `--disk`, `--cpumodel`, `--timeout` - Resource enforcement depends on the runner (same as `resources` field): @@ -374,6 +416,7 @@ The `test-resources` field is primarily **informational** for external scheduler **Resource fields** are identical to `resources` (see above for formats and interpretation). Example where tests need less resources than build: + ```yaml package: resources: @@ -387,6 +430,7 @@ package: ``` Example where tests need more resources than build: + ```yaml package: resources: @@ -399,6 +443,7 @@ package: ``` Example with only test-resources specified: + ```yaml package: test-resources: @@ -408,6 +453,7 @@ package: ``` # environment + Environment defines the build environment, including what the dependencies are, including repositories, packages, etc. @@ -418,6 +464,7 @@ from subpackage test definitions, where separate environments can be specified for each subpackage that differ from the main package. ## Local building + When building locally, you'll also need to include information about where to find Wolfi packages. This is not needed when submitting the package to the Wolfi OS repository. The "contents" node is used for that: ```yaml @@ -432,23 +479,28 @@ environment: ``` ## contents + Contents has 3 lists that define where to look for packages, how to validate the repository, and which packages to install. ### repositories + Which repositories to fetch the packages from. **NOTE** Do not mix Alpine apk repositories with Wolfi apk repositories. ### keyring + These are used to validate the authenticity of a repository. TODO(vaikas): Are there any constraints here, or if any key in the keyring matches a repository, then all is well. I'd assume so. ### packages + Packages is the list of packages to install in the build environment for running the pipeline; in other words, these are the necessary build time dependencies for the package. For example: + ```yaml environment: contents: @@ -463,6 +515,7 @@ environment: ``` To specify a version for packages, you can use the following syntax: + ```yaml environment: packages: @@ -470,14 +523,17 @@ environment: - foo=~4.5.6 # install any version with a name starting with "4.5.6" (e.g., 4.5.6-r7) - python3 # install the latest stable version of python3. ``` + For additional information, see the [Chainguard Academy article](https://edu.chainguard.dev/open-source/wolfi/apk-version-selection/). ## accounts + Accounts support adding additional users and groups into the build environment, as well as running the build under a different user than the build runner's default. ### run-as + Specifies which user to run the build under, the user must already exist or be created in the build environment using the `users` field. @@ -491,24 +547,31 @@ Tests are more likely to be situations where running as the non-default user may be desired. ### users + List of users to inject into the build image #### username + The name of the user #### uid + The uid of the user #### gid + The primary gid of the user ### groups + List of groups to inject into the build image #### groupname + The name of the group #### gid + The gid of the grpup An example creating two users in the same group, and running the build as @@ -531,6 +594,7 @@ environment: ``` ## environment + environment allows you to control environmental variables to set while running the pipeline. For example, to set the env variable `CGO_ENABLED` to `0`: @@ -541,10 +605,10 @@ environment: ``` TODO(vaikas): melange config points to apko here: - https://github.com/chainguard-dev/melange/blob/main/pkg/config/config.go#L256 + which points to [ImageConfiguration](https://github.com/chainguard-dev/apko/blob/main/pkg/build/types/types.go#L106), which has a ton of stuff, is all that really supported, or just `environment` # pipeline -Pipeline defines the ordered steps to build the package. +Pipeline defines the ordered steps to build the package. diff --git a/docs/BUILD-PROCESS.md b/docs/BUILD-PROCESS.md index 29c9883de..6e1dde31a 100644 --- a/docs/BUILD-PROCESS.md +++ b/docs/BUILD-PROCESS.md @@ -7,7 +7,7 @@ This document describes the Melange build process. The melange yaml file consists of the following components that are key to the build process. Note that this is not the official or a comprehensive `melange.yaml` reference. -* `package.target-architectures`: describes which architectures to build for (if empty, build for all available archs). +* `package.target-architectures`: describes which architectures to build for (if empty, build for all available archs); `noarch` builds an architecture-independent package once using the host architecture. * `package.dependences.runtime`: list of apk packages that need to be available in the final apk package, hence `runtime`. * `environment.contents`: list of apk packages and their source repositories that need to be available during the build, but not in the final apk package. * `pipeline`: list of steps to execute during the build. diff --git a/e2e-tests/README.md b/e2e-tests/README.md index 8bdcae778..da4438681 100644 --- a/e2e-tests/README.md +++ b/e2e-tests/README.md @@ -7,6 +7,7 @@ Melange options are based on yaml file name. * `*-build.yaml`: run 'melange build' * `*-test.yaml`: run 'melange test' * `*-build-test`: run 'melange build && melange test' + * `*-multiarch-build`: run 'melange build' for `x86_64,aarch64` and assert the package and index are written to both arch directories If the yaml file name matches '*-nopkg', then the flag `--test-package-append` will be appended for `busybox` and `python-3`. The intent of these tests diff --git a/e2e-tests/noarch-build-test.yaml b/e2e-tests/noarch-build-test.yaml new file mode 100644 index 000000000..3c63e4cc3 --- /dev/null +++ b/e2e-tests/noarch-build-test.yaml @@ -0,0 +1,53 @@ +# Verify noarch packages build once on host architecture, execute real melange +# tests, and contain only architecture-independent shell/text content. +package: + name: noarch-test + version: 1.0 + epoch: 0 + target-architecture: + - noarch + dependencies: + runtime: + - busybox + +environment: + contents: + packages: + - busybox + +pipeline: + - name: install noarch script + runs: | + bdir=${{targets.destdir}}/usr/bin + mkdir -p "$bdir" + cat > "$bdir/noarch-test" <<"EOF" + #!/bin/sh + echo "noarch works" + EOF + chmod 755 "$bdir/noarch-test" + +test: + pipeline: + - name: test noarch script + runs: | + ${{vars.tester-blob}} + testrun noarch-test "noarch works" + +vars: + tester-blob: | + set +x + testfail() { echo "FAIL:" "$tname${1:+ - $*}" 1>&2; exit 1; } + testpass() { echo "PASS:" "$tname"; } + error() { echo "FATAL:" "$@" 1>&2; exit 1; } + testrun() { + local cmd="$1" expected="$2" tname="" out="" + [ -n "$cmd" ] || error "cmd must set cmd" + + tname="'$cmd' is in PATH" + out=$(command -v "$cmd") && testpass || testfail + + tname="'$cmd' outputs '$expected'" + out=$($cmd) || testfail "'$cmd' exited $?" + [ "$out" = "$expected" ] && + testpass || testfail "found '$out'" + } diff --git a/e2e-tests/noarch-multiarch-build.yaml b/e2e-tests/noarch-multiarch-build.yaml new file mode 100644 index 000000000..3c717c744 --- /dev/null +++ b/e2e-tests/noarch-multiarch-build.yaml @@ -0,0 +1,26 @@ +# Verify one noarch build is copied and indexed for multiple requested arches. +package: + name: noarch-multiarch-test + version: 1.0 + epoch: 0 + target-architecture: + - noarch + dependencies: + runtime: + - busybox + +environment: + contents: + packages: + - busybox + +pipeline: + - name: install noarch script + runs: | + bdir=${{targets.destdir}}/usr/bin + mkdir -p "$bdir" + cat > "$bdir/noarch-multiarch-test" <<"EOF" + #!/bin/sh + echo "noarch multiarch works" + EOF + chmod 755 "$bdir/noarch-multiarch-test" diff --git a/e2e-tests/run-tests b/e2e-tests/run-tests index adb8bed25..ce61fdc62 100755 --- a/e2e-tests/run-tests +++ b/e2e-tests/run-tests @@ -41,6 +41,8 @@ fails="" for yaml in "$@"; do args="${ARGS}" ops="" + arch_flag="${ARCH}" + multiarch="" base=${yaml%.yaml} case "$base" in *-build-test) @@ -58,6 +60,11 @@ for yaml in "$@"; do args="${base%-test}" args="$args --debug" ;; + *-multiarch-build) + ops="build" + arch_flag="x86_64,aarch64" + multiarch="true" + ;; *-build) ops="build" ;; @@ -77,17 +84,33 @@ for yaml in "$@"; do "test") opargs="--pipeline-dirs $PWD/pipelines";; esac - vrc "Testing $base from $yaml for $op" \ + if vrc "Testing $base from $yaml for $op" \ ${MELANGE} "$op" \ - --arch=${ARCH} --source-dir=./test-fixtures \ + --arch=${arch_flag} --source-dir=./test-fixtures \ --runner=qemu \ "$yaml" \ ${args} $opargs \ "--keyring-append=$PWD/$key.pub" \ "--repository-append=$PWD/packages" \ "--repository-append=https://packages.wolfi.dev/os" \ - "--keyring-append=https://packages.wolfi.dev/os/wolfi-signing.rsa.pub" || - fails="${fails} $yaml/$op" + "--keyring-append=https://packages.wolfi.dev/os/wolfi-signing.rsa.pub"; then + if [ -n "$multiarch" ]; then + for arch in x86_64 aarch64; do + apk="packages/$arch/noarch-multiarch-test-1.0-r0.apk" + index="packages/$arch/APKINDEX.tar.gz" + if [ ! -f "$apk" ]; then + echo "ERROR: missing noarch package $apk" + fails="${fails} $yaml/$op" + fi + if [ ! -f "$index" ]; then + echo "ERROR: missing package index $index" + fails="${fails} $yaml/$op" + fi + done + fi + else + fails="${fails} $yaml/$op" + fi done done diff --git a/pkg/build/build.go b/pkg/build/build.go index e14f56703..2dca0cbad 100644 --- a/pkg/build/build.go +++ b/pkg/build/build.go @@ -108,6 +108,8 @@ type Build struct { EmptyWorkspace bool OutDir string Arch apko_types.Architecture + ReplicateArchs []apko_types.Architecture + noArchStagingDir string Libc string ExtraKeys []string ExtraRepos []string @@ -159,56 +161,40 @@ type Build struct { PkgResolver *apk.PkgResolver } -func New(ctx context.Context, opts ...Option) (*Build, error) { - b := Build{ - WorkspaceIgnore: ".melangeignore", - SourceDir: ".", - OutDir: ".", - CacheDir: "./melange-cache/", - Arch: apko_types.ParseArchitecture(runtime.GOARCH), - GuestFS: tarfs.New(), - Start: time.Now(), - SBOMGenerator: &spdx.Generator{}, - } - - for _, opt := range opts { - if err := opt(&b); err != nil { - return nil, err - } +// PackageArch returns architecture recorded in package metadata and output. +func (b *Build) PackageArch() string { + if b.Configuration != nil && b.Configuration.Package.IsNoArch() { + return "noarch" } + return b.Arch.ToAPK() +} - log := clog.FromContext(ctx).With("arch", b.Arch.ToAPK()) - ctx = clog.WithLogger(ctx, log) - - // If no workspace directory is explicitly requested, create a - // temporary directory for it. Otherwise, ensure we are in a - // subdir for this specific build context. - if b.WorkspaceDir != "" { - b.WorkspaceDir = filepath.Join(b.WorkspaceDir, b.Arch.ToAPK()) - - // Get the absolute path to the workspace dir, which is needed for bind - // mounts. - absdir, err := filepath.Abs(b.WorkspaceDir) +func (b *Build) getNoArchStagingDir() (string, error) { + if b.noArchStagingDir == "" { + dir, err := os.MkdirTemp(os.TempDir(), "melange-noarch-*") if err != nil { - return nil, fmt.Errorf("unable to resolve path %s: %w", b.WorkspaceDir, err) + return "", fmt.Errorf("creating noarch staging directory: %w", err) } + b.noArchStagingDir = dir + } + return b.noArchStagingDir, nil +} - b.WorkspaceDir = absdir - } else if b.Runner != nil { - tmpdir, err := os.MkdirTemp(b.Runner.TempDir(), "melange-workspace-*") - if err != nil { - return nil, fmt.Errorf("unable to create workspace dir: %w", err) - } - b.WorkspaceDir = tmpdir +func (b *Build) replicateTargets() []apko_types.Architecture { + if b.Configuration != nil && b.Configuration.Package.IsNoArch() && len(b.ReplicateArchs) > 0 { + return b.ReplicateArchs } + return []apko_types.Architecture{b.Arch} +} +func (b *Build) resolveConfiguration(ctx context.Context) error { // If no config file is explicitly requested for the build context // we check if .melange.yaml or melange.yaml exist. checks := []string{".melange.yaml", ".melange.yml", "melange.yaml", "melange.yml"} if b.ConfigFile == "" { for _, chk := range checks { if _, err := os.Stat(chk); err == nil { - log.Infof("no configuration file provided -- using %s", chk) + clog.FromContext(ctx).Infof("no configuration file provided -- using %s", chk) b.ConfigFile = chk break } @@ -217,13 +203,13 @@ func New(ctx context.Context, opts ...Option) (*Build, error) { // If no config file could be automatically detected, error. if b.ConfigFile == "" { - return nil, fmt.Errorf("melange.yaml is missing") + return fmt.Errorf("melange.yaml is missing") } if b.ConfigFileRepositoryURL == "" { - return nil, fmt.Errorf("config file repository URL was not set") + return fmt.Errorf("config file repository URL was not set") } if b.ConfigFileRepositoryCommit == "" { - return nil, fmt.Errorf("config file repository commit was not set") + return fmt.Errorf("config file repository commit was not set") } if b.Configuration == nil { @@ -239,19 +225,93 @@ func New(ctx context.Context, opts ...Option) (*Build, error) { config.WithCommit(b.ConfigFileRepositoryCommit), ) if err != nil { - return nil, fmt.Errorf("failed to load configuration: %w", err) + return fmt.Errorf("failed to load configuration: %w", err) } b.Configuration = parsedCfg } - if len(b.Configuration.Package.TargetArchitecture) == 1 && - b.Configuration.Package.TargetArchitecture[0] == "all" { + return nil +} + +// PeekIsNoArch resolves just enough configuration to report whether the build +// targets the noarch architecture, without doing New's heavier setup. +func PeekIsNoArch(ctx context.Context, opts ...Option) (bool, error) { + b := &Build{} + for _, opt := range opts { + if err := opt(b); err != nil { + return false, err + } + } + if err := b.resolveConfiguration(ctx); err != nil { + return false, err + } + return b.Configuration.Package.IsNoArch(), nil +} + +func New(ctx context.Context, opts ...Option) (*Build, error) { + b := Build{ + WorkspaceIgnore: ".melangeignore", + SourceDir: ".", + OutDir: ".", + CacheDir: "./melange-cache/", + Arch: apko_types.ParseArchitecture(runtime.GOARCH), + GuestFS: tarfs.New(), + Start: time.Now(), + SBOMGenerator: &spdx.Generator{}, + } + + for _, opt := range opts { + if err := opt(&b); err != nil { + return nil, err + } + } + + log := clog.FromContext(ctx).With("arch", b.Arch.ToAPK()) + ctx = clog.WithLogger(ctx, log) + + if err := b.resolveConfiguration(ctx); err != nil { + return nil, err + } + + switch { + case b.Configuration.Package.IsNoArch(): + hostArch := apko_types.ParseArchitecture(runtime.GOARCH) + if b.Arch != hostArch { + log.Infof("ignoring requested architecture %s for noarch package; building once using host architecture %s", b.Arch.ToAPK(), hostArch.ToAPK()) + } + b.Arch = hostArch + log = log.With("arch", b.Arch.ToAPK()) + ctx = clog.WithLogger(ctx, log) + case len(b.Configuration.Package.TargetArchitecture) == 1 && + b.Configuration.Package.TargetArchitecture[0] == "all": log.Warnf("target-architecture: ['all'] is deprecated and will become an error; remove this field to build for all available archs") - } else if len(b.Configuration.Package.TargetArchitecture) != 0 && - !slices.Contains(b.Configuration.Package.TargetArchitecture, b.Arch.ToAPK()) { + case len(b.Configuration.Package.TargetArchitecture) != 0 && + !slices.Contains(b.Configuration.Package.TargetArchitecture, b.Arch.ToAPK()): return nil, ErrSkipThisArch } + // If no workspace directory is explicitly requested, create a + // temporary directory for it. Otherwise, ensure we are in a + // subdir for this specific build context. + if b.WorkspaceDir != "" { + b.WorkspaceDir = filepath.Join(b.WorkspaceDir, b.Arch.ToAPK()) + + // Get the absolute path to the workspace dir, which is needed for bind + // mounts. + absdir, err := filepath.Abs(b.WorkspaceDir) + if err != nil { + return nil, fmt.Errorf("unable to resolve path %s: %w", b.WorkspaceDir, err) + } + + b.WorkspaceDir = absdir + } else if b.Runner != nil { + tmpdir, err := os.MkdirTemp(b.Runner.TempDir(), "melange-workspace-*") + if err != nil { + return nil, fmt.Errorf("unable to create workspace dir: %w", err) + } + b.WorkspaceDir = tmpdir + } + // SOURCE_DATE_EPOCH will always overwrite the build flag if _, ok := os.LookupEnv("SOURCE_DATE_EPOCH"); ok { t, err := sourceDateEpoch(b.SourceDateEpoch) @@ -804,7 +864,7 @@ func (b *Build) BuildPackage(ctx context.Context) error { outDir = b.OutDir } - if err := linter.LintBuild(ctx, b.Configuration, lt.pkgName, require, warn, fsys, outDir, b.Arch.ToAPK()); err != nil { + if err := linter.LintBuild(ctx, b.Configuration, lt.pkgName, require, warn, fsys, outDir, b.PackageArch()); err != nil { return fmt.Errorf("unable to lint package %s: %w", lt.pkgName, err) } } @@ -834,7 +894,7 @@ func (b *Build) BuildPackage(ctx context.Context) error { OutputFS: outfs, SourceDateEpoch: b.SourceDateEpoch, Namespace: namespace, - Arch: b.Arch.ToAPK(), + Arch: b.PackageArch(), ConfigFile: &sbom.ConfigFile{ Path: b.ConfigFile, RepositoryURL: b.ConfigFileRepositoryURL, @@ -848,6 +908,25 @@ func (b *Build) BuildPackage(ctx context.Context) error { return fmt.Errorf("generating SBOMs: %w", err) } + var noArchStagingDir string + // The staging dir is created and its cleanup deferred here, before any + // Emit call, rather than in Emit/package.go or after the Emit calls + // below. Emit is invoked once per package (main + each subpackage), + // each a separate function call with its own defer scope, so a defer + // inside Emit would fire after the FIRST Emit call and wipe the staging + // dir before later subpackages get emitted into it. Registering the + // defer here, before Emit runs at all, also ensures the staging dir is + // still cleaned up if any Emit call fails and this function returns + // early, instead of leaking a temp dir on every failed noarch build. + if b.Configuration.Package.IsNoArch() { + dir, err := b.getNoArchStagingDir() + if err != nil { + return err + } + noArchStagingDir = dir + defer os.RemoveAll(noArchStagingDir) + } + // emit main package if err := b.Emit(ctx, pkg); err != nil { return fmt.Errorf("unable to emit package: %w", err) @@ -860,6 +939,33 @@ func (b *Build) BuildPackage(ctx context.Context) error { } } + if b.Configuration.Package.IsNoArch() { + stagingDir := noArchStagingDir + + entries, err := os.ReadDir(stagingDir) + if err != nil { + return fmt.Errorf("reading noarch staging directory: %w", err) + } + for _, arch := range b.replicateTargets() { + packageDir := filepath.Join(b.OutDir, arch.ToAPK()) + if err := os.MkdirAll(packageDir, 0o755); err != nil { + return fmt.Errorf("creating noarch package directory %s: %w", packageDir, err) + } + for _, entry := range entries { + if entry.IsDir() { + continue + } + info, err := entry.Info() + if err != nil { + return fmt.Errorf("statting staged noarch package %s: %w", entry.Name(), err) + } + if err := copyFile(stagingDir, entry.Name(), packageDir, info.Mode().Perm()); err != nil { + return fmt.Errorf("replicating noarch package %s to %s: %w", entry.Name(), packageDir, err) + } + } + } + } + // clean build environment log.Debugf("cleaning workspacedir") cleanEnv := map[string]string{} @@ -873,32 +979,35 @@ func (b *Build) BuildPackage(ctx context.Context) error { // generate APKINDEX.tar.gz and sign it if b.GenerateIndex { - packageDir := filepath.Join(b.OutDir, b.Arch.ToAPK()) - log.Infof("generating apk index from packages in %s", packageDir) + for _, arch := range b.replicateTargets() { + packageDir := filepath.Join(b.OutDir, arch.ToAPK()) + log.Infof("generating apk index from packages in %s", packageDir) - var apkFiles []string - pkgFileName := fmt.Sprintf("%s-%s-r%d.apk", b.Configuration.Package.Name, b.Configuration.Package.Version, b.Configuration.Package.Epoch) - apkFiles = append(apkFiles, filepath.Join(packageDir, pkgFileName)) + var apkFiles []string + pkgFileName := fmt.Sprintf("%s-%s-r%d.apk", b.Configuration.Package.Name, b.Configuration.Package.Version, b.Configuration.Package.Epoch) + apkFiles = append(apkFiles, filepath.Join(packageDir, pkgFileName)) - for _, subpkg := range b.Configuration.Subpackages { - subpkgFileName := fmt.Sprintf("%s-%s-r%d.apk", subpkg.Name, b.Configuration.Package.Version, b.Configuration.Package.Epoch) - apkFiles = append(apkFiles, filepath.Join(packageDir, subpkgFileName)) - } + for _, subpkg := range b.Configuration.Subpackages { + subpkgFileName := fmt.Sprintf("%s-%s-r%d.apk", subpkg.Name, b.Configuration.Package.Version, b.Configuration.Package.Epoch) + apkFiles = append(apkFiles, filepath.Join(packageDir, subpkgFileName)) + } - opts := []index.Option{ - index.WithPackageFiles(apkFiles), - index.WithSigningKey(b.SigningKey), - index.WithMergeIndexFileFlag(true), - index.WithIndexFile(filepath.Join(packageDir, "APKINDEX.tar.gz")), - } + opts := []index.Option{ + index.WithPackageFiles(apkFiles), + index.WithExpectedArch(arch.ToAPK()), + index.WithSigningKey(b.SigningKey), + index.WithMergeIndexFileFlag(true), + index.WithIndexFile(filepath.Join(packageDir, "APKINDEX.tar.gz")), + } - idx, err := index.New(opts...) - if err != nil { - return fmt.Errorf("unable to create index: %w", err) - } + idx, err := index.New(opts...) + if err != nil { + return fmt.Errorf("unable to create index: %w", err) + } - if err := idx.GenerateIndex(ctx); err != nil { - return fmt.Errorf("unable to generate index: %w", err) + if err := idx.GenerateIndex(ctx); err != nil { + return fmt.Errorf("unable to generate index: %w", err) + } } } diff --git a/pkg/build/build_test.go b/pkg/build/build_test.go index eb890f62e..40f5d945a 100644 --- a/pkg/build/build_test.go +++ b/pkg/build/build_test.go @@ -15,6 +15,7 @@ package build import ( + "context" "fmt" "os" "path/filepath" @@ -34,6 +35,31 @@ var requireErrInvalidConfiguration require.ErrorAssertionFunc = func(t require.T require.ErrorAs(t, err, &config.ErrInvalidConfiguration{}) } +func TestPeekIsNoArch(t *testing.T) { + for _, tt := range []struct { + name string + targetArch string + want bool + }{ + {name: "noarch", targetArch: "[noarch]", want: true}, + {name: "native", targetArch: "[x86_64]", want: false}, + } { + t.Run(tt.name, func(t *testing.T) { + configFile := filepath.Join(t.TempDir(), "melange.yaml") + contents := fmt.Sprintf("package:\n name: test\n version: 1.0\n epoch: 0\n target-architecture: %s\n", tt.targetArch) + require.NoError(t, os.WriteFile(configFile, []byte(contents), 0o644)) + + got, err := PeekIsNoArch(context.Background(), + WithConfig(configFile), + WithConfigFileRepositoryURL("https://example.com/repo"), + WithConfigFileRepositoryCommit("test-commit"), + ) + require.NoError(t, err) + require.Equal(t, tt.want, got) + }) + } +} + // TestConfiguration_Load is the main set of tests for loading a configuration // file. When in doubt, add your test here. func TestConfiguration_Load(t *testing.T) { diff --git a/pkg/build/compiler_config.go b/pkg/build/compiler_config.go index ba19a8a22..eba8dcfdf 100644 --- a/pkg/build/compiler_config.go +++ b/pkg/build/compiler_config.go @@ -25,7 +25,7 @@ import ( "github.com/chainguard-dev/clog" ) -var packageMetadataTemplate = `{"type":"apk","os":"{{.Namespace}}","name":"{{.Configuration.Package.Name}}","version":"{{.Configuration.Package.FullVersion}}","architecture":"{{.Arch.ToAPK}}"{{if .Configuration.Package.CPE.Vendor}},"appCpe":"{{.Configuration.Package.CPEString}}"{{end}}}` +var packageMetadataTemplate = `{"type":"apk","os":"{{.Namespace}}","name":"{{.Configuration.Package.Name}}","version":"{{.Configuration.Package.FullVersion}}","architecture":"{{.PackageArch}}"{{if .Configuration.Package.CPE.Vendor}},"appCpe":"{{.Configuration.Package.CPEString}}"{{end}}}` var gccLinkTemplate = `*link: + %{!r:--package-metadata=` + packageMetadataTemplate + `} diff --git a/pkg/build/options.go b/pkg/build/options.go index b164dc9ae..f239d4c49 100644 --- a/pkg/build/options.go +++ b/pkg/build/options.go @@ -203,6 +203,14 @@ func WithArch(arch apko_types.Architecture) Option { } } +// WithReplicateArchs sets architectures that receive replicated noarch packages. +func WithReplicateArchs(archs []apko_types.Architecture) Option { + return func(b *Build) error { + b.ReplicateArchs = archs + return nil + } +} + // WithExtraKeys adds a set of extra keys to the build context. func WithExtraKeys(extraKeys []string) Option { return func(b *Build) error { diff --git a/pkg/build/package.go b/pkg/build/package.go index 3eb53fe8b..d49ef09f3 100644 --- a/pkg/build/package.go +++ b/pkg/build/package.go @@ -90,14 +90,24 @@ func pkgFromSub(sub *config.Subpackage) *config.Package { func (b *Build) Emit(ctx context.Context, pkg *config.Package) error { b.End = time.Now() + var outDir string + if b.Configuration.Package.IsNoArch() { + dir, err := b.getNoArchStagingDir() + if err != nil { + return err + } + outDir = dir + } else { + outDir = filepath.Join(b.OutDir, b.Arch.ToAPK()) + } pc := PackageBuild{ Build: b, Origin: &b.Configuration.Package, PackageName: pkg.Name, OriginName: pkg.Name, - OutDir: filepath.Join(b.OutDir, b.Arch.ToAPK()), + OutDir: outDir, Dependencies: pkg.Dependencies, - Arch: b.Arch.ToAPK(), + Arch: b.PackageArch(), Options: pkg.Options, Scriptlets: pkg.Scriptlets, Description: pkg.Description, diff --git a/pkg/build/test.go b/pkg/build/test.go index 3c2ffeefa..69fd2aaf9 100644 --- a/pkg/build/test.go +++ b/pkg/build/test.go @@ -73,6 +73,39 @@ type Test struct { DefaultTimeout time.Duration } +// resolveConfiguration loads the test configuration without performing setup +// that depends on the selected architecture or runner. +func (t *Test) resolveConfiguration(ctx context.Context) error { + parsedCfg, err := config.ParseConfiguration(ctx, t.ConfigFile, + config.WithEnvFilesForParsing(t.EnvFiles), + config.WithDefaultCPU(t.DefaultCPU), + config.WithDefaultCPUModel(t.DefaultCPUModel), + config.WithDefaultDisk(t.DefaultDisk), + config.WithDefaultMemory(t.DefaultMemory), + config.WithDefaultTimeout(t.DefaultTimeout), + ) + if err != nil { + return fmt.Errorf("failed to load configuration: %w", err) + } + t.Configuration = *parsedCfg + return nil +} + +// PeekIsNoArchTest resolves just enough configuration to report whether the +// test run targets the noarch architecture, without doing NewTest's heavier setup. +func PeekIsNoArchTest(ctx context.Context, opts ...TestOption) (bool, error) { + t := &Test{} + for _, opt := range opts { + if err := opt(t); err != nil { + return false, err + } + } + if err := t.resolveConfiguration(ctx); err != nil { + return false, err + } + return t.Configuration.Package.IsNoArch(), nil +} + func NewTest(ctx context.Context, opts ...TestOption) (*Test, error) { t := Test{ WorkspaceIgnore: ".melangeignore", @@ -88,6 +121,20 @@ func NewTest(ctx context.Context, opts ...TestOption) (*Test, error) { log := clog.FromContext(ctx).With("arch", t.Arch) ctx = clog.WithLogger(ctx, log) + if err := t.resolveConfiguration(ctx); err != nil { + return nil, err + } + + if t.Configuration.Package.IsNoArch() { + hostArch := apko_types.ParseArchitecture(runtime.GOARCH) + if t.Arch != hostArch { + log.Infof("ignoring requested architecture %s for noarch package; building once using host architecture %s", t.Arch.ToAPK(), hostArch.ToAPK()) + } + t.Arch = hostArch + log = log.With("arch", t.Arch.ToAPK()) + ctx = clog.WithLogger(ctx, log) + } + // If no workspace directory is explicitly requested, create a // temporary directory for it. Otherwise, ensure we are in a // subdir for this specific build context. @@ -110,20 +157,6 @@ func NewTest(ctx context.Context, opts ...TestOption) (*Test, error) { t.WorkspaceDir = tmpdir } - parsedCfg, err := config.ParseConfiguration(ctx, t.ConfigFile, - config.WithEnvFilesForParsing(t.EnvFiles), - config.WithDefaultCPU(t.DefaultCPU), - config.WithDefaultCPUModel(t.DefaultCPUModel), - config.WithDefaultDisk(t.DefaultDisk), - config.WithDefaultMemory(t.DefaultMemory), - config.WithDefaultTimeout(t.DefaultTimeout), - ) - if err != nil { - return nil, fmt.Errorf("failed to load configuration: %w", err) - } - - t.Configuration = *parsedCfg - // Check that we actually can run things in containers. if t.Runner != nil && !t.Runner.TestUsability(ctx) { return nil, fmt.Errorf("unable to run containers using %s, specify --runner and one of %s", t.Runner.Name(), GetAllRunners()) @@ -266,7 +299,7 @@ func (t *Test) TestPackage(ctx context.Context) error { }) // Unless a specific architecture is requests, we run the test for all. - inarchs := len(pkg.TargetArchitecture) == 0 + inarchs := pkg.IsNoArch() || len(pkg.TargetArchitecture) == 0 for _, ta := range pkg.TargetArchitecture { if apko_types.ParseArchitecture(ta) == t.Arch { inarchs = true diff --git a/pkg/build/test_test.go b/pkg/build/test_test.go index b96675be5..858cf113d 100644 --- a/pkg/build/test_test.go +++ b/pkg/build/test_test.go @@ -15,7 +15,9 @@ package build import ( + "context" "fmt" + "os" "path/filepath" "strings" "testing" @@ -31,6 +33,27 @@ import ( "chainguard.dev/melange/pkg/container" ) +func TestPeekIsNoArchTest(t *testing.T) { + for _, tt := range []struct { + name string + targetArch string + want bool + }{ + {name: "noarch", targetArch: "[noarch]", want: true}, + {name: "native", targetArch: "[x86_64]", want: false}, + } { + t.Run(tt.name, func(t *testing.T) { + configFile := filepath.Join(t.TempDir(), "melange.yaml") + contents := fmt.Sprintf("package:\n name: test\n version: 1.0\n epoch: 0\n target-architecture: %s\n", tt.targetArch) + require.NoError(t, os.WriteFile(configFile, []byte(contents), 0o644)) + + got, err := PeekIsNoArchTest(context.Background(), WithTestConfig(configFile)) + require.NoError(t, err) + require.Equal(t, tt.want, got) + }) + } +} + const ( buildUser = "build" etcResolveConf = "/etc/resolv.conf" diff --git a/pkg/cli/build.go b/pkg/cli/build.go index 2912db94e..4fa26cb46 100644 --- a/pkg/cli/build.go +++ b/pkg/cli/build.go @@ -374,6 +374,11 @@ func BuildCmd(ctx context.Context, archs []apko_types.Architecture, baseOpts ... archs = apko_types.AllArchs } + noArch, err := build.PeekIsNoArch(ctx, baseOpts...) + if err != nil { + return err + } + // Set up the build contexts before running them. This avoids various // race conditions and the possibility that a context may be garbage // collected before it is actually run. @@ -381,21 +386,36 @@ func BuildCmd(ctx context.Context, archs []apko_types.Architecture, baseOpts ... // Yes, this happens. Really. // https://github.com/distroless/nginx/runs/7219233843?check_suite_focus=true bcs := []*build.Build{} - for _, arch := range archs { + if noArch { opts := append([]build.Option{}, baseOpts...) - opts = append(opts, build.WithArch(arch)) + opts = append(opts, + build.WithArch(apko_types.ParseArchitecture(runtime.GOARCH)), + build.WithReplicateArchs(archs), + ) bc, err := build.New(ctx, opts...) - if errors.Is(err, build.ErrSkipThisArch) { - log.Warnf("skipping arch %s", arch) - continue - } else if err != nil { + if err != nil { return err } defer bc.Close(ctx) - bcs = append(bcs, bc) + } else { + for _, arch := range archs { + opts := append([]build.Option{}, baseOpts...) + opts = append(opts, build.WithArch(arch)) + + bc, err := build.New(ctx, opts...) + if errors.Is(err, build.ErrSkipThisArch) { + log.Warnf("skipping arch %s", arch) + continue + } else if err != nil { + return err + } + + defer bc.Close(ctx) + bcs = append(bcs, bc) + } } if len(bcs) == 0 { diff --git a/pkg/cli/test.go b/pkg/cli/test.go index 868f30fc6..38bce47e9 100644 --- a/pkg/cli/test.go +++ b/pkg/cli/test.go @@ -16,9 +16,9 @@ package cli import ( "context" - "errors" "fmt" "os" + "runtime" "strings" "time" @@ -197,6 +197,11 @@ func TestCmd(ctx context.Context, archs []apko_types.Architecture, baseOpts ...b archs = apko_types.AllArchs } + noArch, err := build.PeekIsNoArchTest(ctx, baseOpts...) + if err != nil { + return err + } + // Set up the test contexts before running them. This avoids various // race conditions and the possibility that a context may be garbage // collected before it is actually run. @@ -204,21 +209,30 @@ func TestCmd(ctx context.Context, archs []apko_types.Architecture, baseOpts ...b // Yes, this happens. Really. // https://github.com/distroless/nginx/runs/7219233843?check_suite_focus=true bcs := []*build.Test{} - for _, arch := range archs { + if noArch { opts := make([]build.TestOption, 0, len(baseOpts)+1) - opts = append(opts, build.WithTestArch(arch)) opts = append(opts, baseOpts...) + opts = append(opts, build.WithTestArch(apko_types.ParseArchitecture(runtime.GOARCH))) bc, err := build.NewTest(ctx, opts...) - if errors.Is(err, build.ErrSkipThisArch) { - log.Infof("skipping arch %s", arch) - continue - } else if err != nil { + if err != nil { return err } defer bc.Close() - bcs = append(bcs, bc) + } else { + for _, arch := range archs { + opts := make([]build.TestOption, 0, len(baseOpts)+1) + opts = append(opts, build.WithTestArch(arch)) + opts = append(opts, baseOpts...) + + bc, err := build.NewTest(ctx, opts...) + if err != nil { + return err + } + defer bc.Close() + bcs = append(bcs, bc) + } } if len(bcs) == 0 { diff --git a/pkg/config/config.go b/pkg/config/config.go index fa96f4edd..e42eeca22 100644 --- a/pkg/config/config.go +++ b/pkg/config/config.go @@ -282,6 +282,12 @@ func (p Package) PackageURLForSubpackage(distro, arch, subpackage string) *purl. return newAPKPackageURL(distro, subpackage, p.FullVersion(), arch) } +// IsNoArch reports whether the package targets the architecture-independent +// noarch architecture. +func (p Package) IsNoArch() bool { + return len(p.TargetArchitecture) == 1 && p.TargetArchitecture[0] == "noarch" +} + func newAPKPackageURL(distro, name, version, arch string) *purl.PackageURL { u := &purl.PackageURL{ Type: purlTypeAPK, @@ -1731,6 +1737,9 @@ func ParseConfiguration(ctx context.Context, configurationFilePath string, opts replacer := replacerFromMap(configMap) cfg.Package = replacePackage(replacer, options.commit, cfg.Package) + if slices.Contains(cfg.Package.TargetArchitecture, "noarch") && !cfg.Package.IsNoArch() { + return nil, fmt.Errorf("target-architecture: %q must be the only entry when specified", "noarch") + } cfg.Pipeline = replacePipelines(replacer, cfg.Pipeline) diff --git a/pkg/config/config_test.go b/pkg/config/config_test.go index a4d739cad..717d918d4 100644 --- a/pkg/config/config_test.go +++ b/pkg/config/config_test.go @@ -16,6 +16,45 @@ import ( "chainguard.dev/melange/pkg/sbom" ) +func TestNoArchTargetArchitecture(t *testing.T) { + ctx := slogtest.Context(t) + + tests := []struct { + name string + targetArch string + wantParseErr bool + wantNoArch bool + }{ + { + name: "noarch", + targetArch: "[noarch]", + wantNoArch: true, + }, + { + name: "noarch with another architecture", + targetArch: "[noarch, x86_64]", + wantParseErr: true, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + path := filepath.Join(t.TempDir(), "melange.yaml") + yaml := []byte("package:\n name: test-package\n version: 1.0.0\n epoch: 0\n target-architecture: " + tt.targetArch + "\n") + require.NoError(t, os.WriteFile(path, yaml, 0o644)) + + cfg, err := ParseConfiguration(ctx, path) + if tt.wantParseErr { + require.ErrorContains(t, err, `target-architecture: "noarch" must be the only entry when specified`) + return + } + + require.NoError(t, err) + require.Equal(t, tt.wantNoArch, cfg.Package.IsNoArch()) + }) + } +} + func Test_validateCPE(t *testing.T) { cases := []struct { name string diff --git a/pkg/index/index.go b/pkg/index/index.go index 9a22fc607..cb6715eec 100644 --- a/pkg/index/index.go +++ b/pkg/index/index.go @@ -98,8 +98,9 @@ func WithSigningKey(signingKey string) Option { } } -// WithExpectedArch sets the expected package architecture. Any packages with -// an unexpected architecture will not be indexed. +// WithExpectedArch sets the expected package architecture. Packages with an +// unexpected architecture will not be indexed, except architecture-independent +// noarch packages, which are valid for every architecture. func WithExpectedArch(expectedArch string) Option { return func(idx *Index) error { idx.ExpectedArch = expectedArch @@ -107,6 +108,10 @@ func WithExpectedArch(expectedArch string) Option { } } +func matchesExpectedArch(pkgArch, expectedArch string) bool { + return expectedArch == "" || pkgArch == "noarch" || pkgArch == expectedArch +} + func New(opts ...Option) (*Index, error) { idx := Index{ PackageFiles: []string{}, @@ -175,12 +180,20 @@ func (idx *Index) UpdateIndex(ctx context.Context) error { return fmt.Errorf("failed to parse package %s: %w", apkFile, err) } - if idx.ExpectedArch != "" && pkg.Arch != idx.ExpectedArch { + if !matchesExpectedArch(pkg.Arch, idx.ExpectedArch) { log.Warnf("%s-%s: found unexpected architecture %s, expecting %s", pkg.Name, pkg.Version, pkg.Arch, idx.ExpectedArch) return nil } + // A noarch package is valid in every architecture-specific repository, + // but this index entry describes one concrete repository architecture. + // Stamp that architecture in the in-memory entry without changing the + // package's embedded .PKGINFO, which must remain arch = noarch. + if pkg.Arch == "noarch" && idx.ExpectedArch != "" { + pkg.Arch = idx.ExpectedArch + } + packages[i] = pkg return nil diff --git a/pkg/index/index_test.go b/pkg/index/index_test.go index 162ae6a01..0a23396e3 100644 --- a/pkg/index/index_test.go +++ b/pkg/index/index_test.go @@ -16,6 +16,29 @@ import ( "github.com/google/go-cmp/cmp" ) +func TestMatchesExpectedArch(t *testing.T) { + tests := []struct { + name string + pkgArch string + expectedArch string + want bool + }{ + {name: "matching architecture", pkgArch: "x86_64", expectedArch: "x86_64", want: true}, + {name: "mismatched architecture", pkgArch: "aarch64", expectedArch: "x86_64", want: false}, + {name: "noarch matches x86_64", pkgArch: "noarch", expectedArch: "x86_64", want: true}, + {name: "noarch matches aarch64", pkgArch: "noarch", expectedArch: "aarch64", want: true}, + {name: "no expected architecture", pkgArch: "aarch64", want: true}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := matchesExpectedArch(tt.pkgArch, tt.expectedArch); got != tt.want { + t.Errorf("matchesExpectedArch(%q, %q) = %t, want %t", tt.pkgArch, tt.expectedArch, got, tt.want) + } + }) + } +} + func TestUpdateIndex(t *testing.T) { ctx := slogtest.Context(t) @@ -59,6 +82,13 @@ func TestUpdateIndex(t *testing.T) { } func mangleApk(t *testing.T, newDesc string) string { + t.Helper() + return mangleApkControl(t, func(b []byte) []byte { + return bytes.ReplaceAll(b, []byte("POSIX 1003.1e capabilities"), []byte(newDesc)) + }) +} + +func mangleApkControl(t *testing.T, rewrite func([]byte) []byte) string { t.Helper() file, err := os.Open(filepath.Join("..", "sca", "testdata", "libcap-2.69-r0.apk")) if err != nil { @@ -80,7 +110,7 @@ func mangleApk(t *testing.T, newDesc string) string { t.Fatal(err) } - b = bytes.ReplaceAll(b, []byte("POSIX 1003.1e capabilities"), []byte(newDesc)) + b = rewrite(b) data, err := os.Open(exp.PackageFile) if err != nil { @@ -119,6 +149,39 @@ func mangleApk(t *testing.T, newDesc string) string { return f.Name() } +func TestUpdateIndexStampsNoArch(t *testing.T) { + ctx := slogtest.Context(t) + filename := mangleApkControl(t, func(b []byte) []byte { + return bytes.Replace(b, []byte("arch = aarch64"), []byte("arch = noarch"), 1) + }) + + idx, err := New( + WithPackageFiles([]string{filename}), + WithExpectedArch("x86_64"), + ) + if err != nil { + t.Fatal(err) + } + if err := idx.UpdateIndex(ctx); err != nil { + t.Fatal(err) + } + + if got, want := idx.Index.Packages[0].Arch, "x86_64"; got != want { + t.Fatalf("UpdateIndex() stamped arch %q, want %q", got, want) + } + + withoutExpected, err := New(WithPackageFiles([]string{filename})) + if err != nil { + t.Fatal(err) + } + if err := withoutExpected.UpdateIndex(ctx); err != nil { + t.Fatal(err) + } + if got, want := withoutExpected.Index.Packages[0].Arch, "noarch"; got != want { + t.Fatalf("UpdateIndex() without expected arch changed arch to %q, want %q", got, want) + } +} + func TestMergeIndex(t *testing.T) { ctx := slogtest.Context(t) newDesc := "This should replace the existing description" diff --git a/pkg/linter/linter.go b/pkg/linter/linter.go index 47d915bd4..b080ff2db 100644 --- a/pkg/linter/linter.go +++ b/pkg/linter/linter.go @@ -194,6 +194,11 @@ var linterMap = map[string]linter{ Explain: "This package contains binaries compiled for unsupported architectures (only aarch64/arm64 and amd64/x86_64 binaries are supported)", defaultBehavior: Warn, }, + "noarch": { + LinterFunc: linters.NoArchLinter, + Explain: "Remove compiled binaries from this noarch package, or remove the noarch designation if it needs to ship architecture-specific code", + defaultBehavior: Require, + }, "staticarchive": { LinterFunc: linters.StaticArchiveLinter, Explain: "This package contains static archives (.a files)", diff --git a/pkg/linter/linter_test.go b/pkg/linter/linter_test.go index 396d41b75..c1f0c1444 100644 --- a/pkg/linter/linter_test.go +++ b/pkg/linter/linter_test.go @@ -15,6 +15,9 @@ package linter import ( + "bytes" + "debug/elf" + "encoding/binary" "encoding/json" "fmt" "io/fs" @@ -44,6 +47,33 @@ func TestLinters(t *testing.T) { } } + mkelf := func(t *testing.T, path string) func() string { + return func() string { + d := t.TempDir() + full := filepath.Join(d, path) + assert.NoError(t, os.MkdirAll(filepath.Dir(full), 0o755)) + + var hdr elf.Header64 + copy(hdr.Ident[:], []byte{0x7f, 'E', 'L', 'F', 2, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0}) + hdr.Type = uint16(elf.ET_EXEC) + hdr.Machine = uint16(elf.EM_X86_64) + hdr.Version = uint32(elf.EV_CURRENT) + hdr.Ehsize = 64 + + var buf bytes.Buffer + assert.NoError(t, binary.Write(&buf, binary.LittleEndian, &hdr)) + assert.NoError(t, os.WriteFile(full, buf.Bytes(), 0o755)) + return d + } + } + + noarchCfg := &config.Configuration{ + Package: config.Package{ + Name: "noarch-pkg", + TargetArchitecture: []string{"noarch"}, + }, + } + cfg := &config.Configuration{ Package: config.Package{ Name: "pkgconf", @@ -517,6 +547,25 @@ func TestLinters(t *testing.T) { }, linter: "duplicate", pass: true, // LICENSE files should always be ignored + }, { + dirFunc: mkelf(t, "usr/bin/mybinary"), + linter: "noarch", + cfg: noarchCfg, + pass: false, + }, { + dirFunc: mkelf(t, "usr/bin/mybinary"), + linter: "noarch", + pass: true, + }, { + dirFunc: mkelf(t, "usr/bin/mybinary"), + linter: "noarch", + cfg: cfg, // non-nil, non-noarch config: linter must still no-op + pass: true, + }, { + dirFunc: mkfile(t, "usr/share/data.txt"), + linter: "noarch", + cfg: noarchCfg, + pass: true, }, { dirFunc: mkfile(t, "usr/lib/i386/libfoo.so"), linter: "unsupportedarch", diff --git a/pkg/linter/linters/noarch.go b/pkg/linter/linters/noarch.go new file mode 100644 index 000000000..1bc0b535e --- /dev/null +++ b/pkg/linter/linters/noarch.go @@ -0,0 +1,98 @@ +// Copyright 2025 Chainguard, Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package linters + +import ( + "bytes" + "context" + "fmt" + "io" + "io/fs" + "path/filepath" + + "chainguard.dev/melange/pkg/config" + "chainguard.dev/melange/pkg/linter/types" +) + +// NoArchLinter rejects compiled ELF files from architecture-independent packages. +func NoArchLinter(ctx context.Context, cfg *config.Configuration, pkgname string, fsys fs.FS) error { + if cfg == nil || !cfg.Package.IsNoArch() { + return nil + } + + var paths []string + err := fs.WalkDir(fsys, ".", func(path string, d fs.DirEntry, err error) error { + if err := ctx.Err(); err != nil { + return err + } + if err != nil { + return err + } + if !d.Type().IsRegular() || IsIgnoredPath(path) { + return nil + } + + info, err := d.Info() + if err != nil { + return err + } + if info.Size() < int64(len(ElfMagic)) { + return nil + } + + ext := filepath.Ext(path) + mode := info.Mode() + if mode&0o111 == 0 && !IsObjectFileRegex.MatchString(ext) { + return nil + } + + f, err := fsys.Open(path) + if err != nil { + return nil + } + defer f.Close() + + readerAt, ok := f.(io.ReaderAt) + if !ok { + return nil + } + + hdr := make([]byte, len(ElfMagic)) + if _, err := readerAt.ReadAt(hdr, 0); err != nil { + return nil + } + if bytes.Equal(ElfMagic, hdr) { + paths = append(paths, path) + } + + return nil + }) + if err != nil { + return err + } + + if len(paths) == 0 { + return nil + } + + word := "binary" + if len(paths) != 1 { + word = "binaries" + } + return types.NewStructuredError( + fmt.Sprintf("%s is a noarch package but contains %d ELF %s", pkgname, len(paths), word), + &types.PathListDetails{Paths: paths}, + ) +}