Skip to content

Honour process.resourceLabels in nf-seqera executor#7048

Merged
pditommaso merged 16 commits intomasterfrom
feat/seqera-resource-labels
Apr 20, 2026
Merged

Honour process.resourceLabels in nf-seqera executor#7048
pditommaso merged 16 commits intomasterfrom
feat/seqera-resource-labels

Conversation

@pditommaso
Copy link
Copy Markdown
Member

Summary

Make the nf-seqera executor honour the standard Nextflow process.resourceLabels directive with cumulative semantics:

  • The config-level process.resourceLabels baseline is attached once to the Sched run via CreateRunRequest.labels.
  • Each task only sends the delta between its effective task.config.getResourceLabels() and the run baseline, on the new Task.labels field. Identical key/value entries are stripped; differing or new entries are sent so the scheduler's task-over-run merge gives the correct final values.
  • Removes the now-redundant seqera.executor.labels config option (breaking change for plugin nf-seqera, replaced entirely by process.resourceLabels).

This fixes a gap where the process.resourceLabels directive — emitted by AbstractComputePlatformProvider.addConfigResourceLabels() for every CE type and honoured by AWS Batch / GCP Batch / Azure / K8s — was effectively dead on the Seqera Compute path.

Motivation

SeqeraTaskHandler.submit() previously built the scheduler Task with only name, image, command, environment, resourceRequirement, resourceLimit, machineRequirement, and nextflow(taskId/hash/workDir) — it never read task.config.getResourceLabels(). The plugin's only label path was at the run level (seqera.executor.labels + autoLabels), so per-process labels emitted via the standard directive were silently dropped on Seqera Compute while working on every other supported executor.

Cumulative model rationale: per-task scheduler payloads stay small (only the delta is sent); same-key collisions between the config baseline and a per-process override resolve naturally on the scheduler side (task labels override run labels), preserving Nextflow's existing directive semantics.

Changes

Plugin (plugins/nf-seqera)

  • Labels.withProcessResourceLabels(Map<String,?>) — new instance method that adds config-level labels to the run-label map, coercing values via String.valueOf to satisfy sched-api's Map<String,String> typing.
  • Labels.toStringMap(Map<String,?>) — static helper returning a coerced map (or Collections.emptyMap() for null/empty input).
  • Labels.delta(task, run) — static helper returning entries in task that are missing from run or have a different value. Returns null when the result would be empty (so callers can omit the field).
  • SeqeraExecutor:
    • new private cached field runResourceLabels (initialised to empty map);
    • new @PackageScope void computeRunResourceLabels() reads session.config.process.resourceLabels and stores the coerced baseline;
    • new public getter getRunResourceLabels();
    • createRun() invokes computeRunResourceLabels() and calls labels.withProcessResourceLabels(runResourceLabels) after the autoLabels block.
  • SeqeraTaskHandler.submit() computes Labels.delta(Labels.toStringMap(task.config.getResourceLabels()), executor.runResourceLabels) and attaches it to Task.labels only when non-empty.
  • seqera.executor.labels config option, ExecutorOpts.labels field/getter, and Labels.withUserLabels(Map) removed; obsolete tests dropped from ExecutorOptsTest, LabelsTest, and SeqeraConfigTest.
  • Plugin VERSION bumped to 0.18.0; changelog.txt updated.
  • sched-client bumped to 0.52.0-SNAPSHOT (which exposes Task.labels).

Tests (Spock)

  • LabelsTest covers withProcessResourceLabels (coercion, null/empty, override semantics) plus toStringMap and delta (null/empty handling, full-when-empty-baseline, missing/differing-only deltas).
  • SeqeraExecutorTest adds:
    • isolated test of computeRunResourceLabels() populating the cached map, including non-String value coercion;
    • end-to-end createRun() test asserting CreateRunRequest.labels contains both the config-level process.resourceLabels entries and the nextflow.io/* auto-label entries when autoLabels is enabled.
  • SeqeraTaskHandlerTest:
    • new tests verify submit() attaches Task.labels containing only the delta and leaves Task.labels unset when task labels equal the run baseline;
    • existing Mock(SeqeraExecutor) blocks tightened with getRunResourceLabels() >> [:] to reflect the runtime invariant.

Docs

  • docs/reference/process.md adds seqera-executor to the resourceLabels support list.

Breaking change

The seqera.executor.labels config option is removed. Users on plugin nf-seqera 0.17.x must migrate to the standard Nextflow directive:

// before
seqera.executor.labels = [project: 'genomics', team: 'research']

// after
process.resourceLabels = [project: 'genomics', team: 'research']

The directive may be set at the config level (applied to all processes), under withName:/withLabel: selectors, or inside individual process {} bodies — all forms accumulate per Nextflow's normal merging rules. The config-level entries become the run baseline; selector/process-body entries become per-task deltas.

Test plan

  • ./gradlew :plugins:nf-seqera:test — green
  • Spec coverage cross-checked against the design doc (docs/superpowers/specs/2026-04-17-seqera-resource-labels-design.md)
  • CI reruns the full plugin suite after the new sched-client:0.52.0-SNAPSHOT artifact resolves from the published Maven repo
  • Smoke run on a real Sched-backed compute env with a workflow that combines a config-level process.resourceLabels baseline plus per-process directives, verifying the resulting run labels and per-task labels in the scheduler

Notes

  • Design and implementation plan committed under docs/superpowers/{specs,plans}/2026-04-17-seqera-resource-labels-*.md for review traceability.
  • settings.gradle retains an opt-in commented includeBuild '../sched' hint for local development against an unreleased sched-client; CI builds use the published Maven artifact.

pditommaso and others added 12 commits April 17, 2026 20:07
Spec for honouring process.resourceLabels in nf-seqera with cumulative
semantics: config-level baseline on the Sched run, per-task delta on
Sched task. Removes the now-redundant seqera.executor.labels option.

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Task-by-task TDD plan implementing the spec from
docs/superpowers/specs/2026-04-17-seqera-resource-labels-design.md.

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
…ess.resourceLabels

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Picks up the published artifact so the build resolves without the
includeBuild composite. The transitive sched-api 0.52.0-SNAPSHOT is
now also published.

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@pditommaso pditommaso requested a review from a team as a code owner April 17, 2026 19:49
@netlify
Copy link
Copy Markdown

netlify Bot commented Apr 17, 2026

Deploy Preview for nextflow-docs-staging ready!

Name Link
🔨 Latest commit d083525
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/69e6583866697c00096d635c
😎 Deploy Preview https://deploy-preview-7048--nextflow-docs-staging.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
- Labels.toStringMap now accepts Object and throws IllegalArgumentException
  when the value is not a Map, giving a clear error when
  process.resourceLabels is misconfigured (e.g. as a list).
- SeqeraExecutor.getRunResourceLabels wraps the cached map in
  Collections.unmodifiableMap, matching Labels.getEntries.

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@bentsherman
Copy link
Copy Markdown
Member

Don't forget to remove seqera.executor.labels from docs/reference/config.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@pditommaso
Copy link
Copy Markdown
Member Author

Well spotted, done

@pditommaso pditommaso merged commit 979f684 into master Apr 20, 2026
9 checks passed
@pditommaso pditommaso deleted the feat/seqera-resource-labels branch April 20, 2026 16:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants