Skip to content

Use Fusion trace metrics to replace bash command-trace wrapper#7041

Merged
pditommaso merged 4 commits intomasterfrom
fusion-trace-metrics
Apr 20, 2026
Merged

Use Fusion trace metrics to replace bash command-trace wrapper#7041
pditommaso merged 4 commits intomasterfrom
fusion-trace-metrics

Conversation

@pditommaso
Copy link
Copy Markdown
Member

Summary

  • When Fusion is enabled, use metrics from .fusion/trace.json (proc, cgroup, gpu sections) to populate TraceRecord fields, replacing the bash command-trace.txt wrapper script
  • Prefer cgroup memory metrics over proc-level metrics for more accurate container-aware measurements (vmem, rss, peak_vmem, peak_rss, %mem)
  • Controlled via NXF_FUSION_TRACE env variable (defaults to true); set to false to revert to legacy behavior
  • GPU metrics extraction is preserved regardless of the setting (backward compatible)

Changes

File Change
Global.groovy Add isFusionTraceEnabled() reading NXF_FUSION_TRACE env var
TraceRecord.groovy parseFusionTraceFile returns full JSON; add toLong/toFloat helpers
TaskHandler.groovy applyFusionMetrics maps Fusion JSON to TraceRecord; skip .command.trace when active
BashWrapperBuilder.groovy Skip trace wrapper generation when Fusion trace is active
TaskBean.groovy Add fusionEnabled field
TaskProcessor.groovy Expose isFusionEnabled()

Trace field mapping

TraceRecord field Fusion source Notes
realtime proc.realtime Milliseconds
%cpu proc.pct_cpu Tenths of percent, divided by 10
cpu_model proc.cpu_name String
rchar, wchar proc.rchar, proc.wchar Bytes
syscr, syscw proc.syscr, proc.syscw Count
read_bytes, write_bytes proc.read_bytes, proc.write_bytes Bytes
vol_ctxt, inv_ctxt proc.vol_ctxt, proc.inv_ctxt Count
vmem cgroup.memory_current Bytes (cgroup preferred)
rss cgroup.memory_rss Bytes (cgroup preferred)
peak_vmem cgroup.memory_peak Bytes (cgroup preferred)
peak_rss cgroup.memory_peak_rss Bytes (cgroup preferred)
%mem computed memory_rss / memory_limit * 100

Test plan

  • TraceRecordTest — parse full JSON, parse without gpu
  • TaskHandlerTest — full metrics with cgroup, GPU-only when disabled, skip .command.trace when active
  • BashWrapperBuilderTest — skip trace wrapper when active, keep when disabled
  • CrgExecutorTest — no regression from fusionEnabled on TaskBean

🤖 Generated with Claude Code

When Fusion filesystem is enabled, task metrics (CPU, memory, I/O) are
collected by the Fusion client and written to .fusion/trace.json. This
change extends Nextflow to use those metrics directly, replacing the
bash command-trace.txt wrapper script.

Key changes:
- TraceRecord.parseFusionTraceFile now returns the full parsed JSON
- TaskHandler.applyFusionMetrics maps proc/cgroup/gpu sections to
  TraceRecord fields, preferring cgroup memory metrics over proc
- BashWrapperBuilder skips the trace wrapper when Fusion trace is active
- TaskHandler.getTraceRecord skips .command.trace parsing accordingly
- Global.isFusionTraceEnabled() controls the feature via NXF_FUSION_TRACE
  env variable (defaults to true)
- TaskProcessor.isFusionEnabled() exposes executor fusion state

Signed-off-by: Paolo Di Tommaso <paolo@seqera.io>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@netlify
Copy link
Copy Markdown

netlify Bot commented Apr 16, 2026

Deploy Preview for nextflow-docs-staging ready!

Name Link
🔨 Latest commit e4e1896
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/69e648b6c993260008407908
😎 Deploy Preview https://deploy-preview-7041--nextflow-docs-staging.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

Comment thread modules/nextflow/src/main/groovy/nextflow/processor/TaskHandler.groovy Outdated
@jordeu jordeu self-requested a review April 20, 2026 14:10
pditommaso and others added 3 commits April 20, 2026 17:11
Peak RSS is more representative than the last-sampled memory_rss,
which Fusion overwrites post-exit. Aligns %mem with peak_rss reported
in the trace.

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@pditommaso pditommaso requested a review from a team as a code owner April 20, 2026 15:39
@pditommaso pditommaso merged commit de43766 into master Apr 20, 2026
8 checks passed
@pditommaso pditommaso deleted the fusion-trace-metrics branch April 20, 2026 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants