Skip to content

[CI] upload AIPerf Perfetto traces from benchmark workflows - #2155

Merged
zhuyuhua-v merged 3 commits into
mainfrom
mengqing/integrate_trace_in_ci
Sep 8, 2026
Merged

[CI] upload AIPerf Perfetto traces from benchmark workflows#2155
zhuyuhua-v merged 3 commits into
mainfrom
mengqing/integrate_trace_in_ci

Conversation

@MengqingCao

@MengqingCao MengqingCao commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Convert AIPerf profile_export.jsonl into gzipped Chrome traces after benchmark runs, and upload them as workflow artifacts with Job Summary download links for Perfetto.
  • Share the converter through .github/actions/upload-aiperf-traces and wire it into ATOMesh, native ATOM, vLLM, and SGLang benchmark workflows.
  • Conversion is best-effort: missing JSONL is a no-op, conversion errors warn and do not fail the benchmark.

Motivation

Agentic AIPerf runs already write per-request profile_export.jsonl, but inspecting concurrency slots / TTFT / wait∩prefill currently requires a local conversion step. CI should produce a Perfetto-ready trace and a download link on the run page.

Native / vLLM / SGLang benches today use bench_serving summary JSON, so they will skip this step until they emit profile_export.jsonl. The shared action is already in those workflows so no extra wiring is needed later.

Test plan

  • Mark the PR ready (or workflow_dispatch Atomesh Benchmark) and run one GLM agentic case, e.g. glm-52-mxfp4-1p1d-tp4-agentic-1m-c1
  • Confirm the job uploads aiperf-chrome-traces-* and the Job Summary has a Perfetto download link
  • Download .trace.json.gz, open in https://ui.perfetto.dev (Open trace file), and check slot / TTFT / prefill+decode tracks

Test result

test passed in https://github.com/ROCm/ATOM/actions/runs/34118300963/job/101730333679, the trace could be uploaded and downloaded normally.

Submission Checklist

@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2155 --add-label <label>

MengqingCao and others added 2 commits September 7, 2026 11:40
Convert agentic profile_export.jsonl files into Chrome traces and attach them as downloadable workflow artifacts for inspection in Perfetto.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@MengqingCao
MengqingCao force-pushed the mengqing/integrate_trace_in_ci branch from e1bf971 to 3e1179d Compare September 7, 2026 11:44
@MengqingCao
MengqingCao marked this pull request as ready for review September 8, 2026 08:14

@Jasen2201 Jasen2201 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhuyuhua-v
zhuyuhua-v merged commit 19a64f3 into main Sep 8, 2026
48 of 73 checks passed
@zhuyuhua-v
zhuyuhua-v deleted the mengqing/integrate_trace_in_ci branch September 8, 2026 08:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants