XNACK Stall vs Replay Benchmark Harness

A benchmark suite for measuring the performance impact of AMD GPU XNACK modes (stall vs replay) on concurrent workloads using managed memory.

Background

AMD GPUs support three XNACK (translation not-acknowledged) modes for handling page faults:

Mode	Behavior	Context Switch During Fault
Replay	CU sends wave to context-switch, retries later	Yes
Stall	CU blocked until fault resolved	No
Fatal	Process killed on any fault	N/A

Some GPU generations may support only XNACK stall mode. This benchmark quantifies the impact:

• Deadlock risk when multiple streams contend for CUs under memory pressure
• Throughput degradation of clean kernels when a concurrent kernel is stalled on page faults
• CU utilization loss as a function of CU allocation ratio between faulting and non-faulting workloads
• Fault resolution latency amplification when pages must be fetched from host memory or swap

Building

Requires ROCm with HIP support. Tested on ROCm 6.x+ and 7.x.

cmake -B build -S . -DGPU_TARGETS="gfx942;gfx1250"
cmake --build build

Optional: enable embedded rocprofiler-sdk profiling for page fault event capture:

cmake -B build -S . -DXNACK_BENCH_EMBED_PROFILER=ON

Usage

Quick start

# Device detection only
./build/xnack-bench --scenario 0

# Run a single scenario
HSA_XNACK=1 ./build/xnack-bench --scenario 1 --verbose

# Run all scenarios
HSA_XNACK=1 ./build/xnack-bench --scenario all --output results/xnack_on.json

Full comparison (XNACK off vs on)

./scripts/run_benchmark.sh ./build/xnack-bench ./results all

This runs the benchmark under both HSA_XNACK=0 and HSA_XNACK=1, optionally runs rocprof KFD tracing, and generates a comparison report.

CLI options

  --scenario <0|1|2|3|4|all>  Scenario to run (default: all)
        0 = device detection only
        1 = two-stream contention
        2 = memory pressure / oversubscription
        3 = CU occupancy sweep
        4 = long fault latency
        all = run scenarios 1-4
  --size <MB>            Data size in MB (default: 256)
  --iterations <N>       Measurement iterations (default: 5)
  --warmup <N>           Warmup iterations (default: 2)
  --device <id>          GPU device ID (default: 0)
  --partition <method>   cu_mask, occupancy, or auto (default: auto)
  --ratios <r1,r2,...>   CU ratios for sweep (default: 50,60,75,90,95)
  --output <file.json>   Output file (default: stdout)
  --detect-arch          Print GPU arch and exit
  --verbose              Verbose output

Scenarios

Scenario 1: Two-Stream Contention

Two concurrent kernels on separate streams with 50/50 CU split:

• Stream 1: Faulting kernel reading managed memory (pages not prefetched)
• Stream 2: Clean kernel operating on device memory (no faults)

What it measures: Stream 2 throughput degradation caused by Stream 1's page faults. Under replay mode, Stream 1's faulted waves are context-switched out, freeing CUs. Under stall mode, CUs remain blocked.

Scenario 2: Memory Pressure / Oversubscription

Two streams access overlapping managed memory regions totaling 120% of VRAM:

• Forces page migrations and evictions between host and device
• Under stall mode, eviction requires context-switching the stalled wave — which is blocked
• Includes a 60-second watchdog for deadlock detection

What it measures: Whether the system can complete under memory pressure, or deadlocks/timeouts.

Scenario 3: CU Occupancy Sweep

Sweeps CU allocation ratios between faulting and clean kernels:

• Ratios: 50/50, 60/40, 75/25, 90/10, 95/5
• Solo baselines (100/0 and 0/100) for reference

What it measures: How clean kernel throughput degrades as a function of how many CUs are allocated to the faulting workload.

Scenario 4: Long Fault Latency

Amplifies fault resolution time by:

1. Prefetching managed memory to device (warm)
2. Evicting back to host
3. Using madvise(MADV_PAGEOUT) to push pages to swap

What it measures: Impact of slow fault resolution (disk I/O) on CU blocking duration and clean kernel throughput.

CU Partitioning

Two strategies for splitting CUs between streams:

Strategy	API	Precision	Portability
CU Mask	hipExtStreamCreateWithCUMask	Exact, exclusive CU assignment	AMD-specific
Occupancy	Grid/block size scaling	Approximate	Portable

Auto-detection probes CU mask support at startup and falls back to occupancy-based partitioning.

WGP granularity (gfx10xx+): CU masks are rounded to 2-CU boundaries since one Work Group Processor encompasses 2 CUs.

Output Format

JSON output with per-scenario timing statistics (mean/min/max/stddev), page fault counts, and profiling data. See results/ for examples after a run.

The comparison script produces ASCII tables:

Scenario                       Metric                    XNACK Off          XNACK On        Delta    Delta%
--------------------------------------------------------------------------------------------------------------
two_stream_contention          stream1_kernel_ms    12.34 +/- 0.50    45.67 +/- 2.10      +33.33   +270.2%
                               stream2_kernel_ms     8.10 +/- 0.30    22.45 +/- 1.80      +14.35   +177.2%

XNACK Off Behavior

When HSA_XNACK=0, page faults are fatal. The benchmark handles this by prefetching all managed memory to the device before kernel launch, providing a "no faults" baseline. This is the performance ceiling — what throughput looks like when there are no page faults at all.

Supported GPUs

• gfx942 (MI300 series) — XNACK replay mode
• gfx1201 (MI400 series) — XNACK behavior TBD
• gfx1250 (MI450) — XNACK stall mode (proposed)

Requires HMM (Heterogeneous Memory Management) kernel support and HSA_XNACK=1.