Skip to content

[bugfix] take start time of all ranks into account for performance measurements - #772

Merged
avjves merged 1 commit into
xdit-project:mainfrom
Arech8:arech/fix_benchmarking
Sep 8, 2026
Merged

[bugfix] take start time of all ranks into account for performance measurements#772
avjves merged 1 commit into
xdit-project:mainfrom
Arech8:arech/fix_benchmarking

Conversation

@Arech8

@Arech8 Arech8 commented Sep 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Current implementation measures the latency of each rank in isolation and then returns the latency of the last rank as the latency of the whole model execution. This is based on 2 assumptions:

  1. the last rank start the model execution.
  2. the last rank produces the output user cares about.

While the second assumption usually holds (with exception of SD3.5 and possibly other DP models; And it is not asserted, which is problematic), the first assumption is completely broken.

This change creates a DP replica group of ranks and places a barrier for all ranks participating the group before start of the execution, so all ranks start as closely to each other as possible, and then the latency of current rank is measured and is returned, which for the last rank (following the second assumption) yields the best estimate of the model run time that user cares about.

Implementation on the replica-scale barrier is simple, but not ideal. The problem is, making anything better seems to require a perfect clock synchronization between ranks, and an API update to return timestamps of torch.cuda events instead of providing only elapsed_time(start).

Effect

Note: this was measured using the previous iteration of the changeset based on the world-scale barrier, which also corresponded to a replica scale barrier since there were not a case with DP>1.

Effect of the change was measured 3 times using the following 8 configurations

    - flux.usp
    - flux2.default
    - flux2.quantgemm.gfx942
    - flux2.quantgemm.gfx950
    - stablediffusion_3_5.default
    - z_image.default
    - qwen_image.default
    - wan2_2.quantgemm_fp8attn.gfx950

on 8 x MI355x and compared to the original implementation and to an implementation collecting the longest across all ranks latency. Overall, the original implementation very slightly underestimates the run time for all these configurations with the exception of SD3.5, where underestimation is more than 4-5+%.

Full performance report 
Benchmark comparison results ([Brunner Munzel test](https://en.wikipedia.org/wiki/Brunner_Munzel_Test), alpha=0.00100)                                                         
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                                  Benchmark                                   ┃                                  real_time (means), [0%, 100%]                                  ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ flux.usp/a/res1 | 00_original vs 10_full_timings                             │ 726.4ms < 732.1ms {+0.8%} [720.6m,748.2m] < [725.1m,826.8m] {+0.6%,+10.5%} p=0.00005 (30 vs 30) │
│ flux.usp/a/res1 | 00_original vs 20_barrier_start                            │ 726.4ms < 729.0ms {+0.4%} [720.6m,748.2m] < [723.9m,732.6m] {+0.4%,-2.1%} p=0.00000+(30 vs 30)  │
│ flux.usp/a/res1 | 10_full_timings vs 20_barrier_start                        │ 732.1ms ~ 729.0ms {-0.4%} [725.1m,826.8m] ~ [723.9m,732.6m] {-0.2%,-11.4%} p=0.05980 (30 vs 30) │
│ flux.usp/a/res2 | 00_original vs 10_full_timings                             │ 728.4ms ~ 730.9ms {+0.3%} [721.7m,753.2m] ~ [725.3m,827.3m] {+0.5%,+9.8%} p=0.38502 (30 vs 30)  │
│ flux.usp/a/res2 | 00_original vs 20_barrier_start                            │ 728.4ms ~ 725.6ms {-0.4%} [721.7m,753.2m] ~ [722.3m,729.0m] {+0.1%,-3.2%} p=0.00269 (30 vs 30)  │
│ flux.usp/a/res2 | 10_full_timings vs 20_barrier_start                        │ 730.9ms > 725.6ms {-0.7%} [725.3m,827.3m] > [722.3m,729.0m] {-0.4%,-11.9%} p=0.00000+(30 vs 30) │
│ flux.usp/a/res3 | 00_original vs 10_full_timings                             │ 725.8ms ~ 727.4ms {+0.2%} [720.5m,729.4m] ~ [724.7m,734.6m] {+0.6%,+0.7%} p=0.00629 (30 vs 30)  │
│ flux.usp/a/res3 | 00_original vs 20_barrier_start                            │ 725.8ms ~ 726.5ms {+0.1%} [720.5m,729.4m] ~ [721.4m,746.2m] {+0.1%,+2.3%} p=0.45992 (30 vs 30)  │
│ flux.usp/a/res3 | 10_full_timings vs 20_barrier_start                        │ 727.4ms > 726.5ms {-0.1%} [724.7m,734.6m] > [721.4m,746.2m] {-0.5%,+1.6%} p=0.00099 (30 vs 30)  │
│ flux2.default/a/res1 | 00_original vs 10_full_timings                        │ 3.421s ~ 3.413s {-0.2%} [3.400,3.519] ~ [3.401,3.443] {+0.0%,-2.2%} p=0.03179 (30 vs 30)        │
│ flux2.default/a/res1 | 00_original vs 20_barrier_start                       │ 3.421s ~ 3.418s {-0.1%} [3.400,3.519] ~ [3.403,3.514] {+0.1%,-0.1%} p=0.29636 (30 vs 30)        │
│ flux2.default/a/res1 | 10_full_timings vs 20_barrier_start                   │ 3.413s ~ 3.418s {+0.1%} [3.401,3.443] ~ [3.403,3.514] {+0.0%,+2.1%} p=0.12007 (30 vs 30)        │
│ flux2.default/a/res2 | 00_original vs 10_full_timings                        │ 3.419s ~ 3.418s {-0.0%} [3.398,3.528] ~ [3.399,3.520] {+0.0%,-0.2%} p=0.04789 (30 vs 30)        │
│ flux2.default/a/res2 | 00_original vs 20_barrier_start                       │ 3.419s ~ 3.421s {+0.1%} [3.398,3.528] ~ [3.402,3.530] {+0.1%,+0.1%} p=0.48301 (30 vs 30)        │
│ flux2.default/a/res2 | 10_full_timings vs 20_barrier_start                   │ 3.418s ~ 3.421s {+0.1%} [3.399,3.520] ~ [3.402,3.530] {+0.1%,+0.3%} p=0.00698 (30 vs 30)        │
│ flux2.default/a/res3 | 00_original vs 10_full_timings                        │ 3.420s < 3.424s {+0.1%} [3.400,3.515] < [3.416,3.448] {+0.5%,-1.9%} p=0.00007 (30 vs 30)        │
│ flux2.default/a/res3 | 00_original vs 20_barrier_start                       │ 3.420s ~ 3.429s {+0.3%} [3.400,3.515] ~ [3.406,3.759] {+0.2%,+7.0%} p=0.32078 (30 vs 30)        │
│ flux2.default/a/res3 | 10_full_timings vs 20_barrier_start                   │ 3.424s > 3.429s {+0.1%} [3.416,3.448] > [3.406,3.759] {-0.3%,+9.0%} p=0.00031 (30 vs 30)        │
│ flux2.quantgemm.gfx950/a/res1 | 00_original vs 10_full_timings               │ 2.125s ~ 2.118s {-0.3%} [2.109,2.213] ~ [2.108,2.144] {-0.1%,-3.1%} p=0.00365 (30 vs 30)        │
│ flux2.quantgemm.gfx950/a/res1 | 00_original vs 20_barrier_start              │ 2.125s > 2.112s {-0.6%} [2.109,2.213] > [2.098,2.134] {-0.5%,-3.6%} p=0.00000+(30 vs 30)        │
│ flux2.quantgemm.gfx950/a/res1 | 10_full_timings vs 20_barrier_start          │ 2.118s > 2.112s {-0.3%} [2.108,2.144] > [2.098,2.134] {-0.5%,-0.5%} p=0.00009 (30 vs 30)        │
│ flux2.quantgemm.gfx950/a/res2 | 00_original vs 10_full_timings               │ 2.111s ~ 2.113s {+0.1%} [2.091,2.138] ~ [2.102,2.137] {+0.5%,-0.0%} p=0.13276 (30 vs 30)        │
│ flux2.quantgemm.gfx950/a/res2 | 00_original vs 20_barrier_start              │ 2.111s ~ 2.113s {+0.1%} [2.091,2.138] ~ [2.098,2.214] {+0.4%,+3.5%} p=0.27833 (30 vs 30)        │
│ flux2.quantgemm.gfx950/a/res2 | 10_full_timings vs 20_barrier_start          │ 2.113s ~ 2.113s {-0.0%} [2.102,2.137] ~ [2.098,2.214] {-0.2%,+3.6%} p=0.05941 (30 vs 30)        │
│ flux2.quantgemm.gfx950/a/res3 | 00_original vs 10_full_timings               │ 2.110s < 2.123s {+0.6%} [2.088,2.203] < [2.111,2.218] {+1.1%,+0.7%} p=0.00000+(30 vs 30)        │
│ flux2.quantgemm.gfx950/a/res3 | 00_original vs 20_barrier_start              │ 2.110s < 2.117s {+0.3%} [2.088,2.203] < [2.106,2.137] {+0.8%,-3.0%} p=0.00001 (30 vs 30)        │
│ flux2.quantgemm.gfx950/a/res3 | 10_full_timings vs 20_barrier_start          │ 2.123s ~ 2.117s {-0.3%} [2.111,2.218] ~ [2.106,2.137] {-0.2%,-3.6%} p=0.00705 (30 vs 30)        │
│ qwen_image.default/a/res1 | 00_original vs 10_full_timings                   │ 5.070s < 5.117s {+0.9%} [5.039,5.167] < [5.082,5.224] {+0.8%,+1.1%} p=0.00000+(30 vs 30)        │
│ qwen_image.default/a/res1 | 00_original vs 20_barrier_start                  │ 5.070s < 5.098s {+0.6%} [5.039,5.167] < [5.068,5.214] {+0.6%,+0.9%} p=0.00000+(30 vs 30)        │
│ qwen_image.default/a/res1 | 10_full_timings vs 20_barrier_start              │ 5.117s > 5.098s {-0.4%} [5.082,5.224] > [5.068,5.214] {-0.3%,-0.2%} p=0.00003 (30 vs 30)        │
│ qwen_image.default/a/res2 | 00_original vs 10_full_timings                   │ 5.063s ~ 5.076s {+0.2%} [5.030,5.167] ~ [5.038,5.111] {+0.2%,-1.1%} p=0.00105 (30 vs 30)        │
│ qwen_image.default/a/res2 | 00_original vs 20_barrier_start                  │ 5.063s ~ 5.061s {-0.1%} [5.030,5.167] ~ [5.031,5.173] {+0.0%,+0.1%} p=0.16892 (30 vs 30)        │
│ qwen_image.default/a/res2 | 10_full_timings vs 20_barrier_start              │ 5.076s > 5.061s {-0.3%} [5.038,5.111] > [5.031,5.173] {-0.1%,+1.2%} p=0.00006 (30 vs 30)        │
│ qwen_image.default/a/res3 | 00_original vs 10_full_timings                   │ 5.068s ~ 5.064s {-0.1%} [5.030,5.126] ~ [5.031,5.172] {+0.0%,+0.9%} p=0.10698 (30 vs 30)        │
│ qwen_image.default/a/res3 | 00_original vs 20_barrier_start                  │ 5.068s > 5.048s {-0.4%} [5.030,5.126] > [5.019,5.145] {-0.2%,+0.4%} p=0.00000+(30 vs 30)        │
│ qwen_image.default/a/res3 | 10_full_timings vs 20_barrier_start              │ 5.064s > 5.048s {-0.3%} [5.031,5.172] > [5.019,5.145] {-0.2%,-0.5%} p=0.00041 (30 vs 30)        │
│ stablediffusion_3_5.default/a/res1 | 00_original vs 10_full_timings          │ 786.0ms < 853.3ms {+8.6%} [769.5m,805.4m] < [825.6m,889.5m] {+7.3%,+10.4%} p=0.00000+(30 vs 30) │
│ stablediffusion_3_5.default/a/res1 | 00_original vs 20_barrier_start         │ 786.0ms < 820.4ms {+4.4%} [769.5m,805.4m] < [803.5m,883.4m] {+4.4%,+9.7%} p=0.00000+(30 vs 30)  │
│ stablediffusion_3_5.default/a/res1 | 10_full_timings vs 20_barrier_start     │ 853.3ms > 820.4ms {-3.8%} [825.6m,889.5m] > [803.5m,883.4m] {-2.7%,-0.7%} p=0.00000+(30 vs 30)  │
│ stablediffusion_3_5.default/a/res2 | 00_original vs 10_full_timings          │ 784.8ms < 846.7ms {+7.9%} [773.9m,848.4m] < [819.2m,895.7m] {+5.9%,+5.6%} p=0.00000+(30 vs 30)  │
│ stablediffusion_3_5.default/a/res2 | 00_original vs 20_barrier_start         │ 784.8ms < 839.1ms {+6.9%} [773.9m,848.4m] < [805.6m,919.1m] {+4.1%,+8.3%} p=0.00000+(30 vs 30)  │
│ stablediffusion_3_5.default/a/res2 | 10_full_timings vs 20_barrier_start     │ 846.7ms ~ 839.1ms {-0.9%} [819.2m,895.7m] ~ [805.6m,919.1m] {-1.7%,+2.6%} p=0.04888 (30 vs 30)  │
│ stablediffusion_3_5.default/a/res3 | 00_original vs 10_full_timings          │ 799.0ms < 831.3ms {+4.0%} [771.2m,897.1m] < [820.0m,911.7m] {+6.3%,+1.6%} p=0.00000+(30 vs 30)  │
│ stablediffusion_3_5.default/a/res3 | 00_original vs 20_barrier_start         │ 799.0ms < 833.7ms {+4.3%} [771.2m,897.1m] < [814.3m,865.7m] {+5.6%,-3.5%} p=0.00000+(30 vs 30)  │
│ stablediffusion_3_5.default/a/res3 | 10_full_timings vs 20_barrier_start     │ 831.3ms ~ 833.7ms {+0.3%} [820.0m,911.7m] ~ [814.3m,865.7m] {-0.7%,-5.0%} p=0.13180 (30 vs 30)  │
│ wan2_2.quantgemm_fp8attn.gfx950/b/res1 | 00_original vs 10_full_timings      │ 36.13s ~ 36.14s {+0.0%} [35.97,36.32] ~ [35.98,36.24] {+0.0%,-0.2%} p=0.37708 (25 vs 25)        │
│ wan2_2.quantgemm_fp8attn.gfx950/b/res1 | 00_original vs 20_barrier_start     │ 36.13s ~ 36.14s {+0.0%} [35.97,36.32] ~ [35.94,36.28] {-0.1%,-0.1%} p=0.40624 (25 vs 25)        │
│ wan2_2.quantgemm_fp8attn.gfx950/b/res1 | 10_full_timings vs 20_barrier_start │ 36.14s ~ 36.14s {-0.0%} [35.98,36.24] ~ [35.94,36.28] {-0.1%,+0.1%} p=0.48111 (25 vs 25)        │
│ wan2_2.quantgemm_fp8attn.gfx950/b/res2 | 00_original vs 10_full_timings      │ 36.08s < 36.19s {+0.3%} [35.97,36.24] < [36.05,36.30] {+0.2%,+0.2%} p=0.00000+(25 vs 25)        │
│ wan2_2.quantgemm_fp8attn.gfx950/b/res2 | 00_original vs 20_barrier_start     │ 36.08s ~ 36.08s {-0.0%} [35.97,36.24] ~ [35.99,36.18] {+0.1%,-0.2%} p=0.37088 (25 vs 25)        │
│ wan2_2.quantgemm_fp8attn.gfx950/b/res2 | 10_full_timings vs 20_barrier_start │ 36.19s > 36.08s {-0.3%} [36.05,36.30] > [35.99,36.18] {-0.2%,-0.3%} p=0.00000+(25 vs 25)        │
│ wan2_2.quantgemm_fp8attn.gfx950/b/res3 | 00_original vs 10_full_timings      │ 36.13s < 36.21s {+0.2%} [35.98,36.27] < [36.10,36.33] {+0.3%,+0.2%} p=0.00000+(25 vs 25)        │
│ wan2_2.quantgemm_fp8attn.gfx950/b/res3 | 00_original vs 20_barrier_start     │ 36.13s ~ 36.15s {+0.1%} [35.98,36.27] ~ [35.95,36.86] {-0.1%,+1.6%} p=0.45905 (25 vs 25)        │
│ wan2_2.quantgemm_fp8attn.gfx950/b/res3 | 10_full_timings vs 20_barrier_start │ 36.21s > 36.15s {-0.2%} [36.10,36.33] > [35.95,36.86] {-0.4%,+1.5%} p=0.00035 (25 vs 25)        │
│ z_image.default/a/res1 | 00_original vs 10_full_timings                      │ 2.292s > 2.279s {-0.6%} [2.277,2.313] > [2.263,2.338] {-0.6%,+1.1%} p=0.00000+(30 vs 30)        │
│ z_image.default/a/res1 | 00_original vs 20_barrier_start                     │ 2.292s > 2.276s {-0.7%} [2.277,2.313] > [2.264,2.294] {-0.6%,-0.8%} p=0.00000+(30 vs 30)        │
│ z_image.default/a/res1 | 10_full_timings vs 20_barrier_start                 │ 2.279s ~ 2.276s {-0.1%} [2.263,2.338] ~ [2.264,2.294] {+0.1%,-1.9%} p=0.23783 (30 vs 30)        │
│ z_image.default/a/res2 | 00_original vs 10_full_timings                      │ 2.292s ~ 2.289s {-0.1%} [2.276,2.308] ~ [2.272,2.387] {-0.2%,+3.4%} p=0.00479 (30 vs 30)        │
│ z_image.default/a/res2 | 00_original vs 20_barrier_start                     │ 2.292s > 2.288s {-0.2%} [2.276,2.308] > [2.274,2.396] {-0.1%,+3.8%} p=0.00049 (30 vs 30)        │
│ z_image.default/a/res2 | 10_full_timings vs 20_barrier_start                 │ 2.289s ~ 2.288s {-0.0%} [2.272,2.387] ~ [2.274,2.396] {+0.1%,+0.4%} p=0.17478 (30 vs 30)        │
│ z_image.default/a/res3 | 00_original vs 10_full_timings                      │ 2.292s > 2.282s {-0.4%} [2.274,2.394] > [2.271,2.310] {-0.1%,-3.5%} p=0.00054 (30 vs 30)        │
│ z_image.default/a/res3 | 00_original vs 20_barrier_start                     │ 2.292s ~ 2.289s {-0.1%} [2.274,2.394] ~ [2.277,2.387] {+0.1%,-0.3%} p=0.04580 (30 vs 30)        │
│ z_image.default/a/res3 | 10_full_timings vs 20_barrier_start                 │ 2.282s ~ 2.289s {+0.3%} [2.271,2.310] ~ [2.277,2.387] {+0.2%,+3.3%} p=0.01303 (30 vs 30)        │
└──────────────────────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────┘

@nsakkine
nsakkine requested a review from avjves September 3, 2026 09:33
@Arech8
Arech8 force-pushed the arech/fix_benchmarking branch from 4f515a9 to 787af60 Compare September 8, 2026 09:25
avjves
avjves approved these changes Sep 8, 2026
View reviewed changes

@avjves avjves left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM! 🙏

@avjves
avjves merged commit 559ba33 into xdit-project:main Sep 8, 2026
@Arech8
Arech8 deleted the arech/fix_benchmarking branch September 8, 2026 10:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@avjves avjves avjves approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Arech8 @avjves