[bugfix] take start time of all ranks into account for performance measurements

[Arech8]

Current implementation measures the latency of each rank in isolation and then returns the latency of the last rank as the latency of the whole model execution. This is based on 2 assumptions:

1. the last rank start the model execution.
2. the last rank produces the output user cares about.

While the second assumption usually holds (with exception of SD3.5 and possibly other DP models; And it is not asserted, which is problematic), the first assumption is completely broken.

This change places a world-scale barrier for all ranks before start of the execution, so all ranks start as closely to each other as possible, and then the latency of current rank is measured and is returned, which for the last rank (following the second assumption) yields the best estimate of the model run time that user cares about.

Implementation on the world-scale barrier is simple, but not ideal. The problem is, making anything better seems to require a perfect clock synchronization between ranks, and an API update to return timestamps of torch.cuda events instead of providing only elapsed_time(start).

Effect

Effect of the change was measured 3 times using the following 8 configurations

    - flux.usp
    - flux2.default
    - flux2.quantgemm.gfx942
    - flux2.quantgemm.gfx950
    - stablediffusion_3_5.default
    - z_image.default
    - qwen_image.default
    - wan2_2.quantgemm_fp8attn.gfx950

on 8 x MI355x and compared to the original implementation and to an implementation collecting the longest across all ranks latency. Overall, the original implementation very slightly underestimates the run time for all these configurations with the exception of SD3.5, where underestimation is more than 4-5+%.

Full performance report

Benchmark comparison results ([Brunner Munzel test](https://en.wikipedia.org/wiki/Brunner_Munzel_Test), alpha=0.00100)                                                         
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                                  Benchmark                                   ┃                                  real_time (means), [0%, 100%]                                  ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ flux.usp/a/res1 | 00_original vs 10_full_timings                             │ 726.4ms < 732.1ms {+0.8%} [720.6m,748.2m] < [725.1m,826.8m] {+0.6%,+10.5%} p=0.00005 (30 vs 30) │
│ flux.usp/a/res1 | 00_original vs 20_barrier_start                            │ 726.4ms < 729.0ms {+0.4%} [720.6m,748.2m] < [723.9m,732.6m] {+0.4%,-2.1%} p=0.00000+(30 vs 30)  │
│ flux.usp/a/res1 | 10_full_timings vs 20_barrier_start                        │ 732.1ms ~ 729.0ms {-0.4%} [725.1m,826.8m] ~ [723.9m,732.6m] {-0.2%,-11.4%} p=0.05980 (30 vs 30) │
│ flux.usp/a/res2 | 00_original vs 10_full_timings                             │ 728.4ms ~ 730.9ms {+0.3%} [721.7m,753.2m] ~ [725.3m,827.3m] {+0.5%,+9.8%} p=0.38502 (30 vs 30)  │
│ flux.usp/a/res2 | 00_original vs 20_barrier_start                            │ 728.4ms ~ 725.6ms {-0.4%} [721.7m,753.2m] ~ [722.3m,729.0m] {+0.1%,-3.2%} p=0.00269 (30 vs 30)  │
│ flux.usp/a/res2 | 10_full_timings vs 20_barrier_start                        │ 730.9ms > 725.6ms {-0.7%} [725.3m,827.3m] > [722.3m,729.0m] {-0.4%,-11.9%} p=0.00000+(30 vs 30) │
│ flux.usp/a/res3 | 00_original vs 10_full_timings                             │ 725.8ms ~ 727.4ms {+0.2%} [720.5m,729.4m] ~ [724.7m,734.6m] {+0.6%,+0.7%} p=0.00629 (30 vs 30)  │
│ flux.usp/a/res3 | 00_original vs 20_barrier_start                            │ 725.8ms ~ 726.5ms {+0.1%} [720.5m,729.4m] ~ [721.4m,746.2m] {+0.1%,+2.3%} p=0.45992 (30 vs 30)  │
│ flux.usp/a/res3 | 10_full_timings vs 20_barrier_start                        │ 727.4ms > 726.5ms {-0.1%} [724.7m,734.6m] > [721.4m,746.2m] {-0.5%,+1.6%} p=0.00099 (30 vs 30)  │
│ flux2.default/a/res1 | 00_original vs 10_full_timings                        │ 3.421s ~ 3.413s {-0.2%} [3.400,3.519] ~ [3.401,3.443] {+0.0%,-2.2%} p=0.03179 (30 vs 30)        │
│ flux2.default/a/res1 | 00_original vs 20_barrier_start                       │ 3.421s ~ 3.418s {-0.1%} [3.400,3.519] ~ [3.403,3.514] {+0.1%,-0.1%} p=0.29636 (30 vs 30)        │
│ flux2.default/a/res1 | 10_full_timings vs 20_barrier_start                   │ 3.413s ~ 3.418s {+0.1%} [3.401,3.443] ~ [3.403,3.514] {+0.0%,+2.1%} p=0.12007 (30 vs 30)        │
│ flux2.default/a/res2 | 00_original vs 10_full_timings                        │ 3.419s ~ 3.418s {-0.0%} [3.398,3.528] ~ [3.399,3.520] {+0.0%,-0.2%} p=0.04789 (30 vs 30)        │
│ flux2.default/a/res2 | 00_original vs 20_barrier_start                       │ 3.419s ~ 3.421s {+0.1%} [3.398,3.528] ~ [3.402,3.530] {+0.1%,+0.1%} p=0.48301 (30 vs 30)        │
│ flux2.default/a/res2 | 10_full_timings vs 20_barrier_start                   │ 3.418s ~ 3.421s {+0.1%} [3.399,3.520] ~ [3.402,3.530] {+0.1%,+0.3%} p=0.00698 (30 vs 30)        │
│ flux2.default/a/res3 | 00_original vs 10_full_timings                        │ 3.420s < 3.424s {+0.1%} [3.400,3.515] < [3.416,3.448] {+0.5%,-1.9%} p=0.00007 (30 vs 30)        │
│ flux2.default/a/res3 | 00_original vs 20_barrier_start                       │ 3.420s ~ 3.429s {+0.3%} [3.400,3.515] ~ [3.406,3.759] {+0.2%,+7.0%} p=0.32078 (30 vs 30)        │
│ flux2.default/a/res3 | 10_full_timings vs 20_barrier_start                   │ 3.424s > 3.429s {+0.1%} [3.416,3.448] > [3.406,3.759] {-0.3%,+9.0%} p=0.00031 (30 vs 30)        │
│ flux2.quantgemm.gfx950/a/res1 | 00_original vs 10_full_timings               │ 2.125s ~ 2.118s {-0.3%} [2.109,2.213] ~ [2.108,2.144] {-0.1%,-3.1%} p=0.00365 (30 vs 30)        │
│ flux2.quantgemm.gfx950/a/res1 | 00_original vs 20_barrier_start              │ 2.125s > 2.112s {-0.6%} [2.109,2.213] > [2.098,2.134] {-0.5%,-3.6%} p=0.00000+(30 vs 30)        │
│ flux2.quantgemm.gfx950/a/res1 | 10_full_timings vs 20_barrier_start          │ 2.118s > 2.112s {-0.3%} [2.108,2.144] > [2.098,2.134] {-0.5%,-0.5%} p=0.00009 (30 vs 30)        │
│ flux2.quantgemm.gfx950/a/res2 | 00_original vs 10_full_timings               │ 2.111s ~ 2.113s {+0.1%} [2.091,2.138] ~ [2.102,2.137] {+0.5%,-0.0%} p=0.13276 (30 vs 30)        │
│ flux2.quantgemm.gfx950/a/res2 | 00_original vs 20_barrier_start              │ 2.111s ~ 2.113s {+0.1%} [2.091,2.138] ~ [2.098,2.214] {+0.4%,+3.5%} p=0.27833 (30 vs 30)        │
│ flux2.quantgemm.gfx950/a/res2 | 10_full_timings vs 20_barrier_start          │ 2.113s ~ 2.113s {-0.0%} [2.102,2.137] ~ [2.098,2.214] {-0.2%,+3.6%} p=0.05941 (30 vs 30)        │
│ flux2.quantgemm.gfx950/a/res3 | 00_original vs 10_full_timings               │ 2.110s < 2.123s {+0.6%} [2.088,2.203] < [2.111,2.218] {+1.1%,+0.7%} p=0.00000+(30 vs 30)        │
│ flux2.quantgemm.gfx950/a/res3 | 00_original vs 20_barrier_start              │ 2.110s < 2.117s {+0.3%} [2.088,2.203] < [2.106,2.137] {+0.8%,-3.0%} p=0.00001 (30 vs 30)        │
│ flux2.quantgemm.gfx950/a/res3 | 10_full_timings vs 20_barrier_start          │ 2.123s ~ 2.117s {-0.3%} [2.111,2.218] ~ [2.106,2.137] {-0.2%,-3.6%} p=0.00705 (30 vs 30)        │
│ qwen_image.default/a/res1 | 00_original vs 10_full_timings                   │ 5.070s < 5.117s {+0.9%} [5.039,5.167] < [5.082,5.224] {+0.8%,+1.1%} p=0.00000+(30 vs 30)        │
│ qwen_image.default/a/res1 | 00_original vs 20_barrier_start                  │ 5.070s < 5.098s {+0.6%} [5.039,5.167] < [5.068,5.214] {+0.6%,+0.9%} p=0.00000+(30 vs 30)        │
│ qwen_image.default/a/res1 | 10_full_timings vs 20_barrier_start              │ 5.117s > 5.098s {-0.4%} [5.082,5.224] > [5.068,5.214] {-0.3%,-0.2%} p=0.00003 (30 vs 30)        │
│ qwen_image.default/a/res2 | 00_original vs 10_full_timings                   │ 5.063s ~ 5.076s {+0.2%} [5.030,5.167] ~ [5.038,5.111] {+0.2%,-1.1%} p=0.00105 (30 vs 30)        │
│ qwen_image.default/a/res2 | 00_original vs 20_barrier_start                  │ 5.063s ~ 5.061s {-0.1%} [5.030,5.167] ~ [5.031,5.173] {+0.0%,+0.1%} p=0.16892 (30 vs 30)        │
│ qwen_image.default/a/res2 | 10_full_timings vs 20_barrier_start              │ 5.076s > 5.061s {-0.3%} [5.038,5.111] > [5.031,5.173] {-0.1%,+1.2%} p=0.00006 (30 vs 30)        │
│ qwen_image.default/a/res3 | 00_original vs 10_full_timings                   │ 5.068s ~ 5.064s {-0.1%} [5.030,5.126] ~ [5.031,5.172] {+0.0%,+0.9%} p=0.10698 (30 vs 30)        │
│ qwen_image.default/a/res3 | 00_original vs 20_barrier_start                  │ 5.068s > 5.048s {-0.4%} [5.030,5.126] > [5.019,5.145] {-0.2%,+0.4%} p=0.00000+(30 vs 30)        │
│ qwen_image.default/a/res3 | 10_full_timings vs 20_barrier_start              │ 5.064s > 5.048s {-0.3%} [5.031,5.172] > [5.019,5.145] {-0.2%,-0.5%} p=0.00041 (30 vs 30)        │
│ stablediffusion_3_5.default/a/res1 | 00_original vs 10_full_timings          │ 786.0ms < 853.3ms {+8.6%} [769.5m,805.4m] < [825.6m,889.5m] {+7.3%,+10.4%} p=0.00000+(30 vs 30) │
│ stablediffusion_3_5.default/a/res1 | 00_original vs 20_barrier_start         │ 786.0ms < 820.4ms {+4.4%} [769.5m,805.4m] < [803.5m,883.4m] {+4.4%,+9.7%} p=0.00000+(30 vs 30)  │
│ stablediffusion_3_5.default/a/res1 | 10_full_timings vs 20_barrier_start     │ 853.3ms > 820.4ms {-3.8%} [825.6m,889.5m] > [803.5m,883.4m] {-2.7%,-0.7%} p=0.00000+(30 vs 30)  │
│ stablediffusion_3_5.default/a/res2 | 00_original vs 10_full_timings          │ 784.8ms < 846.7ms {+7.9%} [773.9m,848.4m] < [819.2m,895.7m] {+5.9%,+5.6%} p=0.00000+(30 vs 30)  │
│ stablediffusion_3_5.default/a/res2 | 00_original vs 20_barrier_start         │ 784.8ms < 839.1ms {+6.9%} [773.9m,848.4m] < [805.6m,919.1m] {+4.1%,+8.3%} p=0.00000+(30 vs 30)  │
│ stablediffusion_3_5.default/a/res2 | 10_full_timings vs 20_barrier_start     │ 846.7ms ~ 839.1ms {-0.9%} [819.2m,895.7m] ~ [805.6m,919.1m] {-1.7%,+2.6%} p=0.04888 (30 vs 30)  │
│ stablediffusion_3_5.default/a/res3 | 00_original vs 10_full_timings          │ 799.0ms < 831.3ms {+4.0%} [771.2m,897.1m] < [820.0m,911.7m] {+6.3%,+1.6%} p=0.00000+(30 vs 30)  │
│ stablediffusion_3_5.default/a/res3 | 00_original vs 20_barrier_start         │ 799.0ms < 833.7ms {+4.3%} [771.2m,897.1m] < [814.3m,865.7m] {+5.6%,-3.5%} p=0.00000+(30 vs 30)  │
│ stablediffusion_3_5.default/a/res3 | 10_full_timings vs 20_barrier_start     │ 831.3ms ~ 833.7ms {+0.3%} [820.0m,911.7m] ~ [814.3m,865.7m] {-0.7%,-5.0%} p=0.13180 (30 vs 30)  │
│ wan2_2.quantgemm_fp8attn.gfx950/b/res1 | 00_original vs 10_full_timings      │ 36.13s ~ 36.14s {+0.0%} [35.97,36.32] ~ [35.98,36.24] {+0.0%,-0.2%} p=0.37708 (25 vs 25)        │
│ wan2_2.quantgemm_fp8attn.gfx950/b/res1 | 00_original vs 20_barrier_start     │ 36.13s ~ 36.14s {+0.0%} [35.97,36.32] ~ [35.94,36.28] {-0.1%,-0.1%} p=0.40624 (25 vs 25)        │
│ wan2_2.quantgemm_fp8attn.gfx950/b/res1 | 10_full_timings vs 20_barrier_start │ 36.14s ~ 36.14s {-0.0%} [35.98,36.24] ~ [35.94,36.28] {-0.1%,+0.1%} p=0.48111 (25 vs 25)        │
│ wan2_2.quantgemm_fp8attn.gfx950/b/res2 | 00_original vs 10_full_timings      │ 36.08s < 36.19s {+0.3%} [35.97,36.24] < [36.05,36.30] {+0.2%,+0.2%} p=0.00000+(25 vs 25)        │
│ wan2_2.quantgemm_fp8attn.gfx950/b/res2 | 00_original vs 20_barrier_start     │ 36.08s ~ 36.08s {-0.0%} [35.97,36.24] ~ [35.99,36.18] {+0.1%,-0.2%} p=0.37088 (25 vs 25)        │
│ wan2_2.quantgemm_fp8attn.gfx950/b/res2 | 10_full_timings vs 20_barrier_start │ 36.19s > 36.08s {-0.3%} [36.05,36.30] > [35.99,36.18] {-0.2%,-0.3%} p=0.00000+(25 vs 25)        │
│ wan2_2.quantgemm_fp8attn.gfx950/b/res3 | 00_original vs 10_full_timings      │ 36.13s < 36.21s {+0.2%} [35.98,36.27] < [36.10,36.33] {+0.3%,+0.2%} p=0.00000+(25 vs 25)        │
│ wan2_2.quantgemm_fp8attn.gfx950/b/res3 | 00_original vs 20_barrier_start     │ 36.13s ~ 36.15s {+0.1%} [35.98,36.27] ~ [35.95,36.86] {-0.1%,+1.6%} p=0.45905 (25 vs 25)        │
│ wan2_2.quantgemm_fp8attn.gfx950/b/res3 | 10_full_timings vs 20_barrier_start │ 36.21s > 36.15s {-0.2%} [36.10,36.33] > [35.95,36.86] {-0.4%,+1.5%} p=0.00035 (25 vs 25)        │
│ z_image.default/a/res1 | 00_original vs 10_full_timings                      │ 2.292s > 2.279s {-0.6%} [2.277,2.313] > [2.263,2.338] {-0.6%,+1.1%} p=0.00000+(30 vs 30)        │
│ z_image.default/a/res1 | 00_original vs 20_barrier_start                     │ 2.292s > 2.276s {-0.7%} [2.277,2.313] > [2.264,2.294] {-0.6%,-0.8%} p=0.00000+(30 vs 30)        │
│ z_image.default/a/res1 | 10_full_timings vs 20_barrier_start                 │ 2.279s ~ 2.276s {-0.1%} [2.263,2.338] ~ [2.264,2.294] {+0.1%,-1.9%} p=0.23783 (30 vs 30)        │
│ z_image.default/a/res2 | 00_original vs 10_full_timings                      │ 2.292s ~ 2.289s {-0.1%} [2.276,2.308] ~ [2.272,2.387] {-0.2%,+3.4%} p=0.00479 (30 vs 30)        │
│ z_image.default/a/res2 | 00_original vs 20_barrier_start                     │ 2.292s > 2.288s {-0.2%} [2.276,2.308] > [2.274,2.396] {-0.1%,+3.8%} p=0.00049 (30 vs 30)        │
│ z_image.default/a/res2 | 10_full_timings vs 20_barrier_start                 │ 2.289s ~ 2.288s {-0.0%} [2.272,2.387] ~ [2.274,2.396] {+0.1%,+0.4%} p=0.17478 (30 vs 30)        │
│ z_image.default/a/res3 | 00_original vs 10_full_timings                      │ 2.292s > 2.282s {-0.4%} [2.274,2.394] > [2.271,2.310] {-0.1%,-3.5%} p=0.00054 (30 vs 30)        │
│ z_image.default/a/res3 | 00_original vs 20_barrier_start                     │ 2.292s ~ 2.289s {-0.1%} [2.274,2.394] ~ [2.277,2.387] {+0.1%,-0.3%} p=0.04580 (30 vs 30)        │
│ z_image.default/a/res3 | 10_full_timings vs 20_barrier_start                 │ 2.282s ~ 2.289s {+0.3%} [2.271,2.310] ~ [2.277,2.387] {+0.2%,+3.3%} p=0.01303 (30 vs 30)        │
└──────────────────────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────┘