Skip to content

Update the remaining dataset speeds for the FP16 kernels #64

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
praveenperera merged 3 commits into from
Oct 10, 2026
Merged

Update the remaining dataset speeds for the FP16 kernels #64

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
fdeb945
Update the remaining dataset speeds for the FP16 kernels
praveenperera Oct 10, 2026
3ec8258
Round AMI IHM from the published RTFx
praveenperera Oct 10, 2026
196ed06
Use measured speeds for the remaining datasets
praveenperera Oct 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 12 additions & 16 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,22 +179,18 @@ threads ahead of it; it clusters each finished file while the GPU runs the next.
|---|---:|---:|---:|---:|---:|
| VoxConverse dev | 20.3 h | 7.0% | 1786x | 7.4% | 3325x |
| VoxConverse test | 43.5 h | 11.1% | 1921x | 11.2% | 3722x |
| AMI IHM | 18.7 h | 17.0% | 914x | 17.4% | 1589x |
| AMI SDM | 18.7 h | 19.7% | 899x | 20.6% | 1489x |
| AISHELL-4 | 12.7 h | 11.1% | 1046x | 11.4% | 1929x |
| Earnings-21 | 39.3 h | 9.7% | 915x | 9.2% | 1567x |
| ICSI | 71.7 h | 33.3% | 1037x | 33.7% | 1917x |
| AVA-AVD | 4.4 h | 45.4% | 1047x | 48.9% | 1942x |

That's about 229 hours of audio in 12 minutes with `cuda`, or 7 minutes with
`cuda-fast`. The VoxConverse dev row comes from the runs above, and the
VoxConverse test row from one run of the same version on another RTX 4090
container with 16 CPU cores, 8 GiB of RAM and driver 595.99.02. The other datasets ran on an earlier version,
before the current CUDA kernels, on a different RTX 4090 host, so their speeds
understate this version. On VoxConverse test, `cuda` and pyannote CUDA
both score 11.1% DER; pyannote ran at 25x in an earlier container, about 78
times slower than `cuda`. On AMI IHM and
Earnings-21, `cuda` matches the earlier pyannote runs at 17.0% and 9.7% DER,
| AMI IHM | 18.7 h | 17.0% | 1746x | 17.4% | 3507x |
| AMI SDM | 18.7 h | 19.7% | 1717x | 20.6% | 3286x |
| AISHELL-4 | 12.7 h | 11.1% | 1997x | 11.4% | 4258x |
| Earnings-21 | 39.3 h | 9.7% | 1747x | 9.2% | 3459x |
| ICSI | 71.7 h | 33.3% | 1980x | 33.7% | 4231x |
| AVA-AVD | 4.4 h | 45.4% | 1999x | 48.9% | 4286x |

That's about 229 hours of audio in under 8 minutes with `cuda`, or under 4
minutes with `cuda-fast`. The VoxConverse dev row comes from the runs above.
On VoxConverse test, `cuda` and pyannote CUDA both score 11.1% DER; pyannote
ran at 25x in an earlier container, about 78 times slower than `cuda`. On AMI
IHM and Earnings-21, `cuda` matches the earlier pyannote runs at 17.0% and 9.7% DER,
where pyannote ran at 15x and 18x. On macOS, `coreml` runs these datasets at
450x to 644x, with DER within 1.6 points of pyannote's. See
[benchmarks/](https://github.com/avencera/speakrs/tree/master/benchmarks) for
Expand Down
37 changes: 16 additions & 21 deletions benchmarks/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,15 +87,10 @@ Hardware: Apple M4 Pro, macOS 26.3

## Linux (CUDA)

The VoxConverse rows use NVIDIA RTX 4090 containers with 16 CPU cores and
8 GiB of RAM, one per split. The cloud host doesn't report its CPU model. The speakrs CUDA and
CUDA Fast rows for VoxConverse were measured on the current version. The
VoxConverse pyannote rows ran earlier in a container with the same GPU model,
CPU count and RAM, so those speedups compare separate runs. The speakrs rows
for the other datasets come from an earlier version, before the current CUDA
kernels, so they understate current speakrs speed. Average CPU use for those
earlier speakrs runs was 0.95–1.18 cores; this is measured CPU use, not a
thread limit.
The speakrs CUDA rows use NVIDIA RTX 4090 containers with 16 CPU cores and
8 GiB of RAM. The cloud host doesn't report its CPU model. The VoxConverse
pyannote rows ran earlier in a container with the same GPU model, CPU count and
RAM, so those speedups compare separate runs.
Comment thread
praveenperera marked this conversation as resolved.
Comment thread
coderabbitai[bot] marked this conversation as resolved.

Audio and models were copied to the container's local disk before each dataset
run. Time and RTFx use the `speakrs-bm` timer: WAV loading, inference,
Expand Down Expand Up @@ -149,47 +144,47 @@ Earlier pyannote hardware: NVIDIA L40S, AMD EPYC 9354.

| Implementation | DER | Missed | False Alarm | Confusion | Time | RTFx |
|---|---|---|---|---|---|---|
| **speakrs CUDA** | **17.0%** | 8.1% | 4.3% | 4.5% | 73.7s | 914x |
| speakrs CUDA Fast | 17.4% | 8.2% | 4.3% | 4.9% | 42.4s | **1589x** |
| **speakrs CUDA** | **17.0%** | 8.1% | 4.3% | 4.5% | 38.6s | 1746x |
| speakrs CUDA Fast | 17.4% | 8.2% | 4.3% | 4.9% | 19.2s | **3507x** |
| **pyannote CUDA (L40S, earlier run)** | **17.0%** | 8.1% | 4.3% | 4.5% | 4388.1s | 15x |

### AMI SDM (34 files, 1123.8 min)

| Implementation | DER | Missed | False Alarm | Confusion | Time | RTFx |
|---|---|---|---|---|---|---|
| **speakrs CUDA** | **19.7%** | 9.6% | 4.3% | 5.7% | 75.0s | 899x |
| speakrs CUDA Fast | 20.6% | 9.6% | 4.3% | 6.6% | 45.3s | **1489x** |
| **speakrs CUDA** | **19.7%** | 9.6% | 4.3% | 5.7% | 39.3s | 1717x |
| speakrs CUDA Fast | 20.6% | 9.6% | 4.3% | 6.6% | 20.5s | **3286x** |

### AISHELL-4 (20 files, 763.5 min)

| Implementation | DER | Missed | False Alarm | Confusion | Time | RTFx |
|---|---|---|---|---|---|---|
| **speakrs CUDA** | **11.1%** | 3.9% | 3.9% | 3.3% | 43.8s | 1046x |
| speakrs CUDA Fast | 11.4% | 3.9% | 3.9% | 3.6% | 23.7s | **1929x** |
| **speakrs CUDA** | **11.1%** | 3.9% | 3.9% | 3.3% | 22.9s | 1997x |
| speakrs CUDA Fast | 11.4% | 3.9% | 3.9% | 3.6% | 10.8s | **4258x** |

### Earnings-21 (44 files, 2355.8 min)

Earlier pyannote hardware: NVIDIA RTX 4090, AMD EPYC 7B13.

| Implementation | DER | Missed | False Alarm | Confusion | Time | RTFx |
|---|---|---|---|---|---|---|
| speakrs CUDA | 9.7% | 2.6% | 2.4% | 4.7% | 154.5s | 915x |
| **speakrs CUDA Fast** | **9.2%** | 2.5% | 2.5% | 4.2% | 90.2s | **1567x** |
| speakrs CUDA | 9.7% | 2.6% | 2.4% | 4.7% | 80.9s | 1747x |
| **speakrs CUDA Fast** | **9.2%** | 2.5% | 2.5% | 4.2% | 40.9s | **3459x** |
| pyannote CUDA (RTX 4090, earlier run) | 9.7% | 2.6% | 2.4% | 4.7% | 8036.8s | 18x |

### ICSI (75 files, 4301.2 min)

| Implementation | DER | Missed | False Alarm | Confusion | Time | RTFx |
|---|---|---|---|---|---|---|
| **speakrs CUDA** | **33.3%** | 19.5% | 9.5% | 4.3% | 248.8s | 1037x |
| speakrs CUDA Fast | 33.7% | 19.4% | 9.6% | 4.7% | 134.7s | **1917x** |
| **speakrs CUDA** | **33.3%** | 19.5% | 9.5% | 4.3% | 130.3s | 1980x |
| speakrs CUDA Fast | 33.7% | 19.4% | 9.6% | 4.7% | 61.0s | **4231x** |

### AVA-AVD (54 files, 266.1 min)

| Implementation | DER | Missed | False Alarm | Confusion | Time | RTFx |
|---|---|---|---|---|---|---|
| **speakrs CUDA** | **45.4%** | 16.1% | 10.8% | 18.6% | 15.2s | 1047x |
| speakrs CUDA Fast | 48.9% | 15.9% | 11.3% | 21.8% | 8.2s | **1942x** |
| **speakrs CUDA** | **45.4%** | 16.1% | 10.8% | 18.6% | 8.0s | 1999x |
| speakrs CUDA Fast | 48.9% | 15.9% | 11.3% | 21.8% | 3.7s | **4286x** |

## Other implementations

Expand Down
4 changes: 2 additions & 2 deletions benchmarks/aishell-4.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,5 @@ Native speakrs CUDA: 1s step; CUDA Fast: 2s step. Audio and models use local dis

Implementation DER% Missed% FalseAlarm% Confusion% Time RTFx
──────────────────────────────────────────────────────────────────────────────────────────────────────────
speakrs CUDA 11.1% 3.9% 3.9% 3.3% 43.8s 1046x
speakrs CUDA Fast 11.4% 3.9% 3.9% 3.6% 23.7s 1929x
speakrs CUDA 11.1% 3.9% 3.9% 3.3% 22.9s 1997x
speakrs CUDA Fast 11.4% 3.9% 3.9% 3.6% 10.8s 4258x
4 changes: 2 additions & 2 deletions benchmarks/ami-ihm.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,6 @@ Native speakrs CUDA: 1s step; CUDA Fast: 2s step. Audio and models use local dis

Implementation DER% Missed% FalseAlarm% Confusion% Time RTFx
──────────────────────────────────────────────────────────────────────────────────────────────────────────
speakrs CUDA 17.0% 8.1% 4.3% 4.5% 73.7s 914x
speakrs CUDA Fast 17.4% 8.2% 4.3% 4.9% 42.4s 1589x
speakrs CUDA 17.0% 8.1% 4.3% 4.5% 38.6s 1746x
speakrs CUDA Fast 17.4% 8.2% 4.3% 4.9% 19.2s 3507x
pyannote CUDA (L40S, earlier run) 17.0% 8.1% 4.3% 4.5% 4388.1s 15x
4 changes: 2 additions & 2 deletions benchmarks/ami-sdm.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,5 @@ Native speakrs CUDA: 1s step; CUDA Fast: 2s step. Audio and models use local dis

Implementation DER% Missed% FalseAlarm% Confusion% Time RTFx
──────────────────────────────────────────────────────────────────────────────────────────────────────────
speakrs CUDA 19.7% 9.6% 4.3% 5.7% 75.0s 899x
speakrs CUDA Fast 20.6% 9.6% 4.3% 6.6% 45.3s 1489x
speakrs CUDA 19.7% 9.6% 4.3% 5.7% 39.3s 1717x
speakrs CUDA Fast 20.6% 9.6% 4.3% 6.6% 20.5s 3286x
4 changes: 2 additions & 2 deletions benchmarks/ava-avd.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,5 @@ Native speakrs CUDA: 1s step; CUDA Fast: 2s step. Audio and models use local dis

Implementation DER% Missed% FalseAlarm% Confusion% Time RTFx
──────────────────────────────────────────────────────────────────────────────────────────────────────────
speakrs CUDA 45.4% 16.1% 10.8% 18.6% 15.2s 1047x
speakrs CUDA Fast 48.9% 15.9% 11.3% 21.8% 8.2s 1942x
speakrs CUDA 45.4% 16.1% 10.8% 18.6% 8.0s 1999x
speakrs CUDA Fast 48.9% 15.9% 11.3% 21.8% 3.7s 4286x
4 changes: 2 additions & 2 deletions benchmarks/earnings-21.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,6 @@ Native speakrs CUDA: 1s step; CUDA Fast: 2s step. Audio and models use local dis

Implementation DER% Missed% FalseAlarm% Confusion% Time RTFx
──────────────────────────────────────────────────────────────────────────────────────────────────────────
speakrs CUDA 9.7% 2.6% 2.4% 4.7% 154.5s 915x
speakrs CUDA Fast 9.2% 2.5% 2.5% 4.2% 90.2s 1567x
speakrs CUDA 9.7% 2.6% 2.4% 4.7% 80.9s 1747x
speakrs CUDA Fast 9.2% 2.5% 2.5% 4.2% 40.9s 3459x
pyannote CUDA (RTX 4090, earlier run) 9.7% 2.6% 2.4% 4.7% 8036.8s 18x
4 changes: 2 additions & 2 deletions benchmarks/icsi.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,5 @@ Native speakrs CUDA: 1s step; CUDA Fast: 2s step. Audio and models use local dis

Implementation DER% Missed% FalseAlarm% Confusion% Time RTFx
──────────────────────────────────────────────────────────────────────────────────────────────────────────
speakrs CUDA 33.3% 19.5% 9.5% 4.3% 248.8s 1037x
speakrs CUDA Fast 33.7% 19.4% 9.6% 4.7% 134.7s 1917x
speakrs CUDA 33.3% 19.5% 9.5% 4.3% 130.3s 1980x
speakrs CUDA Fast 33.7% 19.4% 9.6% 4.7% 61.0s 4231x
28 changes: 12 additions & 16 deletions src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,22 +180,18 @@
//! |---|---:|---:|---:|---:|---:|
//! | VoxConverse dev | 20.3 h | 7.0% | 1786x | 7.4% | 3325x |
//! | VoxConverse test | 43.5 h | 11.1% | 1921x | 11.2% | 3722x |
//! | AMI IHM | 18.7 h | 17.0% | 914x | 17.4% | 1589x |
//! | AMI SDM | 18.7 h | 19.7% | 899x | 20.6% | 1489x |
//! | AISHELL-4 | 12.7 h | 11.1% | 1046x | 11.4% | 1929x |
//! | Earnings-21 | 39.3 h | 9.7% | 915x | 9.2% | 1567x |
//! | ICSI | 71.7 h | 33.3% | 1037x | 33.7% | 1917x |
//! | AVA-AVD | 4.4 h | 45.4% | 1047x | 48.9% | 1942x |
//!
//! That's about 229 hours of audio in 12 minutes with `cuda`, or 7 minutes with
//! `cuda-fast`. The VoxConverse dev row comes from the runs above, and the
//! VoxConverse test row from one run of the same version on another RTX 4090
//! container with 16 CPU cores, 8 GiB of RAM and driver 595.99.02. The other datasets ran on an earlier version,
//! before the current CUDA kernels, on a different RTX 4090 host, so their speeds
//! understate this version. On VoxConverse test, `cuda` and pyannote CUDA
//! both score 11.1% DER; pyannote ran at 25x in an earlier container, about 78
//! times slower than `cuda`. On AMI IHM and
//! Earnings-21, `cuda` matches the earlier pyannote runs at 17.0% and 9.7% DER,
//! | AMI IHM | 18.7 h | 17.0% | 1746x | 17.4% | 3507x |
//! | AMI SDM | 18.7 h | 19.7% | 1717x | 20.6% | 3286x |
//! | AISHELL-4 | 12.7 h | 11.1% | 1997x | 11.4% | 4258x |
//! | Earnings-21 | 39.3 h | 9.7% | 1747x | 9.2% | 3459x |
//! | ICSI | 71.7 h | 33.3% | 1980x | 33.7% | 4231x |
//! | AVA-AVD | 4.4 h | 45.4% | 1999x | 48.9% | 4286x |
//!
//! That's about 229 hours of audio in under 8 minutes with `cuda`, or under 4
//! minutes with `cuda-fast`. The VoxConverse dev row comes from the runs above.
//! On VoxConverse test, `cuda` and pyannote CUDA both score 11.1% DER; pyannote
//! ran at 25x in an earlier container, about 78 times slower than `cuda`. On AMI
//! IHM and Earnings-21, `cuda` matches the earlier pyannote runs at 17.0% and 9.7% DER,
//! where pyannote ran at 15x and 18x. On macOS, `coreml` runs these datasets at
//! 450x to 644x, with DER within 1.6 points of pyannote's. See
//! [benchmarks/](https://github.com/avencera/speakrs/tree/master/benchmarks) for
Expand Down
Loading