vulkan: disable MMVQ on AMD UMA devices

[winstonma]

MMVQ's quantization overhead outweighs bandwidth savings on UMA where CPU and GPU share the same memory pool.

Overview

This is the testing result before the patch is written (tested on AMD UMA only):

# Force large workgroup for DMMV
❯ GGML_VK_FORCE_MMVQ=1 llama-bench --model ~/model/gemma-4-26B-A4B-it-qat-GGUF/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -p 0 -n 128 -r 5
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon 880M Graphics (RADV STRIX1) (radv) | uma: 1 | fp16: dot2 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| gemma4 26B.A4B Q4_0            |  13.26 GiB |    25.23 B | Vulkan     |  -1 |           tg128 |         28.78 ± 0.10 |

# Disable MMVQ entirely (uses plain fp16 dequant path)  
❯ GGML_VK_DISABLE_MMVQ=1 llama-bench --model ~/model/gemma-4-26B-A4B-it-qat-GGUF/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -p 0 -n 128 -r 5
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon 880M Graphics (RADV STRIX1) (radv) | uma: 1 | fp16: dot2 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| gemma4 26B.A4B Q4_0            |  13.26 GiB |    25.23 B | Vulkan     |  -1 |           tg128 |         30.79 ± 0.04 |

tg value jump from 28.78 to 30.79 on my device.

The patch is applied to AMD path because this is the device that I could test.

Additional information

The following explanation is written by Claude:

The MMVQ path quantizes the input vector to Q8_1 before dispatching, which adds CPU-side work and an extra quantization kernel on the GPU. On a discrete GPU with fast VRAM, that overhead is worth it because it reduces the weight-loading bandwidth. On my AMD UMA device, the CPU and GPU share the same LPDDR5X pool — the quantization step costs real time and the bandwidth savings are smaller because my device is already memory-bandwidth-limited from both sides simultaneously.

The effectiveness of the patch depends on how much time each model actually spends in these kernels relative to everything else:

• Gemma 4 sparse MoE with very few active experts per token (top-2 out of 128 in its case). Each expert matmul has a very small weight matrix — the k dimension of each expert is tiny. That means the quantization overhead of MMVQ is large relative to the actual compute work, so removing it is felt immediately.
• Dense models have large weight matrices where bandwidth dominates regardless, so whether MMVQ is on or off makes little difference because the bottleneck isn't the quantization step but the weight read itself.

Based on the test I see that this patch would

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes, suggesting and patch writing

[0cc4m]

Your benchmark is invalid, you don't even show the baseline. FORCE_MMVQ enables it anywhere, which is not default behaviour. Also, here's a counter example on AMD Strix Halo:

before is just base master, after is with GGML_VK_DISABLE_MMVQ=1

model	size	params	ngl	fa	mmap	test	t/s (before)	t/s (after)	diff
llama 8B Q4_0	4.33 GiB	8.03 B	-1	1	0	tg128	44.52 ± 0.08	43.32 ± 0.06	-2.7%
llama 8B Q4_K - Small	4.36 GiB	8.03 B	-1	1	0	tg128	44.21 ± 0.12	42.60 ± 0.10	-3.6%
qwen35moe 35B.A3B Q6_K	27.98 GiB	34.66 B	-1	1	0	tg128	53.77 ± 0.28	53.54 ± 0.35	-0.4%
qwen35 27B Q6_K	21.62 GiB	26.90 B	-1	1	0	tg128	8.93 ± 0.00	8.93 ± 0.00	+0.0%

[winstonma]

Yes It was a very quick way to test. After testing with several model I found that the performance was not as prominent. It might due to the thermal behavior my laptop.

Benchmark on the patch

To isolate the behavior, I profiled the changes using a benchmark script that measures the performance impact of MMVQ (Matrix-Matrix-Vector Quantization) during matrix multiplication operations (MUL_MAT).

Below is the summary of the benchmark run on my AMD UMA device (using the maximum GFLOPS over 4 warm repeats, skipping 1 cold burn-in):

SUMMARY (max GFLOPS of 4 warm repeats, 1 cold burn-in skipped)

Type	k	MMVQ Enabled	MMVQ Disabled	Delta	GPU Freq
f16	4096	0.18 GF	0.19 GF	-2.7%	2796 MHz
q4_K	4096	0.24 GF	0.33 GF	-27.2%	2796 MHz
q2_K	4096	0.18 GF	0.27 GF	-33.2%	2796 MHz
q6_K	4096	0.18 GF	0.42 GF	-57.7%	2796 MHz
q8_0	4096	0.21 GF	0.24 GF	-11.6%	2796 MHz
f16	14336	0.50 GF	0.49 GF	2.1%	2796 MHz
q4_K	14336	0.62 GF	0.64 GF	-3.2%	2796 MHz
q2_K	14336	0.60 GF	0.62 GF	-3.0%	2796 MHz
q6_K	14336	0.53 GF	0.70 GF	-23.6%	2796 MHz
q8_0	14336	0.63 GF	0.62 GF	0.9%	2796 MHz

Key Observations:

1. Baseline Consistency: As expected, the f16 baselines are essentially identical across both modes since f16 does not trigger MMVQ paths.
2. Quantization Overhead at $K=4096$: The computational overhead of MMVQ is highly visible at smaller vector widths. Disabling MMVQ yields massive synthetic throughput gains (up to +57.7% for q6_K).
3. Diminishing Returns at Larger $K$: When $K$ expands to 14,336, the gaps narrow significantly. Disabling MMVQ does not trigger a performance regression here, meaning memory bandwidth savings from weight compression begin to balance out the shader arithmetic overhead.

Overall

Following the isolated benchmarks, I tested several real-world models to evaluate the end-to-end impact of the patch during single-token generation (tg128, not using flags):

model	size	params	ngl	fa	mmap	test	t/s(before)	t/s(after)	diff
qwen35 0.8B Q4_K - Medium	497.39 MiB	752.39 M	-1	1	0	tg128	107.60 ± 0.91	109.67 ± 0.72	+1.9%
gemma4 26B.A4B Q4_0	13.26 GiB	25.23 B	-1	1	0	tg128	29.85 ± 0.10	30.51 ± 0.38	+2.2%
llama 8B Q4_K - Medium	4.64 GiB	8.03 B	-1	1	0	tg128	17.25 ± 0.18	18.64 ± 0.02	+8.1%
qwen35moe 35B.A3B Q4_K - Medium	21.27 GiB	35.51 B	-1	1	0	tg128	23.93 ± 0.06	24.48 ± 0.03	+2.3%

Given that AMD UMA hardware can be highly sensitive to background noise, thermal behavior, and initial shader compilation overhead (especially if the Vulkan pipeline cache isn't warmed up, that also happen to me during the benchmark setup). Could you try running a re-test again?

[0cc4m]

And you think I'm not aware of thermal behaviour? llama-bench does multiple runs for that reason, at most you see a higher variance.

With your latest results you didn't even show whether you used FORCE_MMVQ or not.

Regardless, a blanket disable on UMA devices makes no sense, and you should really not be using claude to try to explain the reason for performance behaviour, it is guaranteed to hallucinate.