Feat/gemma4 int4 group32 - #499
Draft
hzjane wants to merge 2 commits into
Draft
Conversation
vllm-xpu 0.21.0 sym_int4 quantizes with QK4_GROUP_SIZE=32 (ggml q4_0), producing scale of shape [N, K/32]. The existing esimd_gemv_int4 hard-coded GROUP_SIZE=128 (scale [N, K/128]) and rejected the 0.21.0 layout. Add a group32-aware path that keeps VL=128 (for BMG bandwidth) but applies 4 distinct group scales per K-loop iteration via a per-lane scale vector (build_lane_scale broadcasts each group over GROUP_SIZE/2 lanes; even/odd halves share the same segmentation since GROUP_SIZE is even). GROUP_SIZE becomes a template param (default 128); host dispatches on the scale's last dim (group_size = K / scale.size(-1)), so the op signature is unchanged and group128 callers are unaffected. kp alignment fixed to 128 (=VL), which also satisfies the 32 constraint. Unit-tested (group32: K=5376/N=8192 and more, K_SPLIT 1/4/8, max abs diff <=0.031; group128 legacy unchanged). In-model: gemma4-31B sym_int4 TP=2 dense-MLP GEMV decode ITL 31.53 -> 29.06ms (-7.8%), gsm8k 5/5, fingerprint identical.
Mirror the GEMV group32 change for esimd_gemm_int4_pgrp. K_LOAD stays 128 (DPAS/XMX efficiency); a K_LOAD now spans 128/GROUP_SIZE groups. Since K_SUB=16 divides both 32 and 128, each K_SUB lands wholly in one group, so the scale is just selected per-sub (group_idx = base + sub/SUBS_PER_GROUP, SUBS_PER_GROUP = GROUP_SIZE/16) and folded into the DPAS b_tile as before — no per-lane vector needed. GROUP_SIZE is a template param (default 128); host dispatches on the scale's last dim. K alignment relaxed to %128 (covers both groups). Wrapper infers group_size = K / scale.size(-1). Unit-tested: group32 M=4/8/16/64 K=5376 N=8192 max abs diff ~0.005; group128 unchanged ~0.005. op signature unchanged; group128 callers unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.