SGL: optimize Qwen3.6 27B and 35B A3B online fp8 quantization performance on BMG w/o XPU Graph and resolve acc issue. - #577
Open
lalalapotter wants to merge 19 commits into
Open
Conversation
ESIMD gdn_conv_fused_seq kernel for the Qwen3 gated-delta-net decode path, replacing the pytorch-glue conv with a single dispatch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…back Add esimd_resadd_norm_gemv2_fp8_pert: fused residual-add + RMSNorm + two fp8 per-tensor GEMVs, with the post-add residual (hidden+residual) written to a separate new_residual buffer by the gid==0 group, removing the follow-up aten::add. Used by the GDN and full-attention input-norm fusions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- moe_up_merged_e5m2: single dispatch for routed + shared up projections. - MoeRouterTopKNormFusedE4M3: fused post-attn norm + fp8 router + topk. - moe_down_finalize_fused_e5m2: optional (env SGL_XPU_MOE_DOWN_FUSED, default off) folding of routed down into the finalize kernel. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Refresh patches/sglang_for_multi_arc.patch from origin/dev-bmg so the Docker build picks up the fused-kernel Python changes (env-read caching, MoE decode fusion, GDN/full-attention input-norm fusions). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…g chunks + lsc_load_2d)
Optimize the ESIMD fp8 MoE prefill up/down DPAS kernels for long prefill
(4k target). Cuts moe up+down from ~560ms to ~185ms (4096-token prefill,
Qwen3.5-35B-A3B, TP=2, xpu-graph disabled), all changes bit-exact.
- doubleGRF: add `-options -doubleGRF` to the moe_fp8 ext (setup.py). The
kernel had a 192B register spill; doubleGRF eliminates it.
- Load-balancing chunks: add MpChunk{eid,t0,nt} + a single-WI
moe_prefill_build_chunks_kernel that pre-splits each expert's token range
into uniform MAX_M blocks. Change up/down grid from (num_experts, N-tile)
with an inner m_base loop to (num_chunks, N-tile) with one block per WI,
removing the serial tail that hot (heavily-routed) experts caused under
skewed routing. Bit-exact: each output element is still summed over the
full K by one WI in the same order.
- lsc_load_2d weight loading: replace the per-ns 8x tiny block_load loop
with a single xesimd::lsc_load_2d 2D block load (N rows x 16 K, row-major)
in both up and down kernels. Row-major result matches the old layout
exactly (bit-exact).
- Merge build_chunks: factor up/down into *_core impls that accept
prebuilt chunks; the fused moe_prefill_full_fp8 driver builds chunks once
and shares them between up and down (valid since MP_UP_MAXM==MP_DOWN_MAXM),
halving build_chunks launches (80->40 per prefill pass).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… runtime - patches/sglang_for_multi_arc.patch: regenerate from origin/dev-bmg (fp8 rowwise fused _scaled_mm + ESIMD fast-path wiring + GGUF q8_0 XPU methods). Verified applies cleanly to upstream v0.5.13. - moe_prefill_fp8: MoE down kernel N tile 32->64 (issues N/32 lsc_load_2d blocks since 2D block height caps at 32 rows). - docker: swap in oneCCL 2021.15 runtime (Arc Pro enhancements), replacing the torch-bundled libccl via /opt/venv/lib symlinks (Dockerfile + Dockerfile.dev). - scripts: build_image tag -> intel registry 0728; add update_sglang_patch.sh to regenerate the multi-arc patch from a dev-bmg checkout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ose copy) Add moe_up_routed_e5m2_nmajor_kernel and moe_up_merged_e5m2_nmajor_kernel that read sglang's native FusedMoE w13 layout [E, 2*inter, hidden] directly via lsc_load_2d + fp8e5m2_block_to_vnni_nk, mirroring the existing e4m3 N-major kernels. Flip the five e5m2 fused decode entries (moe_forward_full, _rtfused, _rtfused_norm, _silu_routed_e5m2, _v2) to compute intermediate_size from size(1) and dispatch the _nmajor up kernels. This lets the host pass w13 without a transposed [E, hidden, 2*inter] replica, eliminating a full extra weight copy (~10GB/tile at E=256) that previously forced mem-fraction-static=0.9. Bit-exact vs the K-major path (VNNI identity); microbench shows only 2-6% per-call cost across cache conditions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…copy) Refresh patches/sglang_for_multi_arc.patch from origin/dev-bmg so the Docker build picks up the e5m2 fused-MoE N-major w13 change: fp8.py drops the load-time _esimd_e5m2_t transpose copy (~10GB/tile), qwen2_moe.py feeds native N-major w13/shared gate_up, triton.py drops _e5m2_gate_up_transposed. Regenerated via scripts/update_sglang_patch.sh (git diff v0.5.13 origin/dev-bmg -- python/); verified it applies cleanly to a pristine v0.5.13 worktree. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
lalalapotter
marked this pull request as ready for review
August 7, 2026 02:44
…ains
GGUF decode was spending most of its time in small oneDNN GEMVs and in the
elementwise glue between them, with a host-side launch per step. Fold each
chain into a single ESIMD op:
esimd_moe_forward_full_gguf topk + routed experts + shared expert
esimd_moe_forward_full_gguf_norm the above, plus the post-attention
GemmaRMSNorm (residual add + RMS norm) and
the fp16 router GEMV
esimd_shared_expert_q8 gate_up + silu + down + gate*sigmoid
esimd_norm_gemv_q8_0 per-head RMSNorm * silu(z), then the q8_0
output GEMV
esimd_resadd_norm_gemv_q8_ba residual add + RMSNorm + the q8_0 qkvz and
fp16 ba projections of the GDN input
The fused ops also keep these GEMVs off oneDNN, which matters beyond
performance: for the narrow N=32 ba projection oneDNN picks a split-K schedule
whose atomic reduction order varies with GPU occupancy, so the same layer with
byte-identical inputs returned one of two different results depending on how
busy the device was. The ESIMD kernels are weight-stationary with no split-K
and are bit-reproducible.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
chunk_gated_delta_rule_extend only returned the final recurrent state, so a mamba track snapshot requested at a position that is not chunk-aligned had nothing to read. Add an h_chunk_size argument: when non-zero the kernel also writes h[i], the state after i full chunks, so the caller can pick the entry covering the snapshot position. Sequence s owns ceil(L_s/chunk) entries and the sum is bounded by T_total/chunk + n_seqs, which is what gets allocated; trailing rows are never indexed. When zero the buffer is empty and the kernel is unchanged. Pairs with the sglang-side change that threads intermediate_chunk_size through the torch/triton wrappers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…pert count esimd_norm_gemv_q8_0 assumed a single token: x/z were [HV, V] and the WG grid covered only the output bands. Infer M from the row count and lay the grid out as M*blocks groups (token = gid / blocks, blk = gid % blocks) so a batched decode step is one launch. The weight offset within K stays lid*V and is shared by all tokens, so weights are read once per band and reused, while only the activation offset carries the extra token*K. Single-token behaviour is the M == 1 case and is unchanged. This is what lets the fused GGUF MoE path serve decode batches above M=1. Also reject n_experts > 512 up front: the topk kernel keeps every expert score in one simd<fp16, PAD> and the host only instantiates PAD = 256 or 512, so a larger count silently indexed past the end of that vector. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…n blocks The final PF_KV_CHUNK iteration covers up to 127 KV rows past seq_len. Their scores are masked to FP32_MIN so they are numerically inert, but the address computation still runs for them and indexed block_table_row[] past the blocks belonging to this request. block_table has exactly ceil(max_seq_len_k / page_size) columns, so the longest request in the batch fills its row completely: the read then runs off the end of the row, and for the last request off the end of the whole tensor. That out-of-bounds device read surfaced as UR_RESULT_ERROR_DEVICE_LOST. Clamp the logical block index to the last block the request actually owns. The loaded values are discarded by the mask either way, so results are unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…stic output) The softmax running max, the partial sums and the S tiles are all exchanged between subgroups through SLM. The split named-barrier form (__esimd_nbarrier_arrive / __esimd_nbarrier) only orders execution: it does not make the SLM stores visible to the other subgroups, so a subgroup could read the previous iteration's values. barrier() carries the required memory fence. The failure matched that mechanism exactly: prompts of at most 128 tokens (a single KV chunk, no cross-iteration reuse) were always reproducible, while multi-chunk prompts diverged more often the more chunks they had -- 1/7 runs at 384 tokens, 7/7 at 1024 and 2427 (max_diff 7.0e-03), measured by calling the kernel directly on fixed tensors. This is also the only kernel in the tree that uses named barriers; every other one already uses barrier(). After the change all of those are bit-identical (0/7, max_diff exactly 0), and end-to-end a 27897-token prompt reproduces its output exactly. Cost is +3.3% on the prefill attention kernel (6.10 -> 6.30 ms at 8192 tokens); decode does not use this kernel. PF_SPLIT_BARRIER=1 restores the old racy form for A/B testing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A one-off out-of-tree build wrote to build_opt/, which `build/` did not match, so 122 MB of stale .so artefacts showed up as untracked files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…fusions) Regenerated from sglang v0.5.13..dev-bmg. Beyond refreshing the existing content this brings in: - fp8 prefill determinism: narrow-N GEMMs now go to the ESIMD kernel instead of torch._scaled_mm. oneDNN picks a split-K schedule for N=32 whose atomic reduction order depends on GPU occupancy, so a chunked-prefill tail block (M=73, above the old M<=64 ESIMD cap) returned one of two different results for byte-identical inputs, and the divergence reached the sampled token. - GGUF decode determinism: layer 0 has residual=None and therefore skipped the fused residual-add/norm/projection path, falling back to the same non-reproducible oneDNN GEMV. It now feeds a zero residual and stays fused, taking the fused GDN layer count from 78 to 80. - GDN per-chunk state materialisation for non-aligned extend snapshots, pairing with the kernel-side h_chunk_size argument. - Fused GGUF MoE batch cap raised, with repack chunks sized by bytes. - An opt-in per-module forward fingerprint trace, off unless its env var is set. Both paths are now bit-identical across repeated runs: GGUF and fp8 both reproduce exactly at 9304/27897/83520-token prompts and on the captured BFCL prompt set. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.