Skip to content

feat: add GLM-5.3-Flash MTP support - #2157

Open
cubezhang wants to merge 2 commits into
ROCm:mainfrom
cubezhang:glm53-flash-mtp
Open

feat: add GLM-5.3-Flash MTP support#2157
cubezhang wants to merge 2 commits into
ROCm:mainfrom
cubezhang:glm53-flash-mtp

Conversation

@cubezhang

Copy link
Copy Markdown
Contributor

feat: add GLM-5.3-Flash MTP support

GLM-5.3-Flash checkpoint layer 45 was previously discarded, so the model could
only serve without speculative decoding. This change registers a GLM-specific
MTP model, reuses the existing NextN predictor prologue, and constructs layer 45
with GLM's k-pool MLA and clamped SwiGLU MoE behavior.

The pooled indexer now preserves position-addressed key/gate history across
speculative verification and rejection, including ragged concurrent requests.
Draft graph staging handles GLM-5.3's two-dimensional NextN residual even though
the backbone config contains hc_mult=4. The existing GLM-5.2/DeepSeek MTP path
keeps its original decoder behavior through the default predictor-layer hook.

Validation

  • Black formatting and Ruff checks pass for the changed MTP implementation and tests.
  • 61 focused GLM/MTP tests pass, including the shared prologue, input masking,
    top-k sharing, decode contracts, scheduling, GLM routing, and k-pool behavior.
  • The final physical KV-slot mapping kernel test passes on ROCm.
  • A standalone image built directly from upstream/main plus this commit
    completed real-model TP8 MTP3 text serving and the 8/8 quality smoke suite on
    8× MI308X.
  • MTP1/MTP3 natural acceptance lengths 0–3, forced rejection, and concurrent
    short/long requests were validated during bring-up.

Measured performance

On a fixed natural-language workload with TP8, 512 output tokens, and BF16
KV/index cache:

Configuration Concurrency Output tok/s Change vs no MTP
No MTP 1 104.67
MTP3 1 181.54 +73.44%
No MTP 8 409.96
MTP1 8 559.98 +36.59%

This branch is based directly on upstream/main (5a9c206).

@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2157 --add-label <label>

@zufayu
zufayu requested a review from yitingw1 September 8, 2026 01:26
@yitingw1

yitingw1 commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Please fix conflicts

@cubezhang

Copy link
Copy Markdown
Contributor Author

Okay,let me check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants