Skip to content

About

Xing4.0 llama.cpp (xing4_0-port) with in-kernel q8_0 MLA FlashAttention and MoE fusion for faster decode

Resources

Contributing

Security policy

Stars

2 stars

Watchers

0 watching

Forks

 
 

Latest commit

 

History

11,010 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Xing4.0 speed fork

This repository is shuxiaoqiong/llama.cpp branch xing4_0-port — the engine behind ggml-org/llama.cpp#29012, which adds the xing4_0 architecture (mHC, absorbed MLA, 64-expert MoE, MTP block) — plus two CUDA changes so Xing4.0-29B-A4B decodes faster when the KV cache is quantized.

Serve the dynamic GGUF from jmarceno/Xing4.0-29B-A4B-GGUF with this tree. The stock xing4_0-port binary loads the same file, and on every FlashAttention step it converts the whole KV cache to f16 first.

Branch: xing4_0-port (default on this fork). License remains MIT. Base-model weights stay under the China Telecom / TeleAI Apache-2.0 license.

What changed

Xing4.0 attention is absorbed MLA. With -ctk q8_0 -ctv q8_0, K is 576 and V is 512, and V is a view of K. The Ampere MMA FlashAttention configs for that shape are single-stage, so the vendor kernel converts the full KV cache to f16 before the matmul and the graph reserves that scratch for the whole context.

  1. In-kernel q8_0 / q4_0 dequant (ggml/src/ggml-cuda/fattn.cu, fattn-common.cuh, fattn-mma-f16.cuh, and the tile/vec kernel signatures). For head shapes 576/512 and 512/512, when V is a view of quantized K, the MMA kernel dequantizes K/V blocks straight into the shared-memory tile. The f16 conversion and its scratch reservation are skipped. This is on by default; GGML_CUDA_FA_INKERNEL_DEQUANT=0 restores convert-then-read. A 2048-token greedy smoke on the published dynamic quant matched the previous binary (same completion, 11, 13, and 17, on the same prime-number prompt). The cost that disappears grows with context length.

  2. Top-k MoE fusion stays valid under the CUDA graph allocator (ggml/src/ggml-cuda/ggml-cuda.cu), the same allocator edges as upstream llama.cpp#28432. Without them the allocator can reuse a fusion input as an output, and ggml_cuda_check_fusion_memory_ranges rejects a valid 64-expert top-k fusion. Xing4.0 routes 64 experts, top-4.

Build this branch with CUDA and point llama-server at the GGUF. Flags used with the published quant: context 131072, -ctk q8_0 -ctv q8_0, Flash Attention on, all layers on GPU. GGML_CUDA_DISABLE_FUSION=1 turns the MoE fusion path off.

The rest of this file is the upstream llama.cpp README.


llama.cpp

llama

Quick start

A few options to get llama.cpp installed on your machine:

Once installed:

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
VLM session with `llama cli` VLM session with llama cli Built-in web UI against `llama serve` running Qwen 3.6 Built-in web UI against llama serve

Description

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

  • Plain C/C++ implementation without any dependencies
  • Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
  • AVX, AVX2, AVX512 and AMX support for x86 architectures
  • RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
  • 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
  • Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
  • Vulkan and SYCL backend support
  • CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is build on top of the ggml library.

Supported backends

Backend Target devices
BLAS All
BLIS All
CANN Ascend NPU
CUDA Nvidia GPU
HIP AMD GPU
Hexagon Snapdragon
IBM zDNN IBM Z & LinuxONE
MUSA Moore Threads GPU
Metal Apple Silicon
OpenCL Adreno GPU
OpenVINO [In Progress] Intel CPUs, GPUs, and NPUs
RPC All
SYCL Intel GPU
VirtGPU VirtGPU APIR
Vulkan GPU
WebGPU All
ZenDNN AMD CPU

Documentation

Tools

Development

Contributing

  • Contributors can open PRs
  • Collaborators will be invited based on contributions
  • Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
  • Any help with managing issues, PRs and projects is very appreciated!
  • Read the CONTRIBUTING.md for more information

Acknowledgements

  • yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
  • nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
  • nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
  • mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
  • sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain

About

Xing4.0 llama.cpp (xing4_0-port) with in-kernel q8_0 MLA FlashAttention and MoE fusion for faster decode

Resources

Contributing

Security policy

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages