TonyMFJ/Qwen3.8-Flash-Next-WinVulkan-GGUF

Qwen3.8-Flash-Next (Qwen 3.8 Next Flash) — Windows Vulkan 全适配包 / Windows Vulkan Full Deployment Kit

EN | Complete local deployment kit for Qwen3.8-Flash-Next — a ~180 B model: 125 B transformer (6 B activated per token) + a 51 B n-gram embedding table + a 4 B MTP draft (ROCmFP4-FAST-v2-ple16 fork quantization) — running on Windows + Vulkan — AMD Ryzen AI MAX+ 395 (ROG Flow Z13, 128GB LPDDR5X, 96GB VRAM carve-out). Includes MTP speculative decoding, PLE n-gram table disk-offload, custom llama.cpp fork binaries, startup scripts and a custom WebUI. This build is original adaptation work — no off-the-shelf guide exists.

中文 | Qwen3.8-Flash-Next（总参约 180B = 主干 125B / 每 token 激活 6B + n-gram 表 51B + MTP 草稿 4B；ROCmFP4-FAST-v2-ple16 fork 专用量化）在 Windows + Vulkan 上的完整本地部署方案 — AMD Ryzen AI MAX+ 395（ROG Flow Z13 / 128GB LPDDR5X / 96GB 显存 carve）。含 MTP 投机解码、PLE 表磁盘化、定制 llama.cpp fork 二进制、启动脚本与定制 WebUI。本包为原创适配，网上无现成方案。

📐 Model facts / 模型参数 — total ~180 B parameters: 125 B transformer (6 B activated per token, 48 layers, hidden 2560, 512 MoE experts with 10 routed + 1 shared, hybrid SSM/full-attention every 4th layer, 262144-token context) + 51 B n-gram embedding table (per_layer_token_embd.weight, [160 × 320001536]) + 4 B MTP draft. Measured from the GGUF headers: main file 176.94 B tensor parameters (125.74 B backbone + 51.20 B n-gram table), MTP draft 3.88 B, mmproj 0.45 B — matching upstream's "125B with 6B activated, plus 51B n-gram embedding and 4B MTP". ⚠️ 87.06 GiB / 93.48 GB is the size of the main quant file at 4.23 bpw — it is not a parameter count. Earlier revisions of this card said "87B MoE", which mixed up file size and parameter count. 中文：总参约 180B = 主干 125B（每 token 激活 6B）+ n-gram 表 51B + MTP 草稿 4B；GGUF 头实测主模型 176.94B 张量参数、草稿 3.88B、mmproj 0.45B，与上游口径一致。87.06 GiB / 93.48 GB 是 4.23 bpw 量化文件的体积，不是参数量（旧版卡片写 "87B MoE" 属单位混用，已更正）。

🔧 Fork binaries, source patches, scripts & tools → TonyMFJ/Qwen3.8-Flash-Next-WinVulkan 🔧 Fork 二进制、源码补丁、脚本与工具 → TonyMFJ/Qwen3.8-Flash-Next-WinVulkan

---

📦 Files / 文件清单

File / 文件	Size / 大小	Purpose / 用途
Qwen3.8-Flash-Next-ROCmFP4-FAST-v2-ple16.gguf	93.48 GB (87.06 GiB)	Main model, per-head PLE layout (original). 176.94 B tensor parameters at 4.23 bpw / 主模型，per-head PLE 布局（原始版），4.23 bpw 下 176.94B 张量参数
ple_sidecar_joined.gguf	22.40 GB (20.86 GiB)	Joined-layout PLE table sidecar for --model-ple — disk-offload, frees ~21 GiB VRAM. The same 51.2 B-parameter n-gram table that already sits inside the main file (byte-for-byte re-layout, not extra weights) / joined 布局 PLE 表侧车，--model-ple 磁盘化加载，腾出约 21GiB 显存；与主模型内的表同一份数据（仅布局不同，非额外权重）
mmproj-Qwen3.8-Flash-Next-f16.gguf	0.90 GB (0.84 GiB)	Vision projector (must stay on GPU) / 视觉投影器（必须驻留 GPU）
mtp-Qwen3.8-Flash-Next-Q4_K_M.gguf	2.79 GB (2.59 GiB)	MTP draft model for speculative decoding (-md) / MTP 草稿模型，投机解码用

📏 Repository total: 119.57 GB (111.35 GiB) of downloads. A single run needs ≈ 97.17 GB (90.5 GiB) of weights — main file 93.48 + mmproj 0.90 + draft 2.79; the sidecar is the same n-gram table in a second layout, counted once. Sizes are decimal GB as shown in the HF file browser; GiB in brackets for du-style comparisons. 📏 仓库合计 **119.57 GB (111.35 GiB)**（下载总量）；单次运行实际占用约 **97.17 GB (90.5 GiB)**（主模型 + mmproj + 草稿；侧车是同一张 n-gram 表的另一种布局，只算一次）。括号内为 GiB（与本地 du/资源管理器口径一致）。

💡 Equivalent alternative to the sidecar: convert the main GGUF to joined layout locally with tools/gguf_split_ple_heads.py (join) + --ngram-on-disk. 💡 侧车的等价方案：用 tools/gguf_split_ple_heads.py（join 模式）把主模型本地转成 joined 布局，再配合 --ngram-on-disk。

⚡ Measured performance / 实测性能（reference machine）

⚠️ All figures below were measured with the APU package power limited to 80 W (AMD Ryzen AI MAX+ 395, 128 GB LPDDR5X, 96 GB VRAM carve-out, Windows 11, Vulkan backend). A different power profile gives different numbers — check the wattage before comparing.

Scenario	Result	Notes
Prefill, 24K tokens	299 t/s	PLE table on disk + ubatch 2048
Generation, short context	14.5 t/s	MTP on, acceptance ≈ 67%
Prefill, 90K tokens	≈ 219 t/s	normal decay with context length
Generation, 90K ctx — KV in host RAM	≈ 14 t/s	--no-kv-offload
Generation, 90K ctx — KV in VRAM	≈ 30 t/s	-ctk q8_0 -ctv q8_0 — roughly doubles long-context decode
Generation, short context, everything in VRAM	≈ 28 t/s	93 / 96 GB used
VRAM usage (disk-offload config)	71 / 96 GB	22 GB spare → 100K context + vision fit

Rows come from different sessions/flag sets — treat them as order-of-magnitude references. Pin the power profile, context length, KV placement and MTP state before comparing.

Full measured curve — huihui-abliterated UD-Q4_K_XL build (111.3 GB / 103.7 GiB, 4 shards)

Same machine and power limit, config -c 102400 -b 2048 -ub 2048 -ctk q8_0 -ctv q8_0 (KV cache in VRAM) + MTP draft (adaptive n-min 2 / n-max 4) + PLE n-gram table on disk:

Prompt	Prefill	Decode @ that ctx	Draft acceptance
4.5K	402.8 t/s	30.6 t/s	82.8 %
8.4K	381.3 t/s	35.0 t/s	—
21.8K	344.4 t/s	24.7 t/s	88.0 %
45.2K	277.5 t/s	22.0 t/s	84.0 %
90.4K	220.6 t/s	25.7 t/s	93.9 %

• PLE n-gram table (28.8 GB) stays on disk → peak VRAM 90.4 / 96 GB
• MTP is the decode lever: without the draft model decode falls to ≈ 6–8 t/s
• End-to-end, 90K prompt + 500 output tokens: 429 s with MTP (410 prefill + 19 decode) vs 453 s without (368 + 85)
• Acceptance above is inflated by highly repetitive test text; on natural text it measures 59–83 % (decode ≈ 21–35 t/s)
• MTP costs ≈ 10 % prefill: the draft context forces ubatch 4096 → 2048 (3072/4096 both fail to allocate)

同机同功耗下的完整曲线：prefill 402.8 → 220.6 t/s（4.5K → 90.4K），decode 22–35 t/s（含 MTP）； 模型本体 huihui UD-Q4_K_XL 4 段共 111.3 GB，PLE 表 28.8 GB 走磁盘，显存峰值 90.4/96 GB。

Environment & reproduction: driver 32.0.31032.1003, w64devkit 2.9.1 / GCC 16.2.0, CMake 4.4.0, Vulkan SDK headers 1.4.309 (MinGW Makefiles). Build commands, model layout, verification steps and the pitfall list live in ENVIRONMENT.md.

Tuning levers & pitfalls: MTP acceptance, KV placement (host vs VRAM, ~2x long-context decode), the ubatch ceiling and the misleading “unused tensor” PLE warning → TUNING_NOTES.md.

HIP / ROCm path — negative result: we also brought this model up on ROCm 7.2 + HIP. The PLE disk-offload port works (418.7 t/s prefill at 2.6K), but with head_dim = 256 FlashAttention falls back to the slow tile kernel and upstream's MMA kernel for head_dim > 128 is an empty stub — HIP never beats Vulkan. Full write-up, root cause and artifacts (patched backend, port patch, official DLL mirror) → HIP_NOTES.md · release assets

中文：HIP / ROCm 路径也跑通过（PLE 磁盘化移植成功），但 head_dim=256 让 FlashAttention 掉进退化路径、 上游 MMA 对 head_dim>128 是空壳 kernel，快不过 Vulkan，详见 HIP_NOTES.md 与 Release 附件。

🚀 Key launch flags / 关键启动参数

llama-server -m <ple16.gguf> --model-ple <ple_sidecar_joined.gguf> \
  --mmproj <mmproj-f16.gguf> -md <mtp-Q4_K_M.gguf> \
  -c 102400 --ubatch-size 2048 -ngl 99 --load-mode auto ...

• EN | --load-mode auto is REQUIRED on this fork/Windows (never none). PLE table on disk ≈ −37% speed but frees ~22 GB VRAM for 100K+ context. Full-VRAM mode reaches ~28 t/s (TG) with MTP acceptance ≈ 67%.
• 中文 | 本 fork 在 Windows 上必须用 --load-mode auto（none 会崩）。PLE 表磁盘化约损失 37% 速度，换来约 22GB 显存用于 100K+ 上下文。全显存模式 TG 约 28 t/s，MTP 接受率约 67%。

📚 Sources / 来源

• Base quant / 基础量化: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF (ple16)
• MTP draft / 草稿模型: quimmedes/Qwen3.8-Flash-Next-MTP-GGUF
• Sidecar / 侧车: generated locally with this fork's tools / 用本 fork 工具本地生成

📜 Restore walkthrough / 换机恢复指南

See GitHub repo README_恢复指南.md and MODELS.md — unzip, drop models, run one script. 见 GitHub 仓库的 README_恢复指南.md 与 MODELS.md —— 解压、放模型、跑一个脚本即可。