steampunque/Qwen3.5-9B-MP-GGUF

Mixed Precision GGUF layer quantization of Qwen3.5-9B by Qwen

Original model: https://huggingface.co/Qwen/Qwen3.5-9B

The mixed precision quant employs different quantization levels on a per layer basis to enable both high performance and small file size at the same time. The quants employed are all K to avoid slow CPU or older GPU processing of IQ quants. For this file the layer quants are as follows:

Q4_K_L : Q4_K_M + attn_o = q6_k
Q5_K_L : attn_v = q8_0 attn_o = q6_k ffn_d = q6_k
Q6_K_S : Q6_K

   LAYER_TYPES='[
   [0 ,"Q5_K_M"], [1 ,"Q5_K_S"], [2 ,"Q4_K_L"], [3 ,"Q4_K_M"], [4 ,"Q4_K_S"], [5 ,"Q4_K_M"], [6 ,"Q4_K_S"], [7 ,"Q4_K_M"],
   [8 ,"Q4_K_S"], [9 ,"Q4_K_S"], [10,"Q4_K_S"], [11,"Q4_K_S"], [12,"Q4_K_M"], [13,"Q4_K_S"], [14,"Q4_K_M"], [15,"Q4_K_S"],
   [16,"Q4_K_M"], [17,"Q4_K_S"], [18,"Q4_K_M"], [19,"Q4_K_M"], [20,"Q4_K_M"], [21,"Q4_K_M"], [22,"Q4_K_M"], [23,"Q4_K_M"],
   [24,"Q4_K_M"], [25,"Q4_K_M"], [26,"Q4_K_M"], [27,"Q4_K_L"], [28,"Q5_K_S"], [29,"Q5_K_M"], [30,"Q5_K_L"], [31,"Q6_K_S"]
   ]'
   FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high"

The layer quants were optimized for very strong performance across a set of curated reasoning prompts. The final quant size is about 0.6B bigger than Q4_K_M.

A second quant is available using new extended K quant layer definitions which provide more flexibility in configuring the attn_v, attn_o, and ffn_d quant levels:

Extended K (QN_E_H) mixed precision layer quant nomenclature:

QN_K_VOD, Q8_0_VOD
N = {2,3,4,5,6}

VOD = attnV/QKV:attnO:ffnD
V,O,D = {0,2,3,4,5,6,8,f,F}

VOD MAP:
2:Q2_K, 3:Q3_K, 4:Q4_K, 5:Q5_K, 6:Q6_K, 8:Q8_0, f:F:F16, 0:F32, default QN_K

   LAYER_TYPES='[
   [0 ,"Q5_K_658"], [1 ,"Q5_K_655"], [2 ,"Q4_K_546"], [3 ,"Q4_K_646"], [4 ,"Q4_K_555"], [5 ,"Q4_K_555"], [6 ,"Q4_K_545"], [7 ,"Q4_K_545"],
   [8 ,"Q4_K_544"], [9 ,"Q4_K_544"], [10,"Q4_K_544"], [11,"Q4_K_644"], [12,"Q4_K_544"], [13,"Q4_K_544"], [14,"Q4_K_545"], [15,"Q4_K_645"],
   [16,"Q4_K_546"], [17,"Q4_K_545"], [18,"Q4_K_546"], [19,"Q4_K_645"], [20,"Q4_K_546"], [21,"Q4_K_545"], [22,"Q4_K_546"], [23,"Q4_K_645"],
   [24,"Q4_K_546"], [25,"Q4_K_546"], [26,"Q4_K_546"], [27,"Q5_K_555"], [28,"Q5_K_655"], [29,"Q5_K_656"], [30,"Q5_K_658"], [31,"Q6_K_666"]
   ]'

   FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high"

A second Q4_E_H_MTP quant is available which adds nextn layer for MTP, supported by llama.cpp b9180 and above (versions less than b9180 will not load this quant, however at b9180 and above the model should load with MTP turned on or off) :

   LAYER_TYPES='[
   [32,"Q4_K_654"]
   ]'

This quant has the same layer 0..31 definitions as Q4_E_H with MTP layer quantized to Q4_K,VOD=654 while upsizing about 0.14G size for the MTP layer. If making use of this layer, it should be loaded fully into VRAM.

Comparison:

Quant	size	PPL	Comment
Q4_K_M	5.6e9	7.7	Q4_K_M with default embedding and output
Q4_K_H	6.1e9	7.8	Mixed precision quant with Q6_K embedding Q6_K
Q4_E_H	6.1e9	7.8	""
Q4_E_H_MTP	6.3e9	"	", including MTP layer at Q4_K_654

Usage:

Qwen3.5-9B is a vision capable dense RL model. It can be used together with its multimedia projector layers to process images and text inputs and generate text outputs. The mmproj file is made available in this repository.

Update 3/18/26: original mmproj had BF16 mmproj tensors. These are still available, unmodified, renamed to *.mmproj.BF16.gguf. New F16 mmproj format is the default to enable working across the widest range of platforms.

Straightforward speculation does not work with the model due to the attention scheme it uses. As of llama.cpp b9180 MTP support for the model was added to upstream and may be experimented with by using the Q4_E_H_MTP quant.

On a 4070 with all layers and context in VRAM with no vision tower approx performance is:

Q	QKV	NKV	gen tps
Q4_K_H	F16	166k +	63
Q4_K_H	Q8_0	280k +	64
Q4_E_H	F16	166k+	71 (b8986 checkpoints disabled)

Long context test (needle in haystack) was tested and passed with fast prompt processing, making large context actually usable with the model. The Q4_E_H prompt was tested against https://huggingface.co/datasets/steampunque/benchlm/blob/main/Qwen3_Runescape_Massive_Prompt.txt and did a pretty good job with it, though almost certainly this prompt is now in the training data of the model.

The model appears to be trained to decide itself whether to do a think block or not. When it does a think block it falls into very heavy overthinking but does come up with accurate answers. Over a small set of eval prompts the model did extremely well. To avoid the overthinking inject think start and think stop tokens first thing after assistant prompt:

THINK_START="<think>\n"
THINK_STOP="\n</think>\n\n"

If the model doesnt feel like doing thinking on a given prompt it will automatically do this. To force the model into a think block inject a bootstrap think block following the assistant prompt:

"<think>\nHere's a thinking process to solve the problem:"

The model was found to be highly capable on reasoning tasks when skipping think block. The model can fall into infinite rep loops on tricky/ambigous prompts when using greedy sampling. This is similar behaviour to other qwen3 thinkers which have trouble with infinite repeat when using greedy sampling particularly at smaller quant sizes (<10B params)

The model was tested in vision mode on a couple pretty tough bird ID image and did well, with a very detailed think block and accurate final conclusion (both Q4_K_H and Q4_E_H are strong)

The model was tested across a small set of code gen prompts and found to be quite intermittent in its ability to generate working code, and went into infinite repeat on two of the code prompts where it decided to use a think block when using greedy sampling. Q4_E_H seems slightly better at generating working code.

Llama.cpp minimum version to run Qwen3.5-9B should be b8148 and above due to correction of a graph error which causes crashes in both RPC and multiple local GPU setups. If the model run over RPC it will crash due to an unresolved memory leak in RPC: https://github.com/ggml-org/llama.cpp/issues/19892, temp workaround set GGML_CUDA_DISABLE_GRAPHS=1 on rpc server launch.

Benchmarks:

A full set of both math and vision benchmarks for the model will eventually be given here: https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

Link	Type	Size/e9 B	Notes
Qwen3.5-9B.Q4_K_H.gguf	Q4_K_H	6.1e9 B	0.6B bigger than Q4_K_M
Qwen3.5-9B.Q4_E_H.gguf	Q4_E_H	6.1e9 B	0.6B bigger than Q4_K_M
Qwen3.5-9B.Q4_E_H_MTP.gguf	Q4_E_H_MTP	6.3e9 B	0.14B bigger than Q4_E_H
Qwen3.5-9B.mmproj.gguf	F16	0.92e9 B	multimedia projector
Qwen3.5-9B.mmproj.BF16.gguf	BF16	0.92e9 B	multimedia projector

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040