rkumar70900/qwen2.5-1.5b-gguf-experiments

Qwen2.5-1.5B-Instruct — GGUF Quantization Experiments

This repo contains Qwen2.5-1.5B-Instruct quantized into multiple GGUF formats using llama.cpp. It was created as part of a hands-on quantization experiment documenting the full process from raw HuggingFace weights → multiple GGUF formats → quality evaluation.

---

What's in This Repo

gguf/
├── qwen2.5-1.5b-f16.gguf          ~2.9 GB   source of truth — full precision
├── qwen2.5-1.5b-Q8_0.gguf         ~1.6 GB   near-lossless
├── qwen2.5-1.5b-Q5_K_M.gguf       ~1.0 GB   great quality/size tradeoff
├── qwen2.5-1.5b-Q4_K_M.gguf       ~935 MB   recommended — sweet spot ★
├── qwen2.5-1.5b-Q4_K_S.gguf       ~865 MB   leaner 4-bit variant
├── qwen2.5-1.5b-Q2_K.gguf         ~530 MB   aggressive K-quant baseline
├── qwen2.5-1.5b-Q2_K_S.gguf       ~530 MB   aggressive K-quant — needs imatrix
├── qwen2.5-1.5b-IQ3_M.gguf        ~680 MB   importance-weighted 3-bit
├── qwen2.5-1.5b-IQ2_XS.gguf       ~480 MB   importance-weighted 2-bit — needs imatrix
├── qwen2.5-1.5b-IQ2_XXS.gguf      ~420 MB   most aggressive — needs imatrix
├── qwen2.5-1.5b-IQ2_S.gguf        ~450 MB   importance-weighted 2.5-bit — needs imatrix
├── qwen2.5-1.5b-IQ1_M.gguf        ~300 MB   extreme 1.75-bit — needs imatrix
└── qwen2.5-1.5b-IQ1_S.gguf        ~280 MB   extreme 1.56-bit — needs imatrix

Note on f16: The F16 file is included as the reference baseline for perplexity comparisons. It is not intended for general inference use — at 2.9 GB it offers no practical advantage over Q8_0 for local deployment.

---

Which File Should I Use?

Use Case	Recommended Format
Best quality, VRAM not a concern	Q8_0
Daily driver — best quality/size tradeoff	Q4_K_M ← start here
Tight on memory, want decent quality	Q4_K_S or Q2_K
Edge deployment / very limited RAM	IQ2_XS or IQ2_XXS
Research / extreme compression testing	IQ1_M or IQ1_S
Partial GPU offload (CPU + GPU split)	Q4_K_M or IQ3_M

If you're not sure, start with Q4_K_M. It's the most tested format in the community and gives you ~68% size reduction with minimal quality loss.

⚠️ IQ1 and IQ2 formats (IQ1_S, IQ1_M, IQ2_S, IQ2_XS, IQ2_XXS, Q2_K_S) were all generated with an importance matrix. Without one, these formats produce significantly degraded output. See the Imatrix Calibration section below for details.

---

Format Guide

K-Quant Family (Q*_K_*)

Standard llama.cpp quantization using superblocks of 256 weights. The suffix means:

• _S (Small) — more aggressive, smaller file
• _M (Medium) — mixed-precision, smarter assignment of bits to sensitive layers

Despite the "4" in Q4_K_M, it is not uniform 4-bit. Critical tensors like the embedding table and output projection are bumped to 6-bit internally. The "4" is the average bits-per-weight.

IQ Family (IQ*_*)

Importance-weighted quantization. These formats use an importance matrix — calibration data was run through the base model to identify which weights matter most, and precision was distributed accordingly. This is why IQ formats punch above their weight class at the same file size compared to K-quants.

The IQ2 files in this repo were generated with a WikiText-2 calibration dataset (see below). Without an importance matrix, these formats produce near-incoherent output — the imatrix is what makes them viable.

---

Quantization Details

Base model: Qwen/Qwen2.5-1.5B-Instruct
Quantization tool: llama.cpp build 7074 (commit 22e1ce2f8)
Source precision: F16 GGUF (converted from original SafeTensors)
Platform: Apple Silicon (arm64)

Imatrix Calibration

The IQ2 formats were quantized using an importance matrix generated from WikiText-2:

from datasets import load_dataset

dataset = load_dataset("wikitext", "wikitext-2-raw-v1", split="train")

with open("calibration.txt", "w") as f:
    for row in dataset:
        text = row["text"].strip()
        if len(text) > 100:
            f.write(text + "\n")

./build/bin/llama-imatrix \
    -m qwen2.5-1.5b-f16.gguf \
    -f calibration.txt \
    -o imatrix.dat \
    --ctx-size 512 \
    -ngl -1 \
    --chunks 100

---

How to Run

llama.cpp CLI

./build/bin/llama-cli \
    -m qwen2.5-1.5b-Q4_K_M.gguf \
    -n 512 \
    -ngl 99 \
    --prompt "Explain the difference between supervised and unsupervised learning."

llama.cpp Server (OpenAI-compatible)

./build/bin/llama-server \
    -m qwen2.5-1.5b-Q4_K_M.gguf \
    -ngl 99 \
    --port 8080

Then hit http://localhost:8080/v1/chat/completions like any OpenAI endpoint.

Python (llama-cpp-python)

from llama_cpp import Llama

llm = Llama(
    model_path="qwen2.5-1.5b-Q4_K_M.gguf",
    n_gpu_layers=-1,  # full GPU offload
    n_ctx=4096,
)

output = llm(
    "Explain quantization in simple terms:",
    max_tokens=256,
    temperature=0.7,
)
print(output["choices"][0]["text"])

Ollama

ollama run hf.co/your-username/qwen2.5-1.5b-gguf-experiments:Q4_K_M

---

Model Architecture (from metadata)

Parameter	Value
Architecture	Qwen2
Parameters	1.5B
Layers	28
Hidden dimension	1536
FFN intermediate	8960
Attention heads (Q)	12
Attention heads (KV)	2
Attention type	Grouped Query Attention (GQA)
Context length	32768
Vocabulary size	151,936
Tokenizer	GPT-2 BPE (Qwen2 variant)

---

License

The quantized weights in this repo are derived from Qwen/Qwen2.5-1.5B-Instruct and inherit its Apache 2.0 license.

---

Citation

If you use these files in your work, please also cite the original Qwen2.5 model:

@misc{qwen2.5,
    title  = {Qwen2.5: A Party of Foundation Models},
    author = {Qwen Team},
    year   = {2024},
    url    = {https://qwenlm.github.io/blog/qwen2.5/}
}