How to use from
llama.cpp
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mudler/Inkling-2-Mini-APEX-GGUF
# Run inference directly in the terminal:
llama cli -hf mudler/Inkling-2-Mini-APEX-GGUF
Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mudler/Inkling-2-Mini-APEX-GGUF
# Run inference directly in the terminal:
llama cli -hf mudler/Inkling-2-Mini-APEX-GGUF
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf mudler/Inkling-2-Mini-APEX-GGUF
# Run inference directly in the terminal:
./build/bin/llama cli -hf mudler/Inkling-2-Mini-APEX-GGUF
Use Docker
docker model run hf.co/mudler/Inkling-2-Mini-APEX-GGUF
Quick Links

โšก Each donation = another big MoE quantized

I host 30+ free APEX MoE quantizations as independent research. My only local hardware is an NVIDIA DGX Spark (122 GB unified memory), enough for ~30-50B-class MoEs, but bigger ones (200B+) require rented compute on H100/H200/Blackwell, typically $20-100 per quant.
If APEX quants are useful to you, your support directly funds those bigger runs.

๐ŸŽ‰ Patreon (Monthly)  |  โ˜• Buy Me a Coffee  |  โญ GitHub Sponsors

Inkling-2-Mini APEX GGUF

APEX quantizations of thinkingmachines/Inkling-2-Mini.

Brought to you by the LocalAI team | APEX Project

Text only: the vision tower is not included. Perplexity and KL divergence are measured for three tiers (see Evaluation below). The other tiers have not been evaluated, and no benchmark suite has run.

Files

File Size For
Inkling-2-Mini-APEX-Quality.gguf 20.07 GB highest quality
Inkling-2-Mini-APEX-I-Quality.gguf 20.07 GB highest quality, imatrix
Inkling-2-Mini-APEX-Balanced.gguf 20.68 GB general purpose
Inkling-2-Mini-APEX-I-Balanced.gguf 20.68 GB general purpose, imatrix
Inkling-2-Mini-APEX-Compact.gguf 14.10 GB consumer GPUs
Inkling-2-Mini-APEX-I-Compact.gguf 14.10 GB consumer GPUs, imatrix
Inkling-2-Mini-APEX-I-Mini.gguf 12.36 GB small, imatrix only
Inkling-2-Mini-APEX-I-Nano.gguf 11.34 GB smallest, imatrix only

I- files use an importance matrix built from diverse calibration data (chat, code, reasoning, tool-calling, agentic traces, Wikipedia).

The official checkpoint is about 19 GB (4-bit and 8-bit float weights). The Quality and Balanced tiers are not smaller than it. They keep more bits per weight than the source stored, so they are for people who want a standard GGUF with the k-quant formats, not for saving space.

Evaluation

Perplexity and KL divergence against the BF16 model, for the three tiers below. The other tiers have not been evaluated, and no benchmark suite has run.

Tier Size PPL Mean KLD Median KLD 99.9% KLD Same top token
BF16 (reference) 52.23 GB 6.862
APEX-Quality 20.07 GB 6.873 0.0063 0.0029 0.174 96.5 %
APEX-Compact 14.10 GB 6.965 0.0245 0.0127 0.632 93.2 %
APEX-I-Nano 11.34 GB 7.227 0.0701 0.0386 1.480 88.6 %

Method: llama.cpp 29b52d398 with CUDA, llama-perplexity on the wikitext-2 test split, context 512, all chunks (same corpus, context size and chunk count for the BF16 reference and every tier). KL divergence uses logits saved once from the BF16 GGUF. The BF16 PPL is 6.862 on its own pass and 6.853 on the pass used for the KL ratios. A short chat prompt gave a coherent answer on every tier.

What is APEX?

APEX (Adaptive Precision for EXpert Models) quantizes a Mixture-of-Experts model by tensor role and layer position. Routed experts hold most of the weights but only a few fire for each token, so they tolerate lower precision. The first and last layers keep higher precision, middle layers compress harder, and attention follows the same gradient.

The model

Inkling-2-Mini is a looped Mixture-of-Experts model from Thinking Machines Lab: 24 layers, hidden size 2304, 128 routed experts with 8 active per token and no shared expert. Layers 6 to 17 form a core that runs three times per token. It is multimodal upstream; these files carry the language model only.

Expert FFN width is 1152, which is not a multiple of 256. llama.cpp therefore stores the ffn_down experts in a legacy 32-wide type when a tier asks for a k-quant or iq type. This is why the small tiers shrink less than the type names suggest.

Usage

llama-cli -m Inkling-2-Mini-APEX-Balanced.gguf -p "Your prompt" -ngl 99

Needs llama.cpp built from commit 29b52d398 (master after PR 32694, "Inkling 2 Mini") or later. The model is a thinking model: give it enough tokens (at least 120) before judging an answer.

License

Apache 2.0, the same as the base model. See the base model card for its acceptable use policy.

Credits

Base model by Thinking Machines Lab. llama.cpp support by the ggml-org contributors. Quantized by the LocalAI team.

Downloads last month
-
GGUF
Model size
26B params
Architecture
inkling2_loop
Hardware compatibility
Log In to add your hardware

We're not able to determine the quantization variants.

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for mudler/Inkling-2-Mini-APEX-GGUF

Quantized
(64)
this model

Collection including mudler/Inkling-2-Mini-APEX-GGUF