Inkling-2-Mini-oQ2.5e-mtp

This model was quantized using oQ (oMLX v0.8.1) mixed-precision quantization.

A mixed-precision MLX checkpoint of thinkingmachines/Inkling-2-Mini, a looped multimodal mixture-of-experts model with 26.4B total and 3.4B active parameters. Its middle 12 layers run three times for each token. It takes text and image inputs with a context window of up to 1M tokens. It includes the language model, the vision encoder and the bundled DSpark draft in mtp.safetensors, which oMLX Lightning MTP uses for speculative decoding.

It needs oMLX 0.8.1 or later.

The effective average is 3.305 bits/weight for the language model and vision encoder, or 3.457 bits/weight including the DSpark draft.

Quantization and Bit Distribution

oQ2.5e requantizes the routed experts of all layers. The gate and up projections use affine 2-bit. The down projections use affine 3-bit in 19 layers, including all looped layers, and affine 2-bit in layers 3, 4, 5, 20 and 22. It stores the attention projections, the embeddings and the vocabulary head as affine 8-bit. The vision encoder and the DSpark draft keep BF16, and the router and the loop controllers keep FP32.

Storage format Logical weights Share of total Tensor storage Effective bits/weight
Affine 2-bit, group 64 18.006B 68.234% 5.240 GiB 2.50
Affine 3-bit, group 64 6.455B 24.461% 2.630 GiB 3.50
Affine 8-bit, group 64 1.572B 5.958% 1.556 GiB 8.50
BF16 0.320B 1.211% 0.595 GiB 16.00
FP32 0.036B 0.135% 0.133 GiB 32.00
Total 26.389B 100% 10.154 GiB 3.305

Storage Breakdown

Component Logical weights Storage
Language model 26.070B 10.267 GB / 9.562 GiB
Vision 0.318B 0.637 GB / 0.593 GiB
DSpark draft (MTP) 0.318B 0.636 GB / 0.592 GiB
Total 26.707B 11.539 GB / 10.746 GiB

License

Apache 2.0, following the source model.

Downloads last month
129
Safetensors
Model size
26B params
Tensor type
U32
路
BF16
路
F32
路
MLX
Hardware compatibility
Log In to add your hardware

2-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 馃檵 Ask for provider support

Model tree for mlx-community/Inkling-2-Mini-oQ2.5e-mtp

Quantized
(64)
this model