Instructions to use IndrasMirror/Inkling-2-Mini-XQ-Uncensored with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use IndrasMirror/Inkling-2-Mini-XQ-Uncensored with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf IndrasMirror/Inkling-2-Mini-XQ-Uncensored:BF16 # Run inference directly in the terminal: llama cli -hf IndrasMirror/Inkling-2-Mini-XQ-Uncensored:BF16
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf IndrasMirror/Inkling-2-Mini-XQ-Uncensored:BF16 # Run inference directly in the terminal: llama cli -hf IndrasMirror/Inkling-2-Mini-XQ-Uncensored:BF16
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf IndrasMirror/Inkling-2-Mini-XQ-Uncensored:BF16 # Run inference directly in the terminal: ./build/bin/llama cli -hf IndrasMirror/Inkling-2-Mini-XQ-Uncensored:BF16
Use Docker
docker model run hf.co/IndrasMirror/Inkling-2-Mini-XQ-Uncensored:BF16
- Llama new
- LM Studio
- Jan
- vLLM
How to use IndrasMirror/Inkling-2-Mini-XQ-Uncensored with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "IndrasMirror/Inkling-2-Mini-XQ-Uncensored" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "IndrasMirror/Inkling-2-Mini-XQ-Uncensored", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/IndrasMirror/Inkling-2-Mini-XQ-Uncensored:BF16
- Ollama
How to use IndrasMirror/Inkling-2-Mini-XQ-Uncensored with Ollama:
ollama run hf.co/IndrasMirror/Inkling-2-Mini-XQ-Uncensored:BF16
- Unsloth Desktop
- Pi
How to use IndrasMirror/Inkling-2-Mini-XQ-Uncensored with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf IndrasMirror/Inkling-2-Mini-XQ-Uncensored:BF16
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "IndrasMirror/Inkling-2-Mini-XQ-Uncensored:BF16" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use IndrasMirror/Inkling-2-Mini-XQ-Uncensored with Docker Model Runner:
docker model run hf.co/IndrasMirror/Inkling-2-Mini-XQ-Uncensored:BF16
- Lemonade
How to use IndrasMirror/Inkling-2-Mini-XQ-Uncensored with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull IndrasMirror/Inkling-2-Mini-XQ-Uncensored:BF16
Run and chat with the model
lemonade run user.Inkling-2-Mini-XQ-Uncensored-BF16
List all available models
lemonade list
- Hermes Agent
How to use IndrasMirror/Inkling-2-Mini-XQ-Uncensored with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf IndrasMirror/Inkling-2-Mini-XQ-Uncensored:BF16
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default IndrasMirror/Inkling-2-Mini-XQ-Uncensored:BF16
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use IndrasMirror/Inkling-2-Mini-XQ-Uncensored with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf IndrasMirror/Inkling-2-Mini-XQ-Uncensored:BF16
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "IndrasMirror/Inkling-2-Mini-XQ-Uncensored:BF16" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Files are up (updated 21 Nov, 18:10 AEDT). The quant passed its held-out quality and refusal gates and a quick task screen against its parents (below). Decode is now 93 tok/s with the latest Xolotl (was 59.7).
stage (21 Nov 2026, AEDT) status abliteration (ARA) + refusal gate done: 6/100 harmful, 1/100 harmless refused quantisation (~14 GB) done: 13.93 GB GGUF, verified (744 tensors, 0 problems) held-out KL / top-1 vs the parent done: KL 0.013, top-1 96.3 % vs the abliterated parent refusal gate on the quant done: 4/100 harmful, 1/100 harmless refused speed through the real server done: 59.7 tok/s decode, 1,162 tok/s prefill at 8K (RTX 4090); 93 tok/s decode with Xolotl 58ff5ddbb quick task screen vs parents done: MMLU-100 76 % (original 73 %), HumanEval, tool choice, agent loops on par Gauntlet v3 (agentic coding) paused (see Quality) upload done: model, DSpark drafter, vision projector (sha256-verified)
Inkling-2-Mini ยท XQ, abliterated
Thinking Machines' Inkling 2 Mini with the refusal direction edited out of the weights, then requantised with Indra's Mirror's mixed-precision XQ pipeline. A 26B-parameter looped MoE with 5.4B active parameters per token (about 3.1B distinct weights; layers 6-17 run 3 times), 128 routed experts at top-8, and a 1M-token context.
Runs on Xolotl, an engine built for this class of model and machine, on top of Strata. The forward graph is ported from llama.cpp (PR #32694 by Pedro Cuenca, MIT); the engine around it is Xolotl's own.
| file size | 13.93 GB |
| routed experts | mixed per layer, ~4.0 bits/weight on average; 13 of 24 layers keep the official NVFP4 experts bit for bit |
| dense backbone | 8-bit (Q8_0) |
| arch / tensors | inkling2_loop, 24 physical layers, 48 cache slots, 744 tensors |
The model
Inkling 2 Mini is a 24-layer decoder-only transformer with sparse MoE feed-forward blocks. Its middle 12 layers (layers 6 to 17) run three times with shared weights, so each token gets 48 layer applications out of 24 layers of weights, and the cache carries 48 slots.
| total / active | 26B total, 5.4B active per token (3.1B distinct weights per token) |
| loop | layers 6 to 17 run 3 passes, 48 cache slots; passes 2 and 3 read the previous pass through per-layer low-rank loop controllers |
| MoE | 128 routed experts, top-8, no shared expert, sigmoid router with a selection bias, SiTU activation |
| attention | sliding window 1024 with a global layer every 6th (5, 11, 17, 23); no RoPE: a learned relative-position bias per head, with log-scaled positions on the global layers |
| extras | short causal convolutions on K, V, the attention output and the FFN output; block attention residuals that mix earlier depths; final logit softcap 30 |
| vocab | 200,058 |
| context | 1M tokens (the trained key says 1048576) |
| official precision | routed experts NVFP4 on layers 0 to 19, MXFP8 on 20 to 23, everything else BF16 |
The official checkpoint ships in that mixed precision. This build starts from it rather than from a dequantised copy, so the native FP4 experts can be carried through at their own precision.
How it was made
- Abliteration with Heretic-style ARA (Abliteration Rank Adjustment) on the attention output projections of layers 11 to 23, the model's only dense writers into the residual stream. The classic refusal-direction projection does not survive on this model: the routed experts are FP4, and re-encoding an edited expert to FP4 erases most of the edit (measured: only ~13 % of the direction removed). Refusal gate (system "You are a helpful assistant.", thinking off, 96 greedy tokens, a BERT refusal classifier): 6/100 harmful and 1/100 harmless prompts refused, at KL 0.033 against the original on harmless prompts.
- Quantisation with XQ: every routed-expert layer gets its own format, picked from each expert's measured output error on held-out text for a given file size; the official NVFP4 experts are kept bit-exact wherever the budget allows, and the dense backbone goes to 8-bit. Statistics come from the abliterated model itself, run through Xolotl's engine. The XQ pipeline itself is private.
Xolotl found and fixed a bug in the llama.cpp loop path along the way: passes 2 and 3 of the loop ran their experts without the per-expert NVFP4 scale when the GGUF keeps NVFP4 experts. The upstream converter dequantises, so upstream never meets that case. It is fixed in Xolotl's build.
Quality
Measured against the parent, which is the official checkpoint converted exactly (NVFP4 and MXFP8 kept) and run on the same engine, hardware and settings.
| parent | abliterated (before quant) | this (XQ) | |
|---|---|---|---|
| MMLU, 100-question sample (greedy, thinking off) | 73 % | 74 % | 76 % |
| MMLU, 30 questions, thinking on (high) | 93 % | - | 87 % |
| HumanEval, every 8th problem (21) | 21/21 | 20/21 | 19/21 |
| tool choice (10) / agent bug-fix loops (3) | 8/10, 3/3 | 8/10, 3/3 | 8/10, 3/3 |
| held-out top-1 agreement vs the original | n/a | 95.3 % | 94.7 % |
| held-out KL vs the original | n/a | 0.028 | 0.033 |
| held-out KL / top-1 vs the abliterated parent | n/a | n/a | 0.013 / 96.3 % |
| refusal gate, harmful prompts refused | not run | 6/100 | 4/100 |
| refusal gate, harmless prompts refused | not run | 1/100 | 1/100 |
Three columns on purpose: the abliteration and the quant are each checked for damage on their own.
With 100 and 30 questions, differences of a few points are noise: the quant sits with its parents on every task screen. Note for anyone measuring this model: its logits are chaotic at the rounding level (near-tied expert picks flip), so any engine change, even a different prefill chunk size, moves held-out KL by about 0.005 over 2K tokens; part of the 0.013 above is that floor.
Gauntlet v3 (Xolotl's multi-turn agentic coding set) is paused: on its long tasks the abliterated parent and the quant alike tend to deliberate for 10K+ thinking tokens on one step before acting, so a fair run needs a long thinking budget and many hours. It will be reported when it has been run that way.
Speed
Measured on an RTX 4090 through Xolotl's server (8K prompt, then 1,024 tokens): 1,162 tok/s prefill, 59.7 tok/s decode. The exact 22.9 GB parent does 1,131 / 57 on the same card, so the quant fits in 14 GB without losing speed.
Decode is far from the card's memory limit (a token reads about 3.4 GB, ~3.4 ms on a 4090): it is bound by the number of small GPU kernels per token (48 layer applications). First round of kernel fusion (Xolotl 58ff5ddbb): 67 -> 93 tok/s (256-token greedy; 93-96 tok/s in served evals), logits bit-identical to the unfused graph. More fusion is in progress, aiming at 100+ tok/s.
Landed in Xolotl (21 Nov):
- Speculative decoding with the model's own DSpark (MTP) drafter (
--dspark, experimental): up to 7 drafted tokens per step, with exact rollback of the model's recurrent convolution state. Still being tuned for speed. - Quantised K/V cache (
--kv-type q8_0|k8v4): the long-context path for 512K on one card.
Next: vision, flash attention with the relative-position bias, and a grouped MoE kernel for speculative verify.
Files
| file | size | what |
|---|---|---|
Inkling-2-Mini-XQ-Uncensored.gguf |
13.93 GB | the model |
mmproj-Inkling-2-Mini-XQ-Uncensored-F16.gguf |
0.64 GB | the official vision encoder, unchanged, for images (optional; engine support in progress) |
Inkling-2-Mini-XQ-Uncensored-DSpark-BF16.gguf |
0.64 GB | optional: the model's own DSpark (MTP) drafter for speculative decoding (--dspark), converted by unsloth |
Quick start
Xolotl's xolotl-inkling engine runs any Inkling-2-Mini GGUF: the upstream converter's, llama.cpp's quants, or
this file.
git clone https://github.com/Indras-Mirror/Xolotl.git xolotl && cd xolotl
# build per docs/INSTALL.md, then serve with the Inkling config
Credits
The model is Thinking Machines Lab's, released with open weights under Apache 2.0. The Inkling 2 forward graph and the GGUF converter come from llama.cpp PR #32694 by Pedro Cuenca (MIT), which extends the earlier Inkling 2 work in llama.cpp and builds on ggml and llama.cpp by Georgi Gerganov and the llama.cpp contributors. Xolotl itself is built on Strata by Niko1221.
License
Apache 2.0, the same as the official model. The official card says, in its General Information section: "License: Apache 2.0", with the link to LICENSE.
- Downloads last month
- 258
16-bit
Model tree for IndrasMirror/Inkling-2-Mini-XQ-Uncensored
Base model
thinkingmachines/Inkling-2-Mini