Instructions to use mudler/Inkling-2-Mini-APEX-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use mudler/Inkling-2-Mini-APEX-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf mudler/Inkling-2-Mini-APEX-GGUF # Run inference directly in the terminal: llama cli -hf mudler/Inkling-2-Mini-APEX-GGUF
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf mudler/Inkling-2-Mini-APEX-GGUF # Run inference directly in the terminal: llama cli -hf mudler/Inkling-2-Mini-APEX-GGUF
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf mudler/Inkling-2-Mini-APEX-GGUF # Run inference directly in the terminal: ./build/bin/llama cli -hf mudler/Inkling-2-Mini-APEX-GGUF
Use Docker
docker model run hf.co/mudler/Inkling-2-Mini-APEX-GGUF
- Llama new
- LM Studio
- Jan
- Ollama
How to use mudler/Inkling-2-Mini-APEX-GGUF with Ollama:
ollama run hf.co/mudler/Inkling-2-Mini-APEX-GGUF
- Unsloth Desktop
- Pi
How to use mudler/Inkling-2-Mini-APEX-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf mudler/Inkling-2-Mini-APEX-GGUF
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "mudler/Inkling-2-Mini-APEX-GGUF" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use mudler/Inkling-2-Mini-APEX-GGUF with Docker Model Runner:
docker model run hf.co/mudler/Inkling-2-Mini-APEX-GGUF
- Lemonade
How to use mudler/Inkling-2-Mini-APEX-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull mudler/Inkling-2-Mini-APEX-GGUF
Run and chat with the model
lemonade run user.Inkling-2-Mini-APEX-GGUF-{{QUANT_TAG}}List all available models
lemonade list
- Hermes Agent
How to use mudler/Inkling-2-Mini-APEX-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf mudler/Inkling-2-Mini-APEX-GGUF
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default mudler/Inkling-2-Mini-APEX-GGUF
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use mudler/Inkling-2-Mini-APEX-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf mudler/Inkling-2-Mini-APEX-GGUF
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "mudler/Inkling-2-Mini-APEX-GGUF" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
โก Each donation = another big MoE quantized
I host 30+ free APEX MoE quantizations as independent research. My only local hardware is an NVIDIA DGX Spark (122 GB unified memory), enough for ~30-50B-class MoEs, but bigger ones (200B+) require rented compute on H100/H200/Blackwell, typically $20-100 per quant.
If APEX quants are useful to you, your support directly funds those bigger runs.
๐ Patreon (Monthly) | โ Buy Me a Coffee | โญ GitHub Sponsors
Inkling-2-Mini APEX GGUF
APEX quantizations of thinkingmachines/Inkling-2-Mini.
Brought to you by the LocalAI team | APEX Project
Text only: the vision tower is not included. Perplexity and KL divergence are measured for three tiers (see Evaluation below). The other tiers have not been evaluated, and no benchmark suite has run.
Files
| File | Size | For |
|---|---|---|
| Inkling-2-Mini-APEX-Quality.gguf | 20.07 GB | highest quality |
| Inkling-2-Mini-APEX-I-Quality.gguf | 20.07 GB | highest quality, imatrix |
| Inkling-2-Mini-APEX-Balanced.gguf | 20.68 GB | general purpose |
| Inkling-2-Mini-APEX-I-Balanced.gguf | 20.68 GB | general purpose, imatrix |
| Inkling-2-Mini-APEX-Compact.gguf | 14.10 GB | consumer GPUs |
| Inkling-2-Mini-APEX-I-Compact.gguf | 14.10 GB | consumer GPUs, imatrix |
| Inkling-2-Mini-APEX-I-Mini.gguf | 12.36 GB | small, imatrix only |
| Inkling-2-Mini-APEX-I-Nano.gguf | 11.34 GB | smallest, imatrix only |
I- files use an importance matrix built from diverse calibration data (chat, code, reasoning, tool-calling, agentic traces, Wikipedia).
The official checkpoint is about 19 GB (4-bit and 8-bit float weights). The Quality and Balanced tiers are not smaller than it. They keep more bits per weight than the source stored, so they are for people who want a standard GGUF with the k-quant formats, not for saving space.
Evaluation
Perplexity and KL divergence against the BF16 model, for the three tiers below. The other tiers have not been evaluated, and no benchmark suite has run.
| Tier | Size | PPL | Mean KLD | Median KLD | 99.9% KLD | Same top token |
|---|---|---|---|---|---|---|
| BF16 (reference) | 52.23 GB | 6.862 | ||||
| APEX-Quality | 20.07 GB | 6.873 | 0.0063 | 0.0029 | 0.174 | 96.5 % |
| APEX-Compact | 14.10 GB | 6.965 | 0.0245 | 0.0127 | 0.632 | 93.2 % |
| APEX-I-Nano | 11.34 GB | 7.227 | 0.0701 | 0.0386 | 1.480 | 88.6 % |
Method: llama.cpp 29b52d398 with CUDA, llama-perplexity on the wikitext-2 test split, context 512, all chunks (same corpus, context size and chunk count for the BF16 reference and every tier). KL divergence uses logits saved once from the BF16 GGUF. The BF16 PPL is 6.862 on its own pass and 6.853 on the pass used for the KL ratios. A short chat prompt gave a coherent answer on every tier.
What is APEX?
APEX (Adaptive Precision for EXpert Models) quantizes a Mixture-of-Experts model by tensor role and layer position. Routed experts hold most of the weights but only a few fire for each token, so they tolerate lower precision. The first and last layers keep higher precision, middle layers compress harder, and attention follows the same gradient.
The model
Inkling-2-Mini is a looped Mixture-of-Experts model from Thinking Machines Lab: 24 layers, hidden size 2304, 128 routed experts with 8 active per token and no shared expert. Layers 6 to 17 form a core that runs three times per token. It is multimodal upstream; these files carry the language model only.
Expert FFN width is 1152, which is not a multiple of 256. llama.cpp therefore stores the ffn_down experts in a legacy 32-wide type when a tier asks for a k-quant or iq type. This is why the small tiers shrink less than the type names suggest.
Usage
llama-cli -m Inkling-2-Mini-APEX-Balanced.gguf -p "Your prompt" -ngl 99
Needs llama.cpp built from commit 29b52d398 (master after PR 32694, "Inkling 2 Mini") or later. The model is a thinking model: give it enough tokens (at least 120) before judging an answer.
License
Apache 2.0, the same as the base model. See the base model card for its acceptable use policy.
Credits
Base model by Thinking Machines Lab. llama.cpp support by the ggml-org contributors. Quantized by the LocalAI team.
- Downloads last month
- -
We're not able to determine the quantization variants.
Model tree for mudler/Inkling-2-Mini-APEX-GGUF
Base model
thinkingmachines/Inkling-2-Mini