Instructions to use mlx-community/gemma-4-e4b-it-mxfp8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mlx-community/gemma-4-e4b-it-mxfp8 with MLX:

# Make sure mlx-vlm is installed
# pip install --upgrade mlx-vlm

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

# Load the model
model, processor = load("mlx-community/gemma-4-e4b-it-mxfp8")
config = load_config("mlx-community/gemma-4-e4b-it-mxfp8")

# Prepare input
image = ["http://images.cocodataset.org/val2017/000000039769.jpg"]
prompt = "Describe this image."

# Apply chat template
formatted_prompt = apply_chat_template(
    processor, config, prompt, num_images=1
)

# Generate output
output = generate(model, processor, formatted_prompt, image)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps Settings
LM Studio

How to use mlx-community/gemma-4-e4b-it-mxfp8 with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "mlx-community/gemma-4-e4b-it-mxfp8"

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "mlx-community/gemma-4-e4b-it-mxfp8"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent

How to use mlx-community/gemma-4-e4b-it-mxfp8 with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "mlx-community/gemma-4-e4b-it-mxfp8"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default mlx-community/gemma-4-e4b-it-mxfp8

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use mlx-community/gemma-4-e4b-it-mxfp8 with OpenClaw:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "mlx-community/gemma-4-e4b-it-mxfp8"

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "mlx-community/gemma-4-e4b-it-mxfp8" \
  --custom-provider-id mlx-lm \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Failed to load the model

by elmagic7379 - opened Apr 4

Discussion

elmagic7379

Apr 4

🥲 Failed to load the model

Failed to load model.

Error when loading model: ValueError: Gemma 4 support is not ready yet, stay tuned!

masoori

May 18

I'm getting this in MLX

2026-05-18 04:35:57,501 - INFO - Starting httpd at 127.0.0.1 on port 8080...
Fetching 9 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [04:56<00:00, 32.99s/it]
Download complete: : 8.76GB [04:56, 156MB/s] Exception in thread Thread-1 (_generate): | 4/9 [04:56<06:41, 80.24s/it]
Traceback (most recent call last):
File "/opt/homebrew/Cellar/python@3.14/3.14.4_1/Frameworks/Python.framework/Versions/3.14/lib/python3.14/threading.py", line 1082, in _bootstrap_inner
self._context.run(self.run)
~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.14/3.14.4_1/Frameworks/Python.framework/Versions/3.14/lib/python3.14/threading.py", line 1024, in run
self._target(*self._args, **self._kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mahesh/prj/my_mlx/.venv/lib/python3.14/site-packages/mlx_lm/server.py", line 695, in _generate
self.model_provider.load_default()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/Users/mahesh/prj/my_mlx/.venv/lib/python3.14/site-packages/mlx_lm/server.py", line 385, in load_default
self.load("default_model", None, "default_model")
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mahesh/prj/my_mlx/.venv/lib/python3.14/site-packages/mlx_lm/server.py", line 394, in load
self._load(*model_key)
~~~~~~~~~~^^^^^^^^^^^^
File "/Users/mahesh/prj/my_mlx/.venv/lib/python3.14/site-packages/mlx_lm/server.py", line 349, in _load
model, tokenizer = load(
~~~~^
model_path,
^^^^^^^^^^^
adapter_path=adapter_path,
^^^^^^^^^^^^^^^^^^^^^^^^^^
tokenizer_config=self._tokenizer_config,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/Users/mahesh/prj/my_mlx/.venv/lib/python3.14/site-packages/mlx_lm/utils.py", line 491, in load
model, config = load_model(model_path, lazy, model_config=model_config)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mahesh/prj/my_mlx/.venv/lib/python3.14/site-packages/mlx_lm/utils.py", line 415, in load_model
model.load_weights(list(weights.items()), strict=strict)
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mahesh/prj/my_mlx/.venv/lib/python3.14/site-packages/mlx/nn/layers/base.py", line 185, in load_weights
raise ValueError(
f"Received {num_extra} parameters not in model: \n{extras}."
)
ValueError: Received 90 parameters not in model:
language_model.model.layers.24.self_attn.k_norm.weight,
language_model.model.layers.24.self_attn.k_proj.scales,
language_model.model.layers.24.self_attn.k_proj.weight,
language_model.model.layers.24.self_attn.v_proj.scales,
language_model.model.layers.24.self_attn.v_proj.weight,
language_model.model.layers.25.self_attn.k_norm.weight,
language_model.model.layers.25.self_attn.k_proj.scales,
language_model.model.layers.25.self_attn.k_proj.weight,
language_model.model.layers.25.self_attn.v_proj.scales,
language_model.model.layers.25.self_attn.v_proj.weight,
language_model.model.layers.26.self_attn.k_norm.weight,
language_model.model.layers.26.self_attn.k_proj.scales,
language_model.model.layers.26.self_attn.k_proj.weight,
language_model.model.layers.26.self_attn.v_proj.scales,
language_model.model.layers.26.self_attn.v_proj.weight,
language_model.model.layers.27.self_attn.k_norm.weight,
language_model.model.layers.27.self_attn.k_proj.scales,
language_model.model.layers.27.self_attn.k_proj.weight,
language_model.model.layers.27.self_attn.v_proj.scales,
language_model.model.layers.27.self_attn.v_proj.weight,
language_model.model.layers.28.self_attn.k_norm.weight,
language_model.model.layers.28.self_attn.k_proj.scales,
language_model.model.layers.28.self_attn.k_proj.weight,
language_model.model.layers.28.self_attn.v_proj.scales,
language_model.model.layers.28.self_attn.v_proj.weight,
language_model.model.layers.29.self_attn.k_norm.weight,
language_model.model.layers.29.self_attn.k_proj.scales,
language_model.model.layers.29.self_attn.k_proj.weight,
language_model.model.layers.29.self_attn.v_proj.scales,
language_model.model.layers.29.self_attn.v_proj.weight,
language_model.model.layers.30.self_attn.k_norm.weight,
language_model.model.layers.30.self_attn.k_proj.scales,
language_model.model.layers.30.self_attn.k_proj.weight,
language_model.model.layers.30.self_attn.v_proj.scales,
language_model.model.layers.30.self_attn.v_proj.weight,
language_model.model.layers.31.self_attn.k_norm.weight,
language_model.model.layers.31.self_attn.k_proj.scales,
language_model.model.layers.31.self_attn.k_proj.weight,
language_model.model.layers.31.self_attn.v_proj.scales,
language_model.model.layers.31.self_attn.v_proj.weight,
language_model.model.layers.32.self_attn.k_norm.weight,
language_model.model.layers.32.self_attn.k_proj.scales,
language_model.model.layers.32.self_attn.k_proj.weight,
language_model.model.layers.32.self_attn.v_proj.scales,
language_model.model.layers.32.self_attn.v_proj.weight,
language_model.model.layers.33.self_attn.k_norm.weight,
language_model.model.layers.33.self_attn.k_proj.scales,
language_model.model.layers.33.self_attn.k_proj.weight,
language_model.model.layers.33.self_attn.v_proj.scales,
language_model.model.layers.33.self_attn.v_proj.weight,
language_model.model.layers.34.self_attn.k_norm.weight,
language_model.model.layers.34.self_attn.k_proj.scales,
language_model.model.layers.34.self_attn.k_proj.weight,
language_model.model.layers.34.self_attn.v_proj.scales,
language_model.model.layers.34.self_attn.v_proj.weight,
language_model.model.layers.35.self_attn.k_norm.weight,
language_model.model.layers.35.self_attn.k_proj.scales,
language_model.model.layers.35.self_attn.k_proj.weight,
language_model.model.layers.35.self_attn.v_proj.scales,
language_model.model.layers.35.self_attn.v_proj.weight,
language_model.model.layers.36.self_attn.k_norm.weight,
language_model.model.layers.36.self_attn.k_proj.scales,
language_model.model.layers.36.self_attn.k_proj.weight,
language_model.model.layers.36.self_attn.v_proj.scales,
language_model.model.layers.36.self_attn.v_proj.weight,
language_model.model.layers.37.self_attn.k_norm.weight,
language_model.model.layers.37.self_attn.k_proj.scales,
language_model.model.layers.37.self_attn.k_proj.weight,
language_model.model.layers.37.self_attn.v_proj.scales,
language_model.model.layers.37.self_attn.v_proj.weight,
language_model.model.layers.38.self_attn.k_norm.weight,
language_model.model.layers.38.self_attn.k_proj.scales,
language_model.model.layers.38.self_attn.k_proj.weight,
language_model.model.layers.38.self_attn.v_proj.scales,
language_model.model.layers.38.self_attn.v_proj.weight,
language_model.model.layers.39.self_attn.k_norm.weight,
language_model.model.layers.39.self_attn.k_proj.scales,
language_model.model.layers.39.self_attn.k_proj.weight,
language_model.model.layers.39.self_attn.v_proj.scales,
language_model.model.layers.39.self_attn.v_proj.weight,
language_model.model.layers.40.self_attn.k_norm.weight,
language_model.model.layers.40.self_attn.k_proj.scales,
language_model.model.layers.40.self_attn.k_proj.weight,
language_model.model.layers.40.self_attn.v_proj.scales,
language_model.model.layers.40.self_attn.v_proj.weight,
language_model.model.layers.41.self_attn.k_norm.weight,
language_model.model.layers.41.self_attn.k_proj.scales,
language_model.model.layers.41.self_attn.k_proj.weight,
language_model.model.layers.41.self_attn.v_proj.scales,
language_model.model.layers.41.self_attn.v_proj.weight.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment