Instructions to use mlx-community/Qwen3.5-397B-A17B-8bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use mlx-community/Qwen3.5-397B-A17B-8bit with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("mlx-community/Qwen3.5-397B-A17B-8bit") config = load_config("mlx-community/Qwen3.5-397B-A17B-8bit") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps
- LM Studio
- Pi
How to use mlx-community/Qwen3.5-397B-A17B-8bit with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "mlx-community/Qwen3.5-397B-A17B-8bit"
Configure the model in Pi
# Install Pi: npm install -g @mariozechner/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "mlx-community/Qwen3.5-397B-A17B-8bit" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent new
How to use mlx-community/Qwen3.5-397B-A17B-8bit with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "mlx-community/Qwen3.5-397B-A17B-8bit"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default mlx-community/Qwen3.5-397B-A17B-8bit
Run Hermes
hermes
Was this created with the fixes in mlx-lm version 0.30.8?
This model mlx-community/Qwen3.5-397B-A17B-8bit was converted to MLX format from Qwen/Qwen3.5-397B-A17B using mlx-lm version 0.30.8.
As the pull request to increment mlx-lm's version number to 0.30.8 was created today at 9:23 AM EST, and the upload of this repository started yesterday night around 10 PM, it must have been made using a version of mlx-lm from someone's private branch. Does it have the important fix for Qwen 3.5 in pull #902?
Yes, this quant was created after Awni had merged pull request 902.
Releasing a version with fixed scope, and bumping the version number for the next version are separate actions. If the version number is not "bumped" when a release happens, it happens that the version numbers of mlx-lm are not always up-to-date.
They had released 0.30.7 already, but did not yet bump up the version number to 0.30.8 for any new code. Knowing that they will have to bump the version number to 0.30.8, I already wrote 0.30.8 in the model card as this is what people should use. Version 0.30.7 that was already released did not have PR #902 included, so I would be giving wrong information if I wrote 0.30.7 in the model card.