Instructions to use OzzyGT/Flux_3_Image_Klein_pup_sdnq_dynamic_8bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use OzzyGT/Flux_3_Image_Klein_pup_sdnq_dynamic_8bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="OzzyGT/Flux_3_Image_Klein_pup_sdnq_dynamic_8bit")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("OzzyGT/Flux_3_Image_Klein_pup_sdnq_dynamic_8bit")
model = AutoModelForMultimodalLM.from_pretrained("OzzyGT/Flux_3_Image_Klein_pup_sdnq_dynamic_8bit", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use OzzyGT/Flux_3_Image_Klein_pup_sdnq_dynamic_8bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "OzzyGT/Flux_3_Image_Klein_pup_sdnq_dynamic_8bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OzzyGT/Flux_3_Image_Klein_pup_sdnq_dynamic_8bit",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/OzzyGT/Flux_3_Image_Klein_pup_sdnq_dynamic_8bit

SGLang

How to use OzzyGT/Flux_3_Image_Klein_pup_sdnq_dynamic_8bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "OzzyGT/Flux_3_Image_Klein_pup_sdnq_dynamic_8bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OzzyGT/Flux_3_Image_Klein_pup_sdnq_dynamic_8bit",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "OzzyGT/Flux_3_Image_Klein_pup_sdnq_dynamic_8bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OzzyGT/Flux_3_Image_Klein_pup_sdnq_dynamic_8bit",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use OzzyGT/Flux_3_Image_Klein_pup_sdnq_dynamic_8bit with Docker Model Runner:
```
docker model run hf.co/OzzyGT/Flux_3_Image_Klein_pup_sdnq_dynamic_8bit
```

FLUX 3 Image Klein Prompt Upsampler SDNQ INT8

bf16	INT8

Left: original bf16 upsampler · Right: this INT8 upsampler (same request and seed, both generated with the bf16 FLUX 3 Image Klein).

This is a dynamic 8-bit quantized version of black-forest-labs/flux-3-image-klein-pup, the prompt upsampler of FLUX 3 Image Klein, using SDNQ. The language model, the vision tower and the token embeddings are quantized; lm_head stays in bf16.

Note: You need SDNQ v0.2.8 or superior and a diffusers version with the FLUX 3 prompt upsampler block.

Usage

You can find ready-to-use scripts in the diffusers-recipes repository. For 16 GB GPUs and up, use sdnq_prompt_upsampler_model_cpu_offload.py; for 12 GB GPUs, use sdnq_prompt_upsampler_two_stage.py, which writes the caption first and then generates the image.

The upsampler replaces the prompt_upsampler component of the FLUX 3 Image Klein pipeline. Import sdnq before loading so the quantized weights are registered. Decoding is slower than with the bf16 upsampler (about 22 vs 70 tokens/s on an RTX 5090).

For the system prompts, layout boxes and image editing, see the original model card.

Sample image

The INT8 image above was generated with the following request and settings (seed 7):

import sdnq  # noqa: F401
import torch
from diffusers import ModularPipeline
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration


upsampler_id = "OzzyGT/Flux_3_Image_Klein_pup_sdnq_dynamic_8bit"

pipe = ModularPipeline.from_pretrained("black-forest-labs/flux-3-image-klein")
pipe.update_components(
    prompt_upsampler=Qwen3_5ForConditionalGeneration.from_pretrained(upsampler_id, dtype=torch.bfloat16),
    prompt_upsampler_processor=AutoProcessor.from_pretrained(upsampler_id),
)
pipe.load_components(dtype=torch.bfloat16)
pipe.to("cuda")

prompt = "A small neighborhood pet shop: a tabby cat on the counter, a golden retriever in a wicker basket and a scarlet macaw perched above an aquarium. The sign over the door reads WHISKERS & WAGS."

output = pipe(
    prompt=prompt,
    height=1152,
    width=2048,
    generator=torch.Generator().manual_seed(7),
    output=["images", "prompt"],
)
print(output["prompt"])  # the caption written by the upsampler
output["images"][0].save("flux3_klein_pup_int8.png")

License

This model is released under the Apache License 2.0, the same as the original. It is derived from Qwen3.5-9B by the Qwen team at Alibaba Cloud.

Downloads last month: 69

Safetensors

Model size

9B params

Tensor type

BF16

Model tree for OzzyGT/Flux_3_Image_Klein_pup_sdnq_dynamic_8bit

Base model

Qwen/Qwen3.5-9B-Base

Finetuned

Qwen/Qwen3.5-9B

Finetuned

black-forest-labs/flux-3-image-klein-pup

Quantized

(18)

this model