Uni-MOPD

Uni-MOPD

A single multimodal model that unifies five domain specialists: a uniform-merge strong start refined by short multi-teacher on-policy distillation.

Uni-MOPD

Strong Starts, Simple Recipes: Unified Multi-Teacher On-Policy Distillation from Multimodal Specialists

Multi-teacher on-policy distillation (MOPD) integrates separately trained domain specialists into a unified model: the student generates responses to training queries, and the teacher of each query's domain supervises them. Existing methods refine the distillation procedure while the student starts from a base model or a multi-domain supervised fine-tuning checkpoint, which leaves capability integration largely to distillation itself.

Unified MOPD (Uni-MOPD) treats the starting point as a design variable and pairs a strong start with a simple recipe: a training-free uniform merge of specialists from the same base model initializes the student, and short MOPD with uniform domain sampling and full-vocabulary reverse KL then refines it.

The bank covers five domains: STEM reasoning, general visual question answering, document/OCR/chart understanding, GUI grounding, and embodied understanding. All five specialists are post-trained from the same base checkpoint, Qwen/Qwen3-VL-30B-A3B-Instruct, an instruction-tuned mixture-of-experts vision-language model, and follow different post-training paths, each ending with reinforcement learning on its own domain. Because they share the base model, the specialists share one architecture and vocabulary, which permits both weight-space merging and token-level distillation.

Repository What it is
Uni-MOPD The unified student
Uni-MOPD-Merge The uniform merge that initializes the student
Uni-MOPD-Teacher-STEM STEM reasoning specialist
Uni-MOPD-Teacher-VQA General visual question answering specialist
Uni-MOPD-Teacher-OCR Document/OCR/chart understanding specialist (RL from a multi-domain SFT start)
Uni-MOPD-Teacher-OCR-Bank2 Document/OCR/chart understanding specialist (OCR-domain SFT and RL)
Uni-MOPD-Teacher-GUI GUI grounding specialist
Uni-MOPD-Teacher-Embodied Embodied understanding specialist

Uni-MOPD-Teacher-OCR-Bank2 comes from the paper's second specialist bank. The two OCR specialists follow different post-training paths: the first starts its domain reinforcement learning from a multi-domain SFT checkpoint, which gives it broader coverage, while the second is trained with supervised fine-tuning and reinforcement learning on the document/OCR domain only.

All of them sit under the AmberHeart namespace; Uni-MOPD is this repository.

This model

The unified student. Five specialists post-trained from a shared base are combined through a training-free uniform merge to initialize the student, which is then refined with short MOPD โ€” 50 updates using uniform domain sampling and full-vocabulary reverse KL.

GUI grounding

For GUI grounding, use this prompt:

You are a GUI grounding agent. Given a screenshot and an element description, locate the element and output a click action.

Briefly reason about the target element location, then output the click action as JSON.

The expected JSON output format is as follows:

{"action": {"name": "click", "button": "left", "click_type": "single", "x": 250, "y": 845}}

Here, "name" is the action to perform, "button" is the mouse button, "click_type" is how the click is performed, and "x" and "y" represent the target location on screen.

The coordinate system is normalized: output x and y as integers from 0 to 1000, where the entire screenshot maps to this range.

The model answers with brief reasoning followed by the click action as JSON, with x and y as integers in [0, 1000] normalized to the screenshot.

For instruction-driven grounding (clicking the target of a natural-language command rather than a described element), only the first paragraph changes:

You are a GUI interaction agent. Given a screenshot and a natural language instruction, identify the target element and output a click action.

The rest of the prompt and the output format are identical.

Weights are bfloat16, in 13 safetensors shards.

Usage

from transformers import AutoProcessor, Qwen3VLMoeForConditionalGeneration

model_id = "AmberHeart/Uni-MOPD"
model = Qwen3VLMoeForConditionalGeneration.from_pretrained(
    model_id, dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained(model_id)

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "url": "<your-image>"},
        {"type": "text", "text": "Describe this image."},
    ],
}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)
out = model.generate(**inputs, max_new_tokens=128)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:],
                       skip_special_tokens=True))

Requires transformers>=4.57.1.

License

Apache License 2.0, inherited from the base model. See LICENSE for the license text and NOTICE for the statement of modifications.

Downloads last month
74
Safetensors
Model size
31B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for AmberHeart/Uni-MOPD

Finetuned
(49)
this model

Collection including AmberHeart/Uni-MOPD