---
license: apache-2.0
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- multimodal
- vision-language-model
- mage-vl
- video-understanding
- streaming
---
Mage-VL
An Efficient Codec-Native Streaming Multimodal Foundation Model
---
**Mage-VL** is a **codec-native, proactive-streaming multimodal foundation model** for image and video understanding, whose visual encoder is trained **entirely from scratch** at a compact **4B** scale. It targets a modern *Moravec's paradox* of VLMs — strong at complex offline reasoning, yet slow and compute-heavy on simple real-time streaming perception. Instead of decoding video into uniformly-sampled frames and pushing a dense grid of patch tokens through a frozen web-pretrained ViT, Mage-VL follows the structure of modern video codecs: it separates a stream into **anchor (I) frames** and **predicted (P) frames**, keeps every anchor patch, and retains only the predicted-frame patches where the codec spends bits — the regions carrying real motion and new detail. This codec-aligned sparsity cuts visual tokens by **over 75%** while preserving spatio-temporal context, yielding **up to 3.5× wall-clock inference speedup** over uniform frame sampling.
The system pairs **two components**:
- **Mage-ViT** — a from-scratch *Codec-ViT* visual encoder that allocates tokens by codec-derived spatio-temporal importance, on a shared `16×16` patch grid with 3D rotary position encoding. It is **codec-agnostic**: the same interface accepts a traditional codec (H.264/AVC, HEVC/H.265) via motion vectors + residual energy, or a neural codec (DCVC-RT) via its learned rate map — no architecture or retraining change.
- **Qwen3-4B causal decoder** — a Qwen3-4B-Instruct-2507 language backbone (the only pretrained component) that consumes Mage-ViT's variable-length token stream through a lightweight two-layer MLP projector, with a unified interface for images, short/long/ultra-long video, and streaming.
On top of this pair, a **System 1 & System 2 dual-process design** adds proactive streaming inside a single model: a lightweight **cognition gate** (System 1) watches each rolling codec window and stays silent on routine content, invoking the full VLM (System 2) only when a response-worthy event completes — no multi-agent pipeline required.
## ✨ Highlights
- **Codec-native & from scratch.** The entire visual stack is trained from scratch — no billion-scale image-text ViT initialization. The bio-inspired predictive-patch mechanism (I/P frames at `16×16`) cuts visual-token consumption by **over 75%** (**~1/8 or less** of dense frame sampling), letting the model train on videos **8× longer** under the same budget.
- **Codec-native speedup.** Codec tokenization sets a superior accuracy–efficiency frontier — **up to 3.5× wall-clock inference speedup** over uniform frame sampling at matched accuracy, and the fastest of all compared models on most video benchmarks (single 8×B200 node).
- **Data-efficient tokenizer.** Trained on only **~100M unlabeled images/videos**, Mage-ViT matches or beats frontier encoders trained on billions of image-text pairs (SigLIP2 @ 10B, MoonViT @ 2B) — e.g. **99.33% on CIFAR-10** and **85.69% on ImageNet** with 256 tokens, showing web-scale pretraining is *not* essential for a strong VLM front-end.
- **Native-resolution scaling.** Variable-resolution pretraining lets Mage-ViT improve *monotonically* with the token budget (peaking **>96.1% Food-101 / >86.3% ImageNet** at 676 tokens) where fixed-resolution encoders saturate or degrade.
- **Matched-LLM video gains.** With the 4B Qwen3 backbone held fixed and only the ViT swapped, Mage-VL improves over Qwen3-VL-4B on **every** reported video and temporal-grounding benchmark — largest on localization-heavy tasks (**+22.5 QVHighlight**, +17.1 ActivityNet, +11.0 VSI-Bench, +24.5 VideoEval-Pro).
- **Strong for its size.** On par with Qwen3-VL-4B on static images, and clearly ahead on video understanding and spatial intelligence (**+11.0** VSI-Bench, **+53.1** CrossPoint, **+5.2** EmbSpatial, **+22.5** QVHighlight).
- **Proactive streaming, single model.** A frozen-backbone cognition gate delivers low-latency, event-gated commentary; it tops **TimVal / F1 / ROC-AUC / PR-AUC** on SoccerNet streaming and generalizes to real 2026 World Cup broadcasts.
## 📥 Model
A **single checkpoint**, `microsoft/Mage-VL`, is one unified model that **simultaneously** provides image & video understanding **and** the proactive streaming gate — the same weights answer offline image/video questions and drive event-gated commentary. It covers every Mage-VL capability: image understanding, frame-sampled video, traditional H.264/HEVC codec video, neural DCVC-RT codec video, and event-gated streaming. The repository bundles the codec processor, the neural codec package, and the proactive gate weights — no separate understanding, NVC, or streaming checkpoint is required.
We additionally release **`microsoft/Mage-ViT`** — the standalone visual encoder from the two-stage, from-scratch ViT pre-training (cluster-discrimination on ~100M unlabeled image/video frames). This is the **ViT-pre-trained checkpoint only**: it has **not** gone through the joint VLM training with the language model. Use it as a data-efficient, codec-native visual encoder or as a drop-in ViT for your own multimodal training.
| Model | Task | Backbone | Hugging Face |
| :--- | :--- | :--- | :--- |
| `Mage-VL` | image & video understanding **+** proactive streaming gate | Mage-ViT + Qwen3-4B-Instruct-2507 | [🤗 microsoft/Mage-VL](https://huggingface.co/microsoft/Mage-VL) |
| `Mage-ViT` | codec-native visual encoder — ViT pre-training only, no VLM joint training | Codec-ViT (from scratch) | [🤗 microsoft/Mage-ViT](https://huggingface.co/microsoft/Mage-ViT) |
## 🏗️ Architecture
Proactive streaming framework — Mage-ViT incrementally encodes the continuous stream into codec-native visual features shared by the event gate and the causal decoder. The gate scores each rolling window and stays silent on routine content; when it opens, the decoder emits an event-conditioned response.
**Mage-ViT** — a from-scratch Codec-ViT visual encoder. On a `16×16` patch grid it keeps all anchor (I) frame patches and only the motion-salient predicted (P) frame patches, cutting visual tokens by over 75% while a shared 3D RoPE preserves spatio-temporal positions.
**Mage-VL** — a unified model where projected visual tokens and text tokens share one causal Qwen3 decoder. Still images become a single spatial block; videos become temporally-ordered codec windows. In **streaming** mode, a lightweight **cognition gate** predicts `p_speak = g(h_t)` per rolling window (over a recurrent streaming memory kept by an event-preserving feature extractor) and triggers generation when `p_speak ≥ τ`; the response is decoded by the frozen base model from a local sliding window of the most recent codec segments, and a text query can be injected at any time.
**Training** — a progressive **five-stage** supervised curriculum (no preference/RL post-training) that produces one unified model:
1. **Multimodal alignment via captions** — ~350M dense image captions + 4.2M short-video captions.
2. **Instruction tuning + short temporal grounding** — ~54M image-instruction samples + 3.4M 30–180s video captions.
3. **Temporal-horizon expansion** — medium/long video (LLaVA-Video, TimeLens, VideoChat-Flash, Molmo2) with retained image SFT.
4. **Codec-native long-context adaptation** — 350K long videos as rolling codec windows (up to 384/768 frames).
5. **Proactive streaming alignment** — a cognition gate fine-tuned on ~3.3M streaming samples with the visual encoder and LLM kept frozen (only the gate is trained).
The five stages together produce a **single unified model**, `Mage-VL`, that handles image understanding, offline video reasoning, and proactive streaming — no separate variants are shipped.
Two parts of the pipeline apply an **AI4AI** (AI-for-AI) paradigm: (1) dense recaptioning runs through an agentic closed loop where a GPT-5 rubric scorer grades captions and a Copilot coding agent co-designs the prompt *and* harness code (e.g. rendering timestamp overlays) under a human validation gate — improving every downstream OCR/doc/chart/perception benchmark and inspiring SkillOpt-Lite; and (2) Stage-3 uses AI-based diagnostics to decide which video categories, resolutions, and frame counts to train on.
## 📊 Performance
Image understanding & spatial intelligence — click to expand
Performance comparison across models. Mage-VL-4B and Qwen3-VL-4B use the same 4B Qwen3 LLM backbone; Phi-4-Multimodal-Instruct (Phi-4-MM, 5.6B) and Phi-4-Reasoning-Vision (Phi-4-R-V, 15B) are reported for reference. `–` = not run. **Bold** = best in row.
| Benchmark | Mage-VL-4B | Qwen3-VL-4B | Phi-4-MM-5.6B | Phi-4-R-V-15B |
| :--- | :---: | :---: | :---: | :---: |
| *Document understanding* | | | | |
| DocVQA-val | **95.14** | 94.69 | 92.79 | 76.20 |
| InfoVQA-val | **80.33** | 79.50 | 71.84 | 55.41 |
| AI2D w/ Mask | **83.16** | 81.54 | 81.83 | 82.87 |
| ChartQA | **84.88** | 83.96 | 83.76 | 83.40 |
| OCRBench | **81.80** | 81.60 | 81.70 | 73.90 |
| MultiDocVQA-val | **87.46** | 87.21 | 46.84 | 58.35 |
| ChartQAPro | **32.57** | 26.79 | 0.13 | 25.38 |
| TextVQA-val | 77.28 | **80.55** | 39.93 | 76.06 |
| CC-OCR Doc | 32.25 | **39.69** | 4.99 | 17.65 |
| *General VQA* | | | | |
| MMBench-EN-dev | 84.02 | 83.25 | 65.81 | **84.19** |
| MMBench-CN-dev | **82.04** | 80.58 | 75.17 | 79.47 |
| MMStar | **67.32** | 62.04 | 61.24 | 59.63 |
| MME-Perception | **1709.54** | 1703.50 | 1409.66 | 1590.21 |
| SeedBench (All) | **76.78** | 75.65 | 68.28 | 73.70 |
| CV-Bench | **87.79** | 85.37 | 57.09 | 81.31 |
| MME-RealWorld | **66.52** | 63.20 | 32.45 | 57.80 |
| *Spatial intelligence* | | | | |
| CV-Bench-2D | **82.13** | 81.00 | 56.12 | 80.11 |
| CV-Bench-3D | **94.75** | 92.30 | 56.92 | 82.50 |
| BLINK | **65.11** | 65.10 | 35.24 | 57.80 |
| EmbSpatial | **82.67** | 77.50 | 41.51 | 72.67 |
| CrossPoint | **80.00** | 26.90 | 12.20 | 47.73 |
| CRPE-Relation | 76.12 | **77.70** | 34.60 | 74.46 |
| SAT | 67.33 | **69.30** | 55.33 | 66.67 |
Video understanding & temporal grounding — click to expand
**Bold** = best in row.
| Benchmark | Mage-VL-4B | Qwen3-VL-4B | Phi-4-MM-5.6B | Phi-4-R-V-15B |
| :--- | :---: | :---: | :---: | :---: |
| *Video QA* | | | | |
| MV-Bench | 65.1 | **66.7** | 44.9 | 49.2 |
| NextQA | **83.1** | 79.8 | 54.1 | 69.0 |
| VideoMME | **64.0** | 59.7 | 44.7 | 55.3 |
| LongVideoBench | **61.3** | 57.7 | 41.14 | 51.2 |
| LVBench | **41.8** | 39.2 | 25.31 | 34.4 |
| MLVU-dev | **68.7** | 61.5 | 44.18 | 51.8 |
| VideoEval-Pro | **45.2** | 20.7 | 14.35 | 16.8 |
| *Temporal grounding* | | | | |
| Timelens-Charades | **50.7** | 43.1 | 4.09 | 20.6 |
| Timelens-ActivityNet | **45.4** | 28.4 | 2.03 | 23.0 |
| Timelens-QVHighlight | **57.4** | 34.9 | 2.47 | 11.6 |
| *Spatial reasoning* | | | | |
| VSI-Bench | **64.3** | 53.3 | 24.09 | 25.5 |
| *Tracking (J&F)* | | | | |
| Ref-DAVIS17 | **25.83** | 7.48 | 3.14 | 2.15 |
| MeViS-ValidU | **22.55** | 3.16 | 10.28 | 1.53 |
| ReasonVOS | **17.76** | 9.66 | 9.50 | 9.77 |
| Ref-YT-VOS | **25.57** | 5.28 | 8.64 | 3.85 |
Proactive streaming (SoccerNet) & online video (OVO-Bench) — click to expand
**SoccerNet — response timing** (StreamMind protocol, codec-native inputs, zero-tolerance canvas matching). **Bold** = best in column.
| Method | TriggerAcc | TimVal | F1 | ROC-AUC | PR-AUC |
| :--- | :---: | :---: | :---: | :---: | :---: |
| StreamMind | 52.18 | 47.36 | – | – | – |
| JoyAI-VL-Interaction-9B | **97.98** | 19.25 | 3.55 | 56.26 | 1.68 |
| **Mage-VL-4B** | 79.21 | **55.54** | **16.35** | **83.14** | **9.30** |
JoyAI's high TriggerAcc comes from predicting silence almost everywhere under SoccerNet's heavy class imbalance, so it collapses on the precision-sensitive metrics; StreamMind is trained *in-distribution* on SoccerNet, whereas Mage-VL is not.
**OVO-Bench** — online video understanding (SimpleStream recent-window protocol, 4 frames @ 1 fps; no streaming-specific fine-tuning). Mage-VL sets a new state-of-the-art overall score **among streaming architectures**. RT-Avg / BT-Avg are the Real-Time Visual Perception / Backward Tracing sub-task averages; Overall is their mean. **Bold** = best model per column (Human is the reference upper bound).
| Model | #Frames | RT-Avg | BT-Avg | Overall |
| :--- | :---: | :---: | :---: | :---: |
| Human | – | 93.2 | 92.3 | 92.77 |
| *Offline video LLMs* | | | | |
| Qwen2.5-VL-7B | 1 fps | 59.9 | 44.7 | 52.28 |
| LLaVA-Video-7B | 64 | 63.5 | 40.4 | 51.95 |
| Qwen3-VL-4B | 64 | 72.8 | **53.1** | 63.00 |
| *Online / streaming video LLMs* | | | | |
| VideoLLM-online-8B | 2 fps | 20.8 | 17.7 | 19.26 |
| Flash-VStream-7B | 1 fps | 28.4 | 27.4 | 27.90 |
| Dispider-7B | 1 fps | 54.6 | 36.1 | 45.35 |
| TimeChat-Online-7B | 1 fps | 61.9 | 41.7 | 51.80 |
| StreamForest-7B | 1 fps | 61.2 | 52.0 | 56.60 |
| Streamo-7B | 1 fps | 66.0 | 46.1 | 56.05 |
| HERMES-7B† | 1 fps | 69.0 | 49.4 | 59.20 |
| JoyAI-VL-Interaction-9B | 1 fps | 68.4 | 48.6 | 58.50 |
| **Mage-VL-4B** | 1 fps | **79.84** | 48.15 | **64.00** |
† HERMES = Qwen2.5-VL-7B + HERMES (4K tokens). Baseline results and table structure follow SimpleStream.
## 🔬 Key Findings
Beyond the model, the report distills **seven empirical findings** for efficient multimodal training:
1. **Web-scale pretraining is not essential.** A from-scratch backbone on ~100M unlabeled frames matches encoders trained on billions of image-text pairs.
2. **Variable-resolution pretraining scales monotonically.** Quality keeps improving with the visual-token budget instead of saturating/degrading like fixed-resolution encoders.
3. **Codec-native tokenization sets a better accuracy–efficiency frontier** — up to **3.5× wall-clock inference speedup** over uniform frame sampling.
4. **Explicit VideoQA SFT is redundant.** Dense video *captions* + standard image SFT are sufficient for strong zero-shot VideoQA.
5. **Motion–spatial synergy.** Dynamic video training substantially improves static 2D/3D spatial reasoning.
6. **AI4AI data pipeline.** Agentic closed-loop feedback + prompt/code co-design systematically lift caption quality and downstream scores (inspired SkillOpt-Lite).
7. **Zero-Vision SFT for multimodal RL.** Bypassing visual SFT in favor of pure-text reasoning SFT unlocks stronger multimodal RL — a compute-efficient path.
## 🚀 Quick Start
A single checkpoint, `microsoft/Mage-VL`, covers every capability below.
| Capability | Script | How to run |
|---|---|---|
| Image understanding | `inference.py` | `--mode offline --image` |
| Frame-sampled video | `inference.py` | `--mode offline --video --video-backend frames` |
| Traditional H.264/HEVC codec video | `inference.py` | `--mode offline --video --video-backend codec --codec-engine traditional` |
| Neural DCVC-RT codec video | `inference.py` | `--mode offline --video --video-backend codec --codec-engine neural` |
| Online image / video (SGLang) | `inference.py` | `--mode online … --base-url ` |
| Event-gated streaming commentary | `inference_streaming.py` | in the [GitHub repo](https://github.com/microsoft/Mage/tree/main/mage_vl) |
### Installation
For offline Transformers inference:
```bash
pip install "transformers>=5.7" accelerate pillow torch torchvision \
opencv-python codec-video-prep
```
Codec-based video inference also requires `ffmpeg` and `ffprobe` on `PATH`.
### Examples
Two sample inputs ship with this repository:
| Input | Question | Content |
|---|---|---|
| [`examples/dog.jpg`](examples/dog.jpg) | Describe this image in detail. | Photo of a dog sitting in front of a patterned rug |
| [`examples/soccer-broadcast.mp4`](examples/soccer-broadcast.mp4) | Describe this video. | 30s, 960×540 football broadcast clip |
### Offline inference
Download [`inference.py`](inference.py). Offline mode loads the checkpoint with `AutoModelForCausalLM.from_pretrained` and supports images, frame sampling, and both codec engines:
```bash
# image
python inference.py --mode offline --image examples/dog.jpg \
--question "Describe this image in detail."
```
> The image depicts a dog sitting on a patterned rug. The dog appears to be a
> medium-sized breed with a thick, fluffy coat. Its fur is primarily white with
> patches of black and brown. The dog's ears are perked up, and it has a calm and
> attentive expression. [...]
```bash
# video — uniform frame sampling
python inference.py --mode offline --video examples/soccer-broadcast.mp4 \
--video-backend frames --num-frames 32 \
--question "Describe this video."
```
> The video opens with a man in a black polo shirt, sporting a short haircut,
> standing in a stadium. He is holding a yellow microphone with the BBC Sport
> logo on it. The background reveals a large crowd of spectators. [...]
```bash
# video — traditional codec (HEVC/H.264)
python inference.py --mode offline --video examples/soccer-broadcast.mp4 \
--video-backend codec --codec-engine traditional --num-frames 32 \
--question "Describe this video."
```
> The video opens with a BBC Sport broadcast, featuring a presenter in a black
> shirt holding a yellow microphone. The background reveals a packed stadium,
> with the scoreboard displaying "ENG 1 ARG 2 FT", indicating the final score of
> the match. [...]
```bash
# video — neural codec (DCVC-RT)
python inference.py --mode offline --video examples/soccer-broadcast.mp4 \
--video-backend codec --codec-engine neural --num-frames 32 \
--question "Describe this video."
```
> The video opens with a BBC Sport broadcast, featuring a presenter standing in a
> stadium filled with spectators. The presenter, dressed in a black shirt, holds
> a yellow BBC Sport microphone and wears a black earpiece. [...]
### Online inference
Online mode talks to an OpenAI-compatible SGLang server. **First** build and launch the server with the Mage-VL SGLang branch (building it needs `protobuf-compiler` and a Rust toolchain):
```bash
sudo apt-get update && sudo apt-get install -y protobuf-compiler
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \
| sh -s -- -y --profile minimal --default-toolchain 1.90.0
source "$HOME/.cargo/env"
git clone -b feat/mage-vl https://github.com/kcz358/sglang
cd sglang
pip install -e 'python[all]'
python -m sglang.launch_server \
--model-path microsoft/Mage-VL \
--trust-remote-code
```
**Then** send an image or sampled video frames to the running server:
```bash
pip install openai
python inference.py --mode online --image examples/dog.jpg \
--question "Describe this image in detail." \
--base-url http://localhost:30000/v1
python inference.py --mode online --video examples/soccer-broadcast.mp4 \
--num-frames 32 \
--question "Describe this video." \
--base-url http://localhost:30000/v1
```
Use `--model`, `--max-new-tokens`, and `--api-key` to override their defaults.
### Streaming inference
`streammind_gate.safetensors` in this repository holds the event gate. Streaming inference splits a video into non-overlapping segments, stays silent on routine content, and generates a caption only when a response-worthy event is detected. Run it with `inference_streaming.py` from the [GitHub repository](https://github.com/microsoft/Mage/tree/main/mage_vl):
```bash
python inference_streaming.py \
--video examples/soccer-broadcast.mp4 \
--video_backend codec \
--segment_sec 8
```
```text
[t=0.0-8.0s] gate=silence (p=0.19)
[t=8.0-16.0s] gate=response (p=0.55) -> The video features a live sports broadcast from BBC Sport, set in a large stadium filled with spectators. The broadcast focuses on a football match between England and Argentina, with the score displayed as England 1, Argentina 2. [...]
[t=16.0-24.0s] gate=response (p=0.73) -> The video features a sports broadcast set in a large stadium filled with spectators. Four commentators are gathered around a table with a 'BBC Sport' logo, each holding a yellow microphone. [...]
[t=24.0-30.0s] gate=silence (p=0.31)
```
The gate is trained on codec inputs, so `--video_backend codec` is the intended setting. Use `--video_backend frames` for direct frame sampling. Additional controls include `--num_frames`, `--cur_fps`, `--max_segments`, `--max_new_tokens`, `--gate_threshold`, and `--attn_impl`.
## 📝 Citation
```bibtex
@article{yang2026mage,
title={Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model},
author={Yang, Senqiao and Zhang, Kaichen and Jia, Zhaoyang and Guo, Jinghao and Shen, Yifei and Zhang, Xinjie and Zhang, Xiaoyi and Wang, Haoqing and Li, Xiao and Zhang, Peng and others},
journal={arXiv preprint arXiv:2607.24904},
year={2026}
}
```
## 📄 License
Mage-VL is released under the [Apache-2.0 License](https://www.apache.org/licenses/LICENSE-2.0).
