microsoft/Mage-VL

📥 Model

A single checkpoint, microsoft/Mage-VL, is one unified model that simultaneously provides image & video understanding and the proactive streaming gate — the same weights answer offline image/video questions and drive event-gated commentary. It covers every Mage-VL capability: image understanding, frame-sampled video, traditional H.264/HEVC codec video, neural DCVC-RT codec video, and event-gated streaming. The repository bundles the codec processor, the neural codec package, and the proactive gate weights — no separate understanding, NVC, or streaming checkpoint is required.

We additionally release microsoft/Mage-ViT — the standalone visual encoder from the two-stage, from-scratch ViT pre-training (cluster-discrimination on ~100M unlabeled image/video frames). This is the ViT-pre-trained checkpoint only: it has not gone through the joint VLM training with the language model. Use it as a data-efficient, codec-native visual encoder or as a drop-in ViT for your own multimodal training.

🏗️ Architecture

Mage-ViT — a from-scratch Codec-ViT visual encoder. On a 16×16 patch grid it keeps all anchor (I) frame patches and only the motion-salient predicted (P) frame patches, cutting visual tokens by over 75% while a shared 3D RoPE preserves spatio-temporal positions.

Mage-VL — a unified model where projected visual tokens and text tokens share one causal Qwen3 decoder. Still images become a single spatial block; videos become temporally-ordered codec windows. In streaming mode, a lightweight cognition gate predicts p_speak = g(h_t) per rolling window (over a recurrent streaming memory kept by an event-preserving feature extractor) and triggers generation when p_speak ≥ τ; the response is decoded by the frozen base model from a local sliding window of the most recent codec segments, and a text query can be injected at any time.

Training — a progressive five-stage supervised curriculum (no preference/RL post-training) that produces one unified model:

1. Multimodal alignment via captions — ~350M dense image captions + 4.2M short-video captions.
2. Instruction tuning + short temporal grounding — ~54M image-instruction samples + 3.4M 30–180s video captions.
3. Temporal-horizon expansion — medium/long video (LLaVA-Video, TimeLens, VideoChat-Flash, Molmo2) with retained image SFT.
4. Codec-native long-context adaptation — 350K long videos as rolling codec windows (up to 384/768 frames).
5. Proactive streaming alignment — a cognition gate fine-tuned on ~3.3M streaming samples with the visual encoder and LLM kept frozen (only the gate is trained).

The five stages together produce a single unified model, Mage-VL, that handles image understanding, offline video reasoning, and proactive streaming — no separate variants are shipped.

Two parts of the pipeline apply an AI4AI (AI-for-AI) paradigm: (1) dense recaptioning runs through an agentic closed loop where a GPT-5 rubric scorer grades captions and a Copilot coding agent co-designs the prompt and harness code (e.g. rendering timestamp overlays) under a human validation gate — improving every downstream OCR/doc/chart/perception benchmark and inspiring SkillOpt-Lite; and (2) Stage-3 uses AI-based diagnostics to decide which video categories, resolutions, and frame counts to train on.

📊 Performance

Performance comparison across models. Mage-VL-4B and Qwen3-VL-4B use the same 4B Qwen3 LLM backbone; Phi-4-Multimodal-Instruct (Phi-4-MM, 5.6B) and Phi-4-Reasoning-Vision (Phi-4-R-V, 15B) are reported for reference. – = not run. Bold = best in row.

Bold = best in row.

SoccerNet — response timing (StreamMind protocol, codec-native inputs, zero-tolerance canvas matching). Bold = best in column.

JoyAI's high TriggerAcc comes from predicting silence almost everywhere under SoccerNet's heavy class imbalance, so it collapses on the precision-sensitive metrics; StreamMind is trained in-distribution on SoccerNet, whereas Mage-VL is not.

OVO-Bench — online video understanding (SimpleStream recent-window protocol, 4 frames @ 1 fps; no streaming-specific fine-tuning). Mage-VL sets a new state-of-the-art overall score among streaming architectures. RT-Avg / BT-Avg are the Real-Time Visual Perception / Backward Tracing sub-task averages; Overall is their mean. Bold = best model per column (Human is the reference upper bound).

† HERMES = Qwen2.5-VL-7B + HERMES (4K tokens). Baseline results and table structure follow SimpleStream.

🔬 Key Findings

Beyond the model, the report distills seven empirical findings for efficient multimodal training:

1. Web-scale pretraining is not essential. A from-scratch backbone on ~100M unlabeled frames matches encoders trained on billions of image-text pairs.
2. Variable-resolution pretraining scales monotonically. Quality keeps improving with the visual-token budget instead of saturating/degrading like fixed-resolution encoders.
3. Codec-native tokenization sets a better accuracy–efficiency frontier — up to 3.5× wall-clock inference speedup over uniform frame sampling.
4. Explicit VideoQA SFT is redundant. Dense video captions + standard image SFT are sufficient for strong zero-shot VideoQA.
5. Motion–spatial synergy. Dynamic video training substantially improves static 2D/3D spatial reasoning.
6. AI4AI data pipeline. Agentic closed-loop feedback + prompt/code co-design systematically lift caption quality and downstream scores (inspired SkillOpt-Lite).
7. Zero-Vision SFT for multimodal RL. Bypassing visual SFT in favor of pure-text reasoning SFT unlocks stronger multimodal RL — a compute-efficient path.

🚀 Quick Start

A single checkpoint, microsoft/Mage-VL, covers every capability below.

Installation

For offline Transformers inference:

pip install "transformers>=5.7" accelerate pillow torch torchvision \
  opencv-python codec-video-prep

Codec-based video inference also requires ffmpeg and ffprobe on PATH.

Examples

Two sample inputs ship with this repository:

Offline inference

Download inference.py. Offline mode loads the checkpoint with AutoModelForCausalLM.from_pretrained and supports images, frame sampling, and both codec engines:

# image
python inference.py --mode offline --image examples/dog.jpg \
  --question "Describe this image in detail."

The image depicts a dog sitting on a patterned rug. The dog appears to be a medium-sized breed with a thick, fluffy coat. Its fur is primarily white with patches of black and brown. The dog's ears are perked up, and it has a calm and attentive expression. [...]

# video — uniform frame sampling
python inference.py --mode offline --video examples/soccer-broadcast.mp4 \
  --video-backend frames --num-frames 32 \
  --question "Describe this video."

The video opens with a man in a black polo shirt, sporting a short haircut, standing in a stadium. He is holding a yellow microphone with the BBC Sport logo on it. The background reveals a large crowd of spectators. [...]

# video — traditional codec (HEVC/H.264)
python inference.py --mode offline --video examples/soccer-broadcast.mp4 \
  --video-backend codec --codec-engine traditional --num-frames 32 \
  --question "Describe this video."

The video opens with a BBC Sport broadcast, featuring a presenter in a black shirt holding a yellow microphone. The background reveals a packed stadium, with the scoreboard displaying "ENG 1 ARG 2 FT", indicating the final score of the match. [...]

# video — neural codec (DCVC-RT)
python inference.py --mode offline --video examples/soccer-broadcast.mp4 \
  --video-backend codec --codec-engine neural --num-frames 32 \
  --question "Describe this video."

The video opens with a BBC Sport broadcast, featuring a presenter standing in a stadium filled with spectators. The presenter, dressed in a black shirt, holds a yellow BBC Sport microphone and wears a black earpiece. [...]

Online inference

Online mode talks to an OpenAI-compatible SGLang server. First build and launch the server with the Mage-VL SGLang branch (building it needs protobuf-compiler and a Rust toolchain):

sudo apt-get update && sudo apt-get install -y protobuf-compiler
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \
  | sh -s -- -y --profile minimal --default-toolchain 1.90.0
source "$HOME/.cargo/env"

git clone -b feat/mage-vl https://github.com/kcz358/sglang
cd sglang
pip install -e 'python[all]'
python -m sglang.launch_server \
  --model-path microsoft/Mage-VL \
  --trust-remote-code

Then send an image or sampled video frames to the running server:

pip install openai

python inference.py --mode online --image examples/dog.jpg \
  --question "Describe this image in detail." \
  --base-url http://localhost:30000/v1

python inference.py --mode online --video examples/soccer-broadcast.mp4 \
  --num-frames 32 \
  --question "Describe this video." \
  --base-url http://localhost:30000/v1

Use --model, --max-new-tokens, and --api-key to override their defaults.

Streaming inference

streammind_gate.safetensors in this repository holds the event gate. Streaming inference splits a video into non-overlapping segments, stays silent on routine content, and generates a caption only when a response-worthy event is detected. Run it with inference_streaming.py from the GitHub repository:

python inference_streaming.py \
  --video examples/soccer-broadcast.mp4 \
  --video_backend codec \
  --segment_sec 8

[t=0.0-8.0s] gate=silence (p=0.19)
[t=8.0-16.0s] gate=response (p=0.55) -> The video features a live sports broadcast from BBC Sport, set in a large stadium filled with spectators. The broadcast focuses on a football match between England and Argentina, with the score displayed as England 1, Argentina 2. [...]
[t=16.0-24.0s] gate=response (p=0.73) -> The video features a sports broadcast set in a large stadium filled with spectators. Four commentators are gathered around a table with a 'BBC Sport' logo, each holding a yellow microphone. [...]
[t=24.0-30.0s] gate=silence (p=0.31)

The gate is trained on codec inputs, so --video_backend codec is the intended setting. Use --video_backend frames for direct frame sampling. Additional controls include --num_frames, --cur_fps, --max_segments, --max_new_tokens, --gate_threshold, and --attn_impl.

📝 Citation

@article{yang2026mage,
  title={Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model},
  author={Yang, Senqiao and Zhang, Kaichen and Jia, Zhaoyang and Guo, Jinghao and Shen, Yifei and Zhang, Xinjie and Zhang, Xiaoyi and Wang, Haoqing and Li, Xiao and Zhang, Peng and others},
  journal={arXiv preprint arXiv:2607.24904},
  year={2026}
}

📄 License

Mage-VL is released under the Apache-2.0 License.