OpenMOSS-Team/MOSS-VL-Instruct-0708

video-text-to-texttransformersUpstream checksumstransformerssafetensorsmoss_vlfeature-extractionSFTVideo-Understanding

License: apache-2.0
Size: not mirrored yet
Downloads in the last 30 days: 0 / 30 days

No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.

Cataloged from OpenMOSS-Team/MOSS-VL-Instruct-0708

[Image blocked: MOSS-VL]

MOSS-VL-Instruct-0708

MOSS-VL is an open vision-language model family from OpenMOSS, supporting image understanding, long-video understanding, and realtime streaming interaction.

Technical Report: https://arxiv.org/pdf/2608.15045

Introduction

MOSS-VL-Instruct-0708 is the instruction-tuned checkpoint of the MOSS-VL 0708 release, part of the OpenMOSS ecosystem for open visual understanding.

Built on top of MOSS-VL-Base-0708 through supervised fine-tuning (SFT), this checkpoint is designed as a high-performance offline multimodal model. It supports image understanding, OCR, document parsing, visual reasoning, instruction following, and video understanding, with particular strength in long-form video comprehension, temporal reasoning, action recognition, and fine-grained event localization.

The 0708 release keeps the MOSS-VL cross-attention design and a 256K text context window while refreshing the data and instruction-tuning recipe for stronger offline multimodal usage.

Highlights

Strong video understanding: designed for long videos, temporal reasoning, action recognition, and second-level event localization.
General multimodal perception: supports image understanding, fine-grained visual recognition, OCR, and document analysis.
Reliable instruction following: SFT aligns the base checkpoint with user instructions across image, video, and text tasks.
Open model family: released together with MOSS-VL-Base-0708 for continued pretraining, fine-tuning, and applied research.

Model Architecture

MOSS-VL-Instruct-0708 adopts a cross-attention-based vision-language architecture that decouples visual encoding from language reasoning. The model processes images, videos, and text in a unified pipeline and uses cross-attention layers to connect language tokens with visual representations.

[Image blocked: MOSS-VL Architecture]

Key configuration details:

Item	Value
Parameters	11B
Tensor type	BF16
Context length	256K
Vision patch size	16
Temporal patch size	1
Default video FPS	1.0
Default max video frames	256

Absolute Timestamps

For video inputs, MOSS-VL injects absolute timestamps alongside sampled frames. This helps the model reason about event order, duration, pacing, and temporal localization instead of relying only on frame order.

Cross-attention RoPE (XRoPE)

MOSS-VL uses Cross-attention Rotary Position Embedding (XRoPE), which maps text tokens and visual patches into a unified three-dimensional coordinate space defined by Time (t), Height (h), and Width (w). This gives the model a consistent positional representation for image and video reasoning.

Model Performance

MOSS-VL-Instruct-0708 is intended for offline multimodal evaluation across visual perception, multimodal reasoning, OCR/document understanding, and video understanding. Detailed benchmark tables for the 0708 release will be maintained in the MOSS-VL project resources.

[Image blocked: MOSS-VL Offline Benchmark]

For the previous public checkpoint, see MOSS-VL-Instruct-0408.

Quickstart

Installation

Clone the MOSS-VL repository and install the project requirements:

bash

git clone https://github.com/OpenMOSS/MOSS-VL.gitcd MOSS-VLconda create -n moss_vl python=3.12 pip -yconda activate moss_vlpip install -i https://pypi.org/simple --no-build-isolation -r requirements.txt

Load Model

python

import torchfrom transformers import AutoModelForCausalLM, AutoProcessor
checkpoint = "OpenMOSS-Team/MOSS-VL-Instruct-0708"
processor = AutoProcessor.from_pretrained(    checkpoint,    trust_remote_code=True,    frame_extract_num_threads=1,)model = AutoModelForCausalLM.from_pretrained(    checkpoint,    trust_remote_code=True,    device_map="auto",    torch_dtype=torch.bfloat16,    attn_implementation="flash_attention_2",)

Run Inference

Single-image Inference

python

image_path = "data/example_image.jpg"prompt = "Describe this image."
text = model.offline_image_generate(    processor,    prompt=prompt,    image=image_path,    shortest_edge=4096,    longest_edge=16777216,    multi_image_max_pixels=201326592,    patch_size=16,    temporal_patch_size=1,    merge_size=2,    image_mean=[0.5, 0.5, 0.5],    image_std=[0.5, 0.5, 0.5],    max_new_tokens=256,    temperature=1.0,    top_k=50,    top_p=1.0,    repetition_penalty=1.0,    do_sample=False,    vision_chunked_length=64,)
print(text)

Single-video Inference

python

video_path = "data/example_video.mp4"prompt = "Describe this video."
text = model.offline_video_generate(    processor,    prompt=prompt,    video=video_path,    shortest_edge=4096,    longest_edge=16777216,    video_max_pixels=201326592,    patch_size=16,    temporal_patch_size=1,    merge_size=2,    video_fps=1.0,    min_frames=1,    max_frames=256,    num_extract_threads=4,    image_mean=[0.5, 0.5, 0.5],    image_std=[0.5, 0.5, 0.5],    max_new_tokens=256,    temperature=1.0,    top_k=50,    top_p=1.0,    repetition_penalty=1.0,    do_sample=False,    vision_chunked_length=64,)
print(text)

Batched Offline Inference

offline_batch_generate accepts independent image/video/text queries. Queries in the same batch should share the same media_kwargs and generate_kwargs.

python

queries = [    {        "prompt": "Describe sample A.",        "images": [],        "videos": ["data/sample_a.mp4"],        "media_kwargs": {            "video_fps": 1.0,            "min_frames": 8,            "max_frames": 256,        },        "generate_kwargs": {            "temperature": 1.0,            "top_k": 50,            "top_p": 1.0,            "max_new_tokens": 256,            "repetition_penalty": 1.0,            "do_sample": False,        },    },    {        "prompt": "Describe sample B.",        "images": [],        "videos": ["data/sample_b.mp4"],        "media_kwargs": {            "video_fps": 1.0,            "min_frames": 8,            "max_frames": 256,        },        "generate_kwargs": {            "temperature": 1.0,            "top_k": 50,            "top_p": 1.0,            "max_new_tokens": 256,            "repetition_penalty": 1.0,            "do_sample": False,        },    },]
with torch.no_grad():    result = model.offline_batch_generate(        processor,        queries,        vision_chunked_length=64,    )
texts = [item["text"] for item in result["results"]]print(texts)

Related Checkpoints

Model	Parameters	Context	Usage	Hugging Face
MOSS-VL-Realtime	11B	256K	Realtime streaming video interaction	https://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime
MOSS-VL-Instruct	11B	256K	Offline multimodal instruction following	https://huggingface.co/OpenMOSS-Team/MOSS-VL-Instruct-0708
MOSS-VL-Base	11B	256K	Continued pretraining and fine-tuning	https://huggingface.co/OpenMOSS-Team/MOSS-VL-Base-0708
MOSS-VL-Instruct-0408	11B	256K	Previous instruction-tuned checkpoint	https://huggingface.co/OpenMOSS-Team/MOSS-VL-Instruct-0408
MOSS-VL-Base-0408	11B	256K	Previous base checkpoint	https://huggingface.co/OpenMOSS-Team/MOSS-VL-Base-0408

Limitations and Future Work

MOSS-VL-Instruct-0708 is optimized for general offline multimodal understanding. Very dense videos, highly specialized domains, precise small-text OCR, and tasks requiring strict numerical reasoning may still require task-specific prompting, sampling choices, or fine-tuning.

We are continuing to improve mathematical reasoning, code reasoning, RL post-training, and broader task-specific evaluations for future MOSS-VL releases.

Citation

bibtex

@misc{mossvl,  title         = {MOSS-VL Technical Report},  author        = {Wang, Pengyu and Tan, Chenkun and Zhou, Shaojun and Zhou, Qirui and Chen, Yanxin and He, Xingyang and Zeng, Huazheng and Cheng, Jijun and Wang, Chenghao and Qian, Xiaomeng and Wang, Pengfei and Huang, Zhan and Gao, Shanqing and Huang, Wei and Cao, Longjun and Ran, Wu and Liu, Jie and Zhu, Changtai and Wang, Hongkai and Tian, Yixian and Liu, Chenghao and Ye, Zhen and Wang, Xinghao and Jiang, Botian and Feng, Guoguo and Fei, Zhaoye and Li, Ruixiao and Chen, Mingshu and Gao, Yang and Cheng, Qinyuan and Li, Shimin and Qiu, Xipeng},  year          = {2026},  eprint        = {2608.15045},  archivePrefix = {arXiv},  primaryClass  = {cs.CV},  url           = {https://arxiv.org/abs/2608.15045}}
@misc{mossvideopreview,  title         = {{MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention}},  author        = {Pengyu Wang and Chenkun Tan and Shaojun Zhou and Wei Huang and Qirui Zhou and Zhan Huang and Zhen Ye and Jijun Cheng and Xiaomeng Qian and Yanxin Chen and Xingyang He and Huazheng Zeng and Chenghao Wang and Pengfei Wang and Hongkai Wang and Shanqing Gao and Yixian Tian and Chenghao Liu and Xinghao Wang and Botian Jiang and Xipeng Qiu},  year          = {2026},  eprint        = {2606.07639},  archivePrefix = {arXiv},  primaryClass  = {cs.CV},  url           = {https://arxiv.org/abs/2606.07639}}