OpenMOSS-Team/MOSS-VL-Realtime

video-text-to-texttransformersUpstream checksumstransformerssafetensorsmoss_vlfeature-extractionRealtimeStreaming

License: apache-2.0
Size: not mirrored yet
Downloads in the last 30 days: 0 / 30 days

No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.

Cataloged from OpenMOSS-Team/MOSS-VL-Realtime

[Image blocked: MOSS-VL]

MOSS-VL-Realtime

MOSS-VL is an open vision-language model family from OpenMOSS, supporting image understanding, long-video understanding, and realtime streaming interaction.

Technical Report: https://arxiv.org/pdf/2608.15045

Overview

MOSS-VL-Realtime is the realtime streaming checkpoint of the MOSS-VL release, part of the OpenMOSS ecosystem for open visual understanding.

Unlike offline video-language models that first read a complete video and then answer, MOSS-VL-Realtime is designed for continuous video streams. It perceives incoming frames and generates text in parallel, supports questions at arbitrary moments in the stream, and can decide whether to respond or keep observing when the visual evidence is insufficient.

This release keeps the MOSS-VL cross-attention design and a 256K text context window while adding realtime streaming data and an inference interface for timestamped frame-by-frame input.

Key Features

Realtime streaming understanding: processes incoming frames continuously instead of waiting for a complete video.
Interruptible interaction: users can ask questions at any timestamp in a running stream, and the model answers based on the frames observed so far.
Proactive silence: the model can emit <|silence|> and continue observing when there is no meaningful visual update or the context is not sufficient.
Dynamic correction: as new frames arrive, the model can revise earlier responses instead of being locked to an initial interpretation.
Timestamp-aware frames: each streamed frame is associated with an absolute timestamp, helping the model reason about event order, duration, pacing, and fine-grained temporal localization.
Unified MOSS-VL family: released together with MOSS-VL-Instruct and MOSS-VL-Base for offline use, continued pretraining, fine-tuning, and applied research.

Model Design

Architecture

MOSS-VL-Realtime adopts a cross-attention-based vision-language architecture that decouples visual encoding from language reasoning. This design is important for realtime usage because incoming visual content can be integrated into the running generation context without forcing the model into a strictly offline "load all frames, then answer" workflow.

[Image blocked: MOSS-VL Architecture]

Timestamp-aware Video Encoding

For video and realtime frame inputs, MOSS-VL injects absolute timestamps alongside sampled frames. This helps the model reason about when an event happens, how long it lasts, and how the scene changes over time instead of relying only on frame order.

MOSS-VL also uses Cross-attention Rotary Position Embedding (XRoPE), which maps text tokens and visual patches into a unified three-dimensional coordinate space defined by Time (t), Height (h), and Width (w). This gives the model a consistent positional representation for image, offline video, and realtime streaming video reasoning.

Configuration

Item	Value
Parameters	11B
Tensor type	BF16
Context length	256K
Vision patch size	16
Temporal patch size	1
Default video FPS	1.0
Default max video frames	256
Realtime frame format	PIL-compatible image plus timestamp
Realtime session scope	One active realtime session per model instance

Performance

MOSS-VL-Realtime is designed for streaming video understanding benchmarks where questions can arrive before a full video has been observed and correct answers may change as the scene evolves. It targets realtime interaction quality, proactive silence, and dynamic response updates in addition to standard video understanding accuracy.

[Image blocked: MOSS-VL Streaming Benchmark]

Detailed benchmark tables and comparisons for this release will be maintained in the MOSS-VL project resources.

Quickstart

Installation

Clone the MOSS-VL repository and install the project requirements:

bash

git clone https://github.com/OpenMOSS/MOSS-VL.gitcd MOSS-VLconda create -n moss_vl python=3.12 pip -yconda activate moss_vlpip install -i https://pypi.org/simple --no-build-isolation -r requirements.txt

Load the Model

python

import torchfrom transformers import AutoModelForCausalLM, AutoProcessor
checkpoint = "OpenMOSS-Team/MOSS-VL-Realtime"
processor = AutoProcessor.from_pretrained(    checkpoint,    trust_remote_code=True,    frame_extract_num_threads=1,)model = AutoModelForCausalLM.from_pretrained(    checkpoint,    trust_remote_code=True,    device_map="auto",    torch_dtype=torch.bfloat16,    attn_implementation="flash_attention_2",)model.eval()

If FlashAttention is unavailable in your environment, pass attn_implementation="eager" when loading the model.

Inference Examples

Online Inference

Session-style Online Inference

The recommended direct API is create_realtime_session(...). A service or application owns the video capture pipeline, converts camera, screen, or video-file input into PIL-compatible frames, and pushes each frame with a non-decreasing timestamp.

Common session operations:

session.push_frame(image, timestamp=...) appends one visual frame.
session.push_prompt("...") appends a user question while the stream is running.
session.push_prompt_frame(prompt, image, timestamp=...) aligns a prompt with a specific frame.
session.poll_output(...) or session.stream_outputs(...) returns incremental text chunks.

system_prompt and initial_prompt are tokenized as the initial system/user turns before the first frame arrives. Subsequent user turns can be appended with push_prompt(...) while the same session continues observing frames.

For complete real-time inference usage, including local-video replay and service deployment, see realtime_inference in the MOSS-VL GitHub repository.

python

import timefrom PIL import Image
session = model.create_realtime_session(    processor,    initial_prompt=(        "As the video streams frame by frame, describe important changes as they happen. "        "Stay silent when there is no relevant update."    ),    frame_queue_size=256,    max_tokens_per_turn=12,    max_new_tokens=4096,    do_sample=False,)
frame_paths = [    "data/frame_0001.jpg",    "data/frame_0002.jpg",    "data/frame_0003.jpg",]
try:    session.start()
    for index, frame_path in enumerate(frame_paths):        image = Image.open(frame_path).convert("RGB")        session.push_frame(image, timestamp=index / 1.0)
        while True:            chunk = session.poll_output(timeout=0.0)            if chunk is None:                break            print(chunk, end="", flush=True)
        time.sleep(1.0)
    session.push_prompt("What changed in the latest frames?")
    # Realtime sessions stay alive waiting for future input, so use a bounded    # drain window and close the session explicitly when the producer is done.    drain_deadline = time.monotonic() + 5.0    while time.monotonic() < drain_deadline:        chunk = session.poll_output(timeout=0.1)        if chunk is not None:            print(chunk, end="", flush=True)finally:    session.close()

Frame timestamps are measured in seconds and must be non-decreasing within a session. The input producer can be a camera, screen capture, decoded video file, browser frame sampler, or any other source that yields images with timestamps.

Queue-style Online Inference

online_generate(...) is useful for backend systems that separate frame production and model inference through queues. It accepts dictionaries containing frames, prompts, events, reset controls, and stop controls.

python

import queueimport threadingfrom PIL import Image
input_queue = queue.Queue()output_queue = queue.Queue()
worker = threading.Thread(    target=model.online_generate,    args=(processor, input_queue, output_queue),    kwargs={        "frame_queue_size": 256,        "max_tokens_per_turn": 12,        "max_new_tokens": 4096,        "do_sample": False,    },    daemon=True,)worker.start()
input_queue.put({    "initial_prompt": "Answer only when the streamed video provides enough evidence.",})
input_queue.put({"frame": Image.open("data/frame_0001.jpg").convert("RGB"), "timestamp": 0.0})input_queue.put({"frame": Image.open("data/frame_0002.jpg").convert("RGB"), "timestamp": 1.0})input_queue.put({"prompt": "What is happening now?"})
try:    while True:        chunk = output_queue.get(timeout=0.5)        print(chunk, end="", flush=True)except queue.Empty:    pass
input_queue.put({"stop_online_generate": True})worker.join()

Each queue item can contain frame or image, timestamp, prompt, frames, event, events, initial_prompt, system_prompt, generate_kwargs, reset_session, or stop controls such as stop_online_generate.

Offline Inference

MOSS-VL-Realtime also keeps the offline helper APIs for image and video prompts. For purely offline use, MOSS-VL-Instruct is usually the preferred checkpoint, but the realtime checkpoint can still process complete image and video inputs.

Single-video Offline Inference

python

video_path = "data/example_video.mp4"prompt = "Describe this video."
text = model.offline_video_generate(    processor,    prompt=prompt,    video=video_path,    shortest_edge=4096,    longest_edge=16777216,    video_max_pixels=201326592,    patch_size=16,    temporal_patch_size=1,    merge_size=2,    video_fps=1.0,    min_frames=1,    max_frames=256,    num_extract_threads=4,    image_mean=[0.5, 0.5, 0.5],    image_std=[0.5, 0.5, 0.5],    max_new_tokens=256,    temperature=1.0,    top_k=50,    top_p=1.0,    repetition_penalty=1.0,    do_sample=False,    vision_chunked_length=64,)
print(text)

Batched Offline Inference

offline_batch_generate accepts independent image/video/text queries. Queries in the same batch should share the same media_kwargs and generate_kwargs.

python

queries = [    {        "prompt": "Describe sample A.",        "images": [],        "videos": ["data/sample_a.mp4"],        "media_kwargs": {            "video_fps": 1.0,            "min_frames": 8,            "max_frames": 256,        },        "generate_kwargs": {            "temperature": 1.0,            "top_k": 50,            "top_p": 1.0,            "max_new_tokens": 256,            "repetition_penalty": 1.0,            "do_sample": False,        },    },    {        "prompt": "Describe sample B.",        "images": [],        "videos": ["data/sample_b.mp4"],        "media_kwargs": {            "video_fps": 1.0,            "min_frames": 8,            "max_frames": 256,        },        "generate_kwargs": {            "temperature": 1.0,            "top_k": 50,            "top_p": 1.0,            "max_new_tokens": 256,            "repetition_penalty": 1.0,            "do_sample": False,        },    },]
with torch.no_grad():    result = model.offline_batch_generate(        processor,        queries,        vision_chunked_length=64,    )
texts = [item["text"] for item in result["results"]]print(texts)

Related Checkpoints

Model	Parameters	Context	Usage	Hugging Face
MOSS-VL-Realtime	11B	256K	Realtime streaming video interaction	https://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime
MOSS-VL-Instruct	11B	256K	Offline multimodal instruction following	https://huggingface.co/OpenMOSS-Team/MOSS-VL-Instruct
MOSS-VL-Base	11B	256K	Continued pretraining and fine-tuning	https://huggingface.co/OpenMOSS-Team/MOSS-VL-Base
MOSS-VL-Instruct-0408	11B	256K	Previous instruction-tuned checkpoint	https://huggingface.co/OpenMOSS-Team/MOSS-VL-Instruct-0408
MOSS-VL-Base-0408	11B	256K	Previous base checkpoint	https://huggingface.co/OpenMOSS-Team/MOSS-VL-Base-0408

Limitations and Roadmap

MOSS-VL-Realtime is optimized for timestamped frame-by-frame streaming, but production latency depends on GPU hardware, frame sampling rate, transport overhead, and decoding speed. One model instance supports one active realtime session. The default frame queue bounds latency by dropping older pending frames when needed.

We are continuing to improve realtime response timing, dynamic correction, broader streaming evaluations, RL post-training, and task-specific deployment recipes for future MOSS-VL releases.

Citation

bibtex

@misc{mossvl,  title         = {MOSS-VL Technical Report},  author        = {Wang, Pengyu and Tan, Chenkun and Zhou, Shaojun and Zhou, Qirui and Chen, Yanxin and He, Xingyang and Zeng, Huazheng and Cheng, Jijun and Wang, Chenghao and Qian, Xiaomeng and Wang, Pengfei and Huang, Zhan and Gao, Shanqing and Huang, Wei and Cao, Longjun and Ran, Wu and Liu, Jie and Zhu, Changtai and Wang, Hongkai and Tian, Yixian and Liu, Chenghao and Ye, Zhen and Wang, Xinghao and Jiang, Botian and Feng, Guoguo and Fei, Zhaoye and Li, Ruixiao and Chen, Mingshu and Gao, Yang and Cheng, Qinyuan and Li, Shimin and Qiu, Xipeng},  year          = {2026},  eprint        = {2608.15045},  archivePrefix = {arXiv},  primaryClass  = {cs.CV},  url           = {https://arxiv.org/abs/2608.15045}}
@misc{mossvideopreview,  title         = {{MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention}},  author        = {Pengyu Wang and Chenkun Tan and Shaojun Zhou and Wei Huang and Qirui Zhou and Zhan Huang and Zhen Ye and Jijun Cheng and Xiaomeng Qian and Yanxin Chen and Xingyang He and Huazheng Zeng and Chenghao Wang and Pengfei Wang and Hongkai Wang and Shanqing Gao and Yixian Tian and Chenghao Liu and Xinghao Wang and Botian Jiang and Xipeng Qiu},  year          = {2026},  eprint        = {2606.07639},  archivePrefix = {arXiv},  primaryClass  = {cs.CV},  url           = {https://arxiv.org/abs/2606.07639}}