No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from OpenMOSS-Team/MOSS-VL-Instruct-0708-FP8
All models by OpenMOSS-Team →[Image blocked: MOSS-VL]
English | 中文
MOSS-VL is an open vision-language model family from OpenMOSS, supporting image understanding, long-video understanding, and realtime streaming interaction. This repository provides the FP8-quantized checkpoint of MOSS-VL-Instruct-0708.
Technical Report: https://arxiv.org/pdf/2608.15045
This is the 24 GiB quantized release of MOSS-VL-Instruct-0708. The same checkpoint directory is supported by both the standard Transformers inference path and the native SGLang MOSS-VL backend.
The FP8 weights are shared across both engines. Transformers reads the HQQ KV8
configuration from generation_config.json; SGLang loads the same weights with
model_impl=sglang and uses its own cache implementation.
Across the selected benchmarks, the quantized models remain close to their non-quantized BF16 counterparts, showing that overall model quality is largely preserved after quantization.
[Image blocked: MOSS-VL quantization benchmark comparison]
The model is designed to run with the Transformers inference path on a single NVIDIA GPU with 24 GB of VRAM. SGLang memory requirements depend on its server configuration and KV cache allocation.
Start from the standard MOSS-VL release environment, then install the FP8 checkpoint loader and HQQ cache backend:
Core versions used by the standard Transformers path:
Video decoding also requires FFmpeg to be available in PATH.
generation_config.json enables HQQ KV8 automatically. Keep that file beside
the checkpoint and do not pass a conflicting cache configuration.
The standard repository runner accepts JSON/JSONL queries with image or video content and loads this directory directly:
Use a separate environment for SGLang. The native backend in the
mossvl_sglang release directory pins:
Create the environment from its complete lock file:
Start the OpenAI-compatible service with the quantized directory as the model path:
The service listens on http://127.0.0.1:30000 by default. Keep
MODEL_IMPL=sglang; do not force the Transformers remote implementation.
SGLang intentionally uses its native BF16 KV cache rather than the HQQ cache in
generation_config.json.
config.json: model and FP8 weight/activation configuration.generation_config.json: Transformers HQQ KV8 configuration.modeling_moss_vl.py: checkpoint-local cross-attention/QuantizedCache code.git clone https://github.com/OpenMOSS/MOSS-VL.gitcd MOSS-VL
conda create -n moss_vl_quant python=3.12 pip -yconda activate moss_vl_quantpip install -i https://pypi.org/simple --no-build-isolation -r requirements.txtpip install -i https://pypi.org/simple \ compressed-tensors==0.14.0 \ hqq==0.2.8.post1python -m pip checkimport torchfrom transformers import AutoModelForCausalLM, AutoProcessor
checkpoint = "/path/to/MOSS-VL-0708-Instruct-FP8-Dynamic-KV8-HQQ"
processor = AutoProcessor.from_pretrained( checkpoint, trust_remote_code=True, frame_extract_num_threads=1,)model = AutoModelForCausalLM.from_pretrained( checkpoint, trust_remote_code=True, device_map="auto", torch_dtype=torch.bfloat16, attn_implementation="flash_attention_2",)model.eval()text = model.offline_image_generate( processor, prompt="Describe this image.", image="data/example_image.jpg", shortest_edge=4096, longest_edge=16777216, multi_image_max_pixels=201326592, patch_size=16, temporal_patch_size=1, merge_size=2, image_mean=[0.5, 0.5, 0.5], image_std=[0.5, 0.5, 0.5], max_new_tokens=256, temperature=1.0, top_k=50, top_p=1.0, repetition_penalty=1.0, do_sample=False, vision_chunked_length=64,)print(text)text = model.offline_video_generate( processor, prompt="Describe this video.", video="data/example_video.mp4", shortest_edge=4096, longest_edge=16777216, video_max_pixels=201326592, patch_size=16, temporal_patch_size=1, merge_size=2, video_fps=1.0, min_frames=1, max_frames=256, num_extract_threads=4, image_mean=[0.5, 0.5, 0.5], image_std=[0.5, 0.5, 0.5], max_new_tokens=256, temperature=1.0, top_k=50, top_p=1.0, repetition_penalty=1.0, do_sample=False, vision_chunked_length=64,)print(text)python inference/run_inference.py \ --checkpoint /path/to/MOSS-VL-0708-Instruct-FP8-Dynamic-KV8-HQQ \ --mode offline \ --input inference/video_queries.json \ --output video_results.jsoncd /path/to/mossvl_sglangenv -u INDEX_URL -u PIP_INDEX_URL conda create -y -n mossvl-sglang-0511 \ --override-channels \ -c https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main \ python=3.12.8 pip=24.2conda activate mossvl-sglang-0511env -u INDEX_URL -u PIP_INDEX_URL python -m pip install -r requirements.txtpython -m pip checkconda activate mossvl-sglang-0511cd /path/to/mossvl_sglangGPU_IDS=0 TP_SIZE=1 MEM_FRACTION_STATIC=0.35 \ ./start_sglang_moss_vl.sh \ --model-path /path/to/MOSS-VL-0708-Instruct-FP8-Dynamic-KV8-HQQ@misc{mossvl, title = {MOSS-VL Technical Report}, author = {Wang, Pengyu and Tan, Chenkun and Zhou, Shaojun and Zhou, Qirui and Chen, Yanxin and He, Xingyang and Zeng, Huazheng and Cheng, Jijun and Wang, Chenghao and Qian, Xiaomeng and Wang, Pengfei and Huang, Zhan and Gao, Shanqing and Huang, Wei and Cao, Longjun and Ran, Wu and Liu, Jie and Zhu, Changtai and Wang, Hongkai and Tian, Yixian and Liu, Chenghao and Ye, Zhen and Wang, Xinghao and Jiang, Botian and Feng, Guoguo and Fei, Zhaoye and Li, Ruixiao and Chen, Mingshu and Gao, Yang and Cheng, Qinyuan and Li, Shimin and Qiu, Xipeng}, year = {2026}, eprint = {2608.15045}, archivePrefix = {arXiv}, primaryClass = {cs.CV}, url = {https://arxiv.org/abs/2608.15045}}
@misc{mossvideopreview, title = {{MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention}}, author = {Pengyu Wang and Chenkun Tan and Shaojun Zhou and Wei Huang and Qirui Zhou and Zhan Huang and Zhen Ye and Jijun Cheng and Xiaomeng Qian and Yanxin Chen and Xingyang He and Huazheng Zeng and Chenghao Wang and Pengfei Wang and Hongkai Wang and Shanqing Gao and Yixian Tian and Chenghao Liu and Xinghao Wang and Botian Jiang and Xipeng Qiu}, year = {2026}, eprint = {2606.07639}, archivePrefix = {arXiv}, primaryClass = {cs.CV}, url = {https://arxiv.org/abs/2606.07639}}