IFM/K2-Horizon-MoVA-36B-A4B
text-generationtransformersVerified checksumsk2-horizon36bmovamoeopen-weightsifm
- License
- apache-2.0
- Size
- 70 GB
- Revision
- main
Mirrored from IFM/K2-Horizon-MoVA-36B-A4B
All models by IFM →Mirrored from IFM/K2-Horizon-MoVA-36B-A4B
All models by IFM →K2-Horizon-MoVA-36B-A4B is the sparse member of the K2-Horizon family: a Mixture-of-Experts model with Mixture-of-Values attention (MoVA) that stores 36B parameters and runs 4B per token. We have released the final checkpoint; intermediate checkpoints, along with the data and the training code, will be released.
[Image blocked: K2-Horizon-MoVA-36B-A4B benchmark results against open MoE, dense, and closed models]
Scores in %. Bold marks the best score in each row. Sections follow the Artificial Analysis Intelligence Index categories. Baseline scores are from Artificial Analysis; Muse Glimmer-30B at high reasoning effort, all other open models in their reasoning mode.
vLLM, recipe at recipes.vllm.ai/IFM:
Use an exact branch name from the inventory with vLLM's --revision option. For example, --revision pretrain_1100000 selects the final checkpoint of Pretraining, at step 1,100,000.
SGLang recipe validated on 2× H200 in the SGLang K2 Horizon cookbook:
[!Tip] Recommended settings:
reasoning_effort="high",temperature=1.0,top_p=0.95. Reasoning depth is selected per request throughchat_template_kwargs. Thinking is returned inreasoning_contentand the answer incontent.
Our model supports multiple tool calls formats, which can be changed with chat_template_kwargs. The supported values are json, xml, and xml_typed . The default is xml. Keep --tool-call-parser k2_horizon enabled to parse the selected format.
Validated with Transformers 5.15.0, PyTorch 2.13.0, Safetensors 0.8.0.
The table below lists the training stages in order and the purpose of each stage.
Training steps are counted within each stage or phase. Token budgets cover only the additional training in that stage or phase. For example, the 50B tokens listed for SFT Phase 2 are additional to the 219B tokens in Phase 1, bringing the cumulative budget to 269B tokens by the end of Phase 2. Here, B and T denote billion and trillion tokens, respectively.
Each stage or phase continues from the final checkpoint of the preceding stage or phase.
Some stages, such as SFT, have multiple phases with slight changes to the data mix while retaining the same overall purpose.
The tables below list the release artifacts for K2-Horizon-MoVA-36B-A4B, their availability, and the expected release dates for remaining items.
Last updated: 2026-09-11
Status:
Model repository: IFM/K2-Horizon-MoVA-36B-A4B
Branch names below refer to this repository. Patterns containing * group branches by training stage or phase. The * is a placeholder for a training-step number, not a literal branch name. Intermediate checkpoint groups exclude the final checkpoint listed separately; a pattern does not imply that a checkpoint is available at every step.
For example, sft_1_11000 is the checkpoint saved at training step 11,000 within SFT Phase 1, and is the final checkpoint of that phase. The numeric suffix is the step within the named stage or phase, not the cumulative step across all training. Thus, sft_2_2500 refers to step 2,500 within SFT Phase 2.
For a partially released group, the available checkpoints and the remaining checkpoints are listed in the notes.
high. All reported results use high reasoning effort. Pass {"chat_template_kwargs": {"reasoning_effort": "high"}} on every request.temperature=1.0, top_p=0.95.xllm_source_router_gemm_partitions override, which preserves the checkpoint's router numerics. Full recipes for every K2-Horizon size, with measured H200 latency and throughput, are in the SGLang cookbook and the vLLM recipe.k2_horizon reasoning parser for chat, and add the k2_horizon tool-call parser for agent use. Leave both off for plain completion-style generation.vllm serve IFM/K2-Horizon-MoVA-36B-A4B \ --revision main \ --tensor-parallel-size 2 \ --enable-expert-parallel \ --trust-remote-code \ --dtype bfloat16 \ --reasoning-parser k2_horizon \ --tool-call-parser k2_horizon \ --enable-auto-tool-choicepython3 -m sglang.launch_server \ --model-path IFM/K2-Horizon-MoVA-36B-A4B \ --revision main \ --tp 2 \ --ep 2 \ --dtype bfloat16 \ --attention-backend fa3 \ --json-model-override-args '{"xllm_source_router_gemm_partitions":2}' \ --reasoning-parser k2_horizon \ --tool-call-parser k2_horizon \ --host 0.0.0.0 --port 30000from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")response = client.chat.completions.create( model="IFM/K2-Horizon-MoVA-36B-A4B", messages=[{"role": "user", "content": "Explain the result step by step."}], temperature=1.0, top_p=0.95, max_tokens=32768, extra_body={"chat_template_kwargs": {"reasoning_effort": "high", "tool_call_format": "xml"}},)message = response.choices[0].messageprint("Reasoning:", getattr(message, "reasoning_content", None))print("Answer:", message.content)from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "IFM/K2-Horizon-MoVA-36B-A4B"tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)model = AutoModelForCausalLM.from_pretrained( model_id, device_map="auto", dtype="bfloat16", low_cpu_mem_usage=True, trust_remote_code=True)
inputs = tokenizer("Explain why long-context evaluation is difficult.", return_tensors="pt").to(model.device)inputs.pop("token_type_ids", None)outputs = model.generate(**inputs, max_new_tokens=32768, temperature=1.0, top_p=0.95, do_sample=True)print(tokenizer.decode(outputs[0], skip_special_tokens=True))@misc{k2horizon2026, title = {Introducing K2 Horizon: Frontier Performance, Radically Open}, author = {{IFM Team}}, year = {2026}, url = {https://ifm.ai/blog/k2/},}