No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from vllm-sr/mmbert-32k-yarn
All models by vllm-sr →Modern Multilingual BERT with 32K context length - Extended from 8K to 32K tokens using YaRN RoPE scaling.
This model extends jhu-clsp/mmBERT-base (Modern Multilingual BERT supporting 1800+ languages) from 8,192 to 32,768 maximum context length using YaRN (Yet another RoPE extensioN) scaling method.
This model is designed for:
Part of the vLLM Semantic Router Mixture-of-Models (MoM) family.
Summary: Perfect retrieval up to 2048 tokens. Long-range capability improved from baseline ~33% to 50% (averaged across all distances ≥1024).
Overall: 100% (10/10 languages tested)
An ONNX export is available for high-performance inference with ONNX Runtime.
MIT License (same as mmBERT base model)
base_model: jhu-clsp/mmBERT-baserope_scaling_type: yarnoriginal_max_position_embeddings: 8192target_max_position_embeddings: 32768scaling_factor: 4.0yarn_beta_fast: 32.0yarn_beta_slow: 1.0
# Training hyperparameterslearning_rate: 1e-5batch_size: 1 (effective: 16 with gradient accumulation)gradient_accumulation_steps: 16num_epochs: 1warmup_steps: 100lr_scheduler: constant_with_warmupmlm_probability: 0.3bf16: truefrom transformers import AutoModelForMaskedLM, AutoTokenizer
model = AutoModelForMaskedLM.from_pretrained("vllm-sr/mmbert-32k-yarn")tokenizer = AutoTokenizer.from_pretrained("vllm-sr/mmbert-32k-yarn")
# Multilingual MLM exampletext = "The capital of France is <mask>."inputs = tokenizer(text, return_tensors="pt")outputs = model(**inputs)
mask_idx = (inputs["input_ids"] == tokenizer.mask_token_id).nonzero()[0, 1]logits = outputs.logits[0, mask_idx]top5 = tokenizer.decode(logits.topk(5).indices)print(top5) # ['Paris', 'Strasbourg', 'Nice', 'Brussels', 'Lyon']# Process long documents (up to 32K tokens)long_document = "..." * 30000 # Your long text in any of 1800+ languagesinputs = tokenizer( long_document, return_tensors="pt", max_length=32768, truncation=True)outputs = model(**inputs)import torch
# Get embeddings for downstream taskswith torch.no_grad(): outputs = model(**inputs, output_hidden_states=True) # Use last hidden state or pooled output embeddings = outputs.hidden_states[-1].mean(dim=1) # Mean poolingimport onnxruntime as ortfrom transformers import AutoTokenizerimport numpy as np
# Load tokenizer and ONNX modeltokenizer = AutoTokenizer.from_pretrained("vllm-sr/mmbert-32k-yarn")sess = ort.InferenceSession( "onnx/model.onnx", # or download from HF providers=['ROCmExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider'])
# Inferencetext = "What is the weather like today?"inputs = tokenizer(text, return_tensors="np", padding=True)outputs = sess.run(None, { 'input_ids': inputs['input_ids'].astype(np.int64), 'attention_mask': inputs['attention_mask'].astype(np.int64)})embeddings = outputs[0].mean(axis=1) # Mean poolinguse onnx_semantic_router::MmBertEmbeddingModel;
let model = MmBertEmbeddingModel::load("./mmbert-32k-yarn-onnx", false)?;let embeddings = model.embed("What is the weather?")?;@misc{mmbert-32k-yarn, title={mmBERT-32K-YaRN: Extended Context Modern Multilingual BERT}, author={vLLM Semantic Router Team}, year={2026}, publisher={Hugging Face}, url={https://huggingface.co/vllm-sr/mmbert-32k-yarn}}