No official torrent yet β requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from vllm-sr/modernbert-base-32k
All models by vllm-sr βA long-context extension of ModernBERT-base fine-tuned for 32K token context length using YaRN (Yet another RoPE extensioN) scaling.
This model extends the original ModernBERT-base from 8,192 tokens to 32,768 tokens context length while preserving the base model's capabilities. It was fine-tuned using specialized techniques to maintain long-range retrieval performance:
Passkey Retrieval at Long Distances: Like the base model, this model struggles with needle-in-haystack retrieval beyond ~1K tokens. This is a fundamental limitation of MLM architectures, not the fine-tuning.
Context Utilization: At very long contexts (16K+), the model may not always benefit from additional context for MLM predictions. This is expected behavior for encoder-only models.
Memory Requirements: Processing 32K tokens requires significant GPU memory (~16GB+ for inference).
Domain: Trained primarily on web text; may not generalize well to specialized domains without additional fine-tuning.
This model uses several techniques to preserve long-range capabilities during fine-tuning:
Extends positional embeddings from 8K to 32K using Yet another RoPE extensioN with:
Custom MLM objective that creates explicit long-range dependencies:
Prevents catastrophic forgetting by:
If you use this model, please cite:
Also cite the original ModernBERT paper:
Apache 2.0 (same as base model)
learning_rate: 1e-5lr_scheduler: constant_with_warmupwarmup_ratio: 0.1epochs: 1batch_size: 6gradient_accumulation: 1precision: bf16max_length: 32768
# RoPE Scalingrope_scaling_type: yarnrope_scaling_factor: 4.0
# Retrieval Maskingmlm_probability: 0.30retrieval_probability: 0.10min_distance_for_retrieval: 512
# EWC Regularizationewc_lambda: 1000.0ewc_samples: 200from transformers import AutoModelForMaskedLM, AutoTokenizer
model = AutoModelForMaskedLM.from_pretrained("vllm-sr/modernbert-base-32k")tokenizer = AutoTokenizer.from_pretrained("vllm-sr/modernbert-base-32k")
text = "The capital of France is [MASK]."inputs = tokenizer(text, return_tensors="pt")outputs = model(**inputs)
# Get predictionsmask_idx = (inputs.input_ids == tokenizer.mask_token_id).nonzero(as_tuple=True)[1]logits = outputs.logits[0, mask_idx, :]top_tokens = logits.topk(5).indices[0]print([tokenizer.decode(t) for t in top_tokens])# ['Paris', 'Lyon', 'Nancy', ...]# For sequences longer than 8192 tokens, YaRN scaling is automatically appliedlong_text = "..." * 10000 # Very long documentinputs = tokenizer( long_text, return_tensors="pt", max_length=32768, truncation=True)outputs = model(**inputs)from transformers import AutoModel
model = AutoModel.from_pretrained("vllm-sr/modernbert-base-32k")
inputs = tokenizer(text, return_tensors="pt")outputs = model(**inputs)embeddings = outputs.last_hidden_state # [batch, seq_len, 768]
# Mean pooling for sentence embeddingattention_mask = inputs["attention_mask"]masked_embeddings = embeddings * attention_mask.unsqueeze(-1)sentence_embedding = masked_embeddings.sum(1) / attention_mask.sum(1, keepdim=True)@misc{modernbert-32k, title={ModernBERT-base-32K: Long-Context Extension of ModernBERT}, author={LLM Semantic Router}, year={2026}, url={https://huggingface.co/vllm-sr/modernbert-base-32k}}@article{modernbert2024, title={Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference}, author={Warner, Benjamin and Chaffin, Antoine and Geiger, Benjamin and Werra, Leandro von and Tunstall, Lewis and Bartolo, Max and Thrush, Tristan}, journal={arXiv preprint}, year={2024}}