No official torrent yet β requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from unsloth/gte-modernbert-base
All models by unsloth βWe are excited to introduce the gte-modernbert series of models, which are built upon the latest modernBERT pre-trained encoder-only foundation models. The gte-modernbert series models include both text embedding models and rerank models.
The gte-modernbert models demonstrates competitive performance in several text embedding and text retrieval evaluation tasks when compared to similar-scale models from the current open-source community. This includes assessments such as MTEB, LoCO, and COIR evaluation.
[!TIP] For
transformersandsentence-transformers, if your GPU supports it, the efficient Flash Attention 2 will be used automatically if you haveflash_attninstalled. It is not mandatory.
Use with transformers
Use with sentence-transformers:
Use with transformers.js:
Additionally, you can also deploy Alibaba-NLP/gte-modernbert-base with Text Embeddings Inference (TEI) as follows:
Then you can send requests to the deployed API via the OpenAI-compatible v1/embeddings route (more information about the OpenAI Embeddings API):
The gte-modernbert series of models follows the training scheme of the previous GTE models, with the only difference being that the pre-training language model base has been replaced from GTE-MLM to ModernBert. For more training details, please refer to our paper: mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval
The results of other models are retrieved from MTEB leaderboard. Given that all models in the gte-modernbert series have a size of less than 1B parameters, we focused exclusively on the results of models under 1B from the MTEB leaderboard.
We have open positions for Research Interns and Full-Time Researchers to join our team at Tongyi Lab. We are seeking passionate individuals with expertise in representation learning, LLM-driven information retrieval, Retrieval-Augmented Generation (RAG), and agent-based systems. Our team is located in the vibrant cities of Beijing and Hangzhou. If you are driven by curiosity and eager to make a meaningful impact through your work, we would love to hear from you. Please submit your resume along with a brief introduction to dingkun.ldk@alibaba-inc.com.
If you find our paper or models helpful, feel free to give us a cite.
pip install flash_attn# Requires transformers>=4.48.0
import torch.nn.functional as Ffrom transformers import AutoModel, AutoTokenizer
input_texts = [ "what is the capital of China?", "how to implement quick sort in python?", "Beijing", "sorting algorithms"]
model_path = "Alibaba-NLP/gte-modernbert-base"tokenizer = AutoTokenizer.from_pretrained(model_path)model = AutoModel.from_pretrained(model_path)
# Tokenize the input textsbatch_dict = tokenizer(input_texts, max_length=8192, padding=True, truncation=True, return_tensors='pt')
outputs = model(**batch_dict)embeddings = outputs.last_hidden_state[:, 0] # (Optionally) normalize embeddingsembeddings = F.normalize(embeddings, p=2, dim=1)scores = (embeddings[:1] @ embeddings[1:].T) * 100print(scores.tolist())# [[42.89073944091797, 71.30911254882812, 33.664554595947266]]# Requires transformers>=4.48.0from sentence_transformers import SentenceTransformerfrom sentence_transformers.util import cos_sim
input_texts = [ "what is the capital of China?", "how to implement quick sort in python?", "Beijing", "sorting algorithms"]
model = SentenceTransformer("Alibaba-NLP/gte-modernbert-base")embeddings = model.encode(input_texts)print(embeddings.shape)# (4, 768)
similarities = cos_sim(embeddings[0], embeddings[1:])print(similarities)# tensor([[0.4289, 0.7131, 0.3366]])// npm i @huggingface/transformersimport { pipeline, matmul } from "@huggingface/transformers";
// Create a feature extraction pipelineconst extractor = await pipeline( "feature-extraction", "Alibaba-NLP/gte-modernbert-base", { dtype: "fp32" }, // Supported options: "fp32", "fp16", "q8", "q4", "q4f16");
// Embed queries and documentsconst embeddings = await extractor( [ "what is the capital of China?", "how to implement quick sort in python?", "Beijing", "sorting algorithms", ], { pooling: "cls", normalize: true },);
// Compute similarity scoresconst similarities = (await matmul(embeddings.slice([0, 1]), embeddings.slice([1, null]).transpose(1, 0))).mul(100);console.log(similarities.tolist()); // [[42.89077377319336, 71.30916595458984, 33.66455841064453]]docker run --platform linux/amd64 \ -p 8080:80 \ -v $PWD/data:/data \ --pull always \ ghcr.io/huggingface/text-embeddings-inference:cpu-1.7 \ --model-id Alibaba-NLP/gte-modernbert-basedocker run --gpus all \ -p 8080:80 \ -v $PWD/data:/data \ --pull always \ ghcr.io/huggingface/text-embeddings-inference:1.7 \ --model-id Alibaba-NLP/gte-modernbert-basecurl https://0.0.0.0:8080/v1/embeddings \ -H "Content-Type: application/json" \ -d '{ "input": [ "what is the capital of China?", "how to implement quick sort in python?", "Beijing", "sorting algorithms" ], "model": "Alibaba-NLP/gte-modernbert-base", "encoding_format": "float" }'@inproceedings{zhang2024mgte, title={mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval}, author={Zhang, Xin and Zhang, Yanzhao and Long, Dingkun and Xie, Wen and Dai, Ziqi and Tang, Jialong and Lin, Huan and Yang, Baosong and Xie, Pengjun and Huang, Fei and others}, booktitle={Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track}, pages={1393--1412}, year={2024}}
@article{li2023towards, title={Towards general text embeddings with multi-stage contrastive learning}, author={Li, Zehan and Zhang, Xin and Zhang, Yanzhao and Long, Dingkun and Xie, Pengjun and Zhang, Meishan}, journal={arXiv preprint arXiv:2308.03281}, year={2023}}