No official torrent yet β requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from sentence-transformers/msmarco-distilbert-dot-v5
All models by sentence-transformers βThis is a sentence-transformers model: It maps sentences & paragraphs to a 768 dimensional dense vector space and was designed for semantic search. It has been trained on 500K (query, answer) pairs from the MS MARCO dataset. For an introduction to semantic search, have a look at: SBERT.net - Semantic Search
Using this model becomes easy when you have sentence-transformers installed:
Then you can use the model like this:
Without sentence-transformers, you can use the model like this: First, you pass your input through the transformer model, then you have to apply the correct pooling-operation on-top of the contextualized word embeddings.
In the following some technical details how this model must be used:
See train_script.py in this repository for the used training script.
The model was trained with the parameters:
DataLoader:
torch.utils.data.dataloader.DataLoader of length 7858 with parameters:
Loss:
sentence_transformers.losses.MarginMSELoss.MarginMSELoss
Parameters of the fit()-Method:
This model was trained by sentence-transformers.
If you find this model helpful, feel free to cite our publication Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks:
This model is released under the Apache 2 license. However, note that this model was trained on the MS MARCO dataset which has it's own license restrictions: MS MARCO - Terms and Conditions.
pip install -U sentence-transformersfrom sentence_transformers import SentenceTransformer, util
query = "How many people live in London?"docs = ["Around 9 Million people live in London", "London is known for its financial district"]
#Load the modelmodel = SentenceTransformer('sentence-transformers/msmarco-distilbert-dot-v5')
#Encode query and documentsquery_emb = model.encode(query)doc_emb = model.encode(docs)
#Compute dot score between query and all document embeddingsscores = util.dot_score(query_emb, doc_emb)[0].cpu().tolist()
#Combine docs & scoresdoc_score_pairs = list(zip(docs, scores))
#Sort by decreasing scoredoc_score_pairs = sorted(doc_score_pairs, key=lambda x: x[1], reverse=True)
#Output passages & scoresprint("Query:", query)for doc, score in doc_score_pairs: print(score, doc)from transformers import AutoTokenizer, AutoModelimport torch
#Mean Pooling - Take attention mask into account for correct averagingdef mean_pooling(model_output, attention_mask): token_embeddings = model_output.last_hidden_state input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float() return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
#Encode textdef encode(texts): # Tokenize sentences encoded_input = tokenizer(texts, padding=True, truncation=True, return_tensors='pt')
# Compute token embeddings with torch.no_grad(): model_output = model(**encoded_input, return_dict=True)
# Perform pooling embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
return embeddings
# Sentences we want sentence embeddings forquery = "How many people live in London?"docs = ["Around 9 Million people live in London", "London is known for its financial district"]
# Load model from HuggingFace Hubtokenizer = AutoTokenizer.from_pretrained("sentence-transformers/msmarco-distilbert-dot-v5")model = AutoModel.from_pretrained("sentence-transformers/msmarco-distilbert-dot-v5")
#Encode query and docsquery_emb = encode(query)doc_emb = encode(docs)
#Compute dot score between query and all document embeddingsscores = torch.mm(query_emb, doc_emb.transpose(0, 1))[0].cpu().tolist()
#Combine docs & scoresdoc_score_pairs = list(zip(docs, scores))
#Sort by decreasing scoredoc_score_pairs = sorted(doc_score_pairs, key=lambda x: x[1], reverse=True)
#Output passages & scoresprint("Query:", query)for doc, score in doc_score_pairs: print(score, doc){'batch_size': 64, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}{ "callback": null, "epochs": 30, "evaluation_steps": 0, "evaluator": "NoneType", "max_grad_norm": 1, "optimizer_class": "<class 'transformers.optimization.AdamW'>", "optimizer_params": { "lr": 1e-05 }, "scheduler": "WarmupLinear", "steps_per_epoch": null, "warmup_steps": 10000, "weight_decay": 0.01}SentenceTransformer( (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: DistilBertModel (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False}))@inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "http://arxiv.org/abs/1908.10084",}