No official torrent yet β requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from ibm-granite/granite-vision-3.3-2b-embedding
All models by ibm-granite βModel Summary: Granite-vision-3.3-2b-embedding is an efficient embedding model based on granite-vision-3.3-2b. This model is specifically designed for multimodal document retrieval, enabling queries on documents with tables, charts, infographics, and complex layouts. The model generates ColBERT-style multi-vector representations of pages. By removing the need for OCR-based text extractions, granite-vision-3.3-2b-embedding can help simplify and accelerate RAG pipelines.
Evaluations: We evaluated granite-vision-3.3-2b-embedding alongside other top colBERT style multi-modal embedding models in the 1B-4B parameter range using two benchmark: Vidore2 and Real-MM-RAG-Bench which aim to specifically address complex multimodal document retrieval tasks.
Intended Use: The model is intended to be used in enterprise applications that involve retrieval of visual and text data. In particular, the model is well-suited for multi-modal RAG systems where the knowledge base is composed of complex enterprise documents, such as reports, slides, images, canned doscuments, manuals and more. The model can be used as a standalone retriever, or alongside a text-based retriever.
Then run the code:
For an example of MM-RAG using granite-vision-3.3-2b-embedding refer to this notebook.
Model Architecture: The architecture of granite-vision-3.3-2b-embedding follows ColPali(https://arxiv.org/abs/2407.01449) approach and consists of the following components:
(1) Vision-Language model : granite-vision-3.3-2b (https://huggingface.co/ibm-granite/granite-vision-3.3-2b).
(2) Projection layer: linear layer that projects the hidden layer dimension of Vision-Language model to 128 and outputs 729 embedding vectors per image.
The scoring is computed using MaxSim-based late interaction mechanism.
Training Data: Our training data is entirly comprised from DocFM. DocFM is a large-scale comprehensive dataset effort at IBM consisting of 85 million document pages extracted from unique PDF documents sourced from Common Crawl, Wikipedia, and ESG (Environmental, Social, and Governance) reports.
Infrastructure: We train granite-vision-3.3-2b-embedding on IBMβs cognitive computing cluster, which is outfitted with NVIDIA A100 GPUs.
Ethical Considerations and Limitations: The use of Large Vision and Language Models involves risks and ethical considerations people must be aware of, including but not limited to: bias and fairness, misinformation, and autonomous decision-making. Granite-vision-3.3-2b-embedding is not the exception in this regard. Although our alignment processes include safety considerations, the model may in some cases produce inaccurate or biased responses. Regarding ethics, a latent risk associated with all Large Language Models is their malicious utilization. We urge the community to use granite-vision-3.3-2b-embedding with ethical intentions and in a responsible way.
Resources
pip install -q torch torchvision torchaudiopip install transformers==4.50from io import BytesIO
import requestsimport torchfrom PIL import Imagefrom transformers import AutoProcessor, AutoModelfrom transformers.utils.import_utils import is_flash_attn_2_available
device = "cuda" if torch.cuda.is_available() else "cpu"model_name = "ibm-granite/granite-vision-3.3-2b-embedding"model = AutoModel.from_pretrained( model_name, trust_remote_code=True, torch_dtype=torch.float16, device_map=device, attn_implementation="flash_attention_2" if is_flash_attn_2_available() else None ).eval()processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)
# βββββββββββββββββββββββββββββββββββββββββββββ# Inputs: Image + Text# βββββββββββββββββββββββββββββββββββββββββββββimage_url = "https://huggingface.co/datasets/mishig/sample_images/resolve/main/tiger.jpg"print("\nFetching image...")image = Image.open(BytesIO(requests.get(image_url).content)).convert("RGB")
text = "A photo of a tiger"print(f"Image and text inputs ready.")
# Process both inputsprint("Processing inputs...")image_inputs = processor.process_images([image])text_inputs = processor.process_queries([text])
# Move to correct deviceimage_inputs = {k: v.to(device) for k, v in image_inputs.items()}text_inputs = {k: v.to(device) for k, v in text_inputs.items()}
# βββββββββββββββββββββββββββββββββββββββββββββ# Run Inference# βββββββββββββββββββββββββββββββββββββββββββββwith torch.no_grad(): print("π Getting image embedding...") img_emb = model(**image_inputs)
print("βοΈ Getting text embedding...") txt_emb = model(**text_inputs)
# βββββββββββββββββββββββββββββββββββββββββββββ# Score the similarity# βββββββββββββββββββββββββββββββββββββββββββββprint("Scoring similarity...")similarity = processor.score(txt_emb, img_emb, batch_size=1, device=device)
print("\n" + "=" * 50)print(f"π Similarity between image and text: {similarity.item():.4f}")print("=" * 50)