BAAI/Emu3-VisionTokenizer

feature-extractiontransformersUpstream checksumstransformerssafetensorsEmu3VisionVQfeature-extractioncustom_codearxiv:2409.18869

License: apache-2.0
Size: not mirrored yet
Downloads in the last 30 days: 0 / 30 days

No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.

Cataloged from BAAI/Emu3-VisionTokenizer

All models by BAAI →

Emu3: Next-Token Prediction is All You Need

Emu3 Team, BAAI

We introduce Emu3, a new suite of state-of-the-art multimodal models trained solely with next-token prediction! By tokenizing images, text, and videos into a discrete space, we train a single transformer from scratch on a mixture of multimodal sequences.

Emu3 excels in both generation and perception

Emu3 outperforms several well-established task-specific models in both generation and perception tasks, surpassing flagship open models such as SDXL, LLaVA-1.6 and OpenSora-1.2, while eliminating the need for diffusion or compositional architectures.

Highlights

Emu3 is capable of generating high-quality images following the text input, by simply predicting the next vision token. The model naturally supports flexible resolutions and styles.
Emu3 shows strong vision-language understanding capabilities to see the physical world and provides coherent text responses. Notably, this capability is achieved without depending on a CLIP and a pretrained LLM.
Emu3 simply generates a video causally by predicting the next token in a video sequence, unlike the video diffusion model as in Sora. With a video in context, Emu3 can also naturally extend the video and predict what will happen next.

Quickstart for Autoencoding

python

import osimport os.path as osp
from PIL import Imageimport torchfrom transformers import AutoModel, AutoImageProcessor
MODEL_HUB = "BAAI/Emu3-VisionTokenizer"
model = AutoModel.from_pretrained(MODEL_HUB, trust_remote_code=True).eval().cuda()processor = AutoImageProcessor.from_pretrained(MODEL_HUB, trust_remote_code=True)
# TODO: you need to modify the path hereVIDEO_FRAMES_PATH = "YOUR_VIDEO_FRAMES_PATH"
video = os.listdir(VIDEO_FRAMES_PATH)video.sort()video = [Image.open(osp.join(VIDEO_FRAMES_PATH, v)) for v in video]
images = processor(video, return_tensors="pt")["pixel_values"]images = images.unsqueeze(0).cuda()
# image autoencodeimage = images[:, 0]print(image.shape)with torch.no_grad():    # encode    codes = model.encode(image)    # decode    recon = model.decode(codes)
recon = recon.view(-1, *recon.shape[2:])recon_image = processor.postprocess(recon)["pixel_values"][0]recon_image.save("recon_image.png")
# video autoencodeimages = images.view(    -1,    model.config.temporal_downsample_factor,    *images.shape[2:],)
print(images.shape)with torch.no_grad():    # encode    codes = model.encode(images)    # decode    recon = model.decode(codes)
recon = recon.view(-1, *recon.shape[2:])recon_images = processor.postprocess(recon)["pixel_values"]for idx, im in enumerate(recon_images):    im.save(f"recon_video_{idx}.png")

import osimport os.path as osp from PIL import Imageimport torchfrom transformers import AutoModel, AutoImageProcessor MODEL_HUB = "BAAI/Emu3-VisionTokenizer" model = AutoModel.from_pretrained(MODEL_HUB, trust_remote_code=True).eval().cuda()processor = AutoImageProcessor.from_pretrained(MODEL_HUB, trust_remote_code=True) # TODO: you need to modify the path hereVIDEO_FRAMES_PATH = "YOUR_VIDEO_FRAMES_PATH" video = os.listdir(VIDEO_FRAMES_PATH)video.sort()video = [Image.open(osp.join(VIDEO_FRAMES_PATH, v)) for v in video] images = processor(video, return_tensors="pt")["pixel_values"]images = images.unsqueeze(0).cuda() # image autoencodeimage = images[:, 0]print(image.shape)with torch.no_grad(): # encode codes = model.encode(image) # decode recon = model.decode(codes) recon = recon.view(-1, *recon.shape[2:])recon_image = processor.postprocess(recon)["pixel_values"][0]recon_image.save("recon_image.png") # video autoencodeimages = images.view( -1, model.config.temporal_downsample_factor, *images.shape[2:],) print(images.shape)with torch.no_grad(): # encode codes = model.encode(images) # decode recon = model.decode(codes) recon = recon.view(-1, *recon.shape[2:])recon_images = processor.postprocess(recon)["pixel_values"]for idx, im in enumerate(recon_images): im.save(f"recon_video_{idx}.png")