microsoft/UniRG-CXR

image-text-to-texttransformersUpstream checksumstransformerssafetensorsqwen3_vlimage-text-to-textmultimodalconversational

License: apache-2.0
Size: not mirrored yet
Downloads in the last 30 days: 0 / 30 days

No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.

Cataloged from microsoft/UniRG-CXR

All models by microsoft →

Introduction

We introduce UniRG-CXR, a radiology report generation model that obtains SOTA performance on ReXrank. More details can be found in the paper: Scaling medical imaging report generation with multimodal reinforcement learning

Requirements

We recommend installing the transformers version with python=3.12 used in our experiments and other dependencies with this command:

Quickstart

Below, we provide a some examples to show how to use UniRG-CXR with 🤗 Transformers or vLLM.

Inference with HF Transformers 🤗

Here we show a code snippet to show you how chat with UniRG-CXR using `transformers` and `qwen_vl_utils`:

python

import torchfrom transformers import Qwen3VLForConditionalGeneration, AutoProcessorfrom qwen_vl_utils import process_vision_info
# default: Load the model on the available device(s)model = Qwen3VLForConditionalGeneration.from_pretrained(    "microsoft/UniRG-CXR", dtype=torch.bfloat16, device_map="auto")
# We recommend enabling flash_attention_2 for better acceleration and memory saving.# model = Qwen3VLForConditionalGeneration.from_pretrained(#     "microsoft/UniRG-CXR",#     dtype=torch.bfloat16,#     attn_implementation="flash_attention_2",#     device_map="auto",# )
# You can set min_pixels and max_pixels according to your needs.min_pixels = 262144max_pixels = 262144processor = AutoProcessor.from_pretrained("microsoft/UniRG-CXR", min_pixels=min_pixels, max_pixels=max_pixels)
messages = [    {        "role": "user",        "content": [              {"type": "text", "text": "This is a radiology report generation task. Here is the context:"},            {                "type": "image",                "image": "<input your image path here>",            },              {"type": "text", "text": "Given the image and the context, directly provide the report in the following format:\nFindings: [write the findings] Impression: [write the impression]\nNow write the report in the format above."},
        ],    }]
# Preparation for inferencetext = processor.apply_chat_template(    messages, tokenize=False, add_generation_prompt=True)image_inputs, video_inputs = process_vision_info(messages)inputs = processor(    text=[text],    images=image_inputs,    videos=video_inputs,    padding=True,    return_tensors="pt",)
        inputs = inputs.to(device="cuda")
# Inference: Generation of the outputgenerated_ids = model.generate(**inputs, max_new_tokens=4000)generated_ids_trimmed = [    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)]output_text = processor.batch_decode(    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False)print(output_text)

Inference with vLLM

Here we show an example of how to use UniRG-CXR with vLLM (tested with vllm==0.11.2 and transformers==4.57.1):

python

from vllm import LLM, SamplingParamsfrom transformers import AutoProcessor
min_pixels = 262144max_pixels = 262144processor = AutoProcessor.from_pretrained("microsoft/UniRG-CXR", min_pixels=min_pixels, max_pixels=max_pixels)
llm = LLM(    model="microsoft/UniRG-CXR",    trust_remote_code=True,    dtype="bfloat16",    max_model_len=8192,    tensor_parallel_size=4,    gpu_memory_utilization=0.8,    limit_mm_per_prompt={"image": 1})
# Set up sampling parameterssampling_params = SamplingParams(    temperature=0.0,    max_tokens=4000,)
image_data = []


image_data = ['Your image path']messages = [    {        "role": "user",        "content": [              {"type": "text", "text": "This is a radiology report generation task. Here is the context:"},            {                "type": "image",                "image": image_data[0],            },              {"type": "text", "text": "Given the image and the context, directly provide the report in the following format:\nFindings: [write the findings] Impression: [write the impression]\nNow write the report in the format above."},
        ],    }]
prompt = processor.apply_chat_template(    messages, tokenize=False, add_generation_prompt=True)
if image_data:    mm_prompt = {        "prompt": prompt,        "multi_modal_data": {"image": image_data}    }else:    mm_prompt = {"prompt": prompt}
# Generate responseoutputs = llm.generate([mm_prompt], sampling_params)
# Print the generated responsefor output in outputs:    prompt = output.prompt    generated_text = output.outputs[0].text    print(f"Prompt: {prompt}")    print(f"Generated text: {generated_text}")    print("-" * 50)

Citation

If you find our work helpful, feel free to give us a cite.

@article{liu2026scaling,  title={Scaling medical imaging report generation with multimodal reinforcement learning},  author={Liu, Qianchu and Zhang, Sheng and Qin, Guanghui and Gu, Yu and Jin, Ying and Preston, Sam and Xu, Yanbo and Kiblawi, Sid and Yim, Wen-wai and Ossowski, Tim and others},  journal={arXiv preprint arXiv:2601.17151},  year={2026}}

Notices

Microsoft's Privacy Statement: https://go.microsoft.com/fwlink/?LinkId=521839.

Requirements

We recommend installing the transformers version with python=3.12 used in our experiments and other dependencies with this command:

Quickstart

Below, we provide a some examples to show how to use UniRG-CXR with 🤗 Transformers or vLLM.

Inference with HF Transformers 🤗

Here we show a code snippet to show you how chat with UniRG-CXR using `transformers` and `qwen_vl_utils`:

python

import torchfrom transformers import Qwen3VLForConditionalGeneration, AutoProcessorfrom qwen_vl_utils import process_vision_info
# default: Load the model on the available device(s)model = Qwen3VLForConditionalGeneration.from_pretrained(    "microsoft/UniRG-CXR", dtype=torch.bfloat16, device_map="auto")
# We recommend enabling flash_attention_2 for better acceleration and memory saving.# model = Qwen3VLForConditionalGeneration.from_pretrained(#     "microsoft/UniRG-CXR",#     dtype=torch.bfloat16,#     attn_implementation="flash_attention_2",#     device_map="auto",# )
# You can set min_pixels and max_pixels according to your needs.min_pixels = 262144max_pixels = 262144processor = AutoProcessor.from_pretrained("microsoft/UniRG-CXR", min_pixels=min_pixels, max_pixels=max_pixels)
messages = [    {        "role": "user",        "content": [              {"type": "text", "text": "This is a radiology report generation task. Here is the context:"},            {                "type": "image",                "image": "<input your image path here>",            },              {"type": "text", "text": "Given the image and the context, directly provide the report in the following format:\nFindings: [write the findings] Impression: [write the impression]\nNow write the report in the format above."},
        ],    }]
# Preparation for inferencetext = processor.apply_chat_template(    messages, tokenize=False, add_generation_prompt=True)image_inputs, video_inputs = process_vision_info(messages)inputs = processor(    text=[text],    images=image_inputs,    videos=video_inputs,    padding=True,    return_tensors="pt",)
        inputs = inputs.to(device="cuda")
# Inference: Generation of the outputgenerated_ids = model.generate(**inputs, max_new_tokens=4000)generated_ids_trimmed = [    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)]output_text = processor.batch_decode(    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False)print(output_text)

import torchfrom transformers import Qwen3VLForConditionalGeneration, AutoProcessorfrom qwen_vl_utils import process_vision_info
# default: Load the model on the available device(s)model = Qwen3VLForConditionalGeneration.from_pretrained(    "microsoft/UniRG-CXR", dtype=torch.bfloat16, device_map="auto")
# We recommend enabling flash_attention_2 for better acceleration and memory saving.# model = Qwen3VLForConditionalGeneration.from_pretrained(#     "microsoft/UniRG-CXR",#     dtype=torch.bfloat16,#     attn_implementation="flash_attention_2",#     device_map="auto",# )
# You can set min_pixels and max_pixels according to your needs.min_pixels = 262144max_pixels = 262144processor = AutoProcessor.from_pretrained("microsoft/UniRG-CXR", min_pixels=min_pixels, max_pixels=max_pixels)
messages = [    {        "role": "user",        "content": [              {"type": "text", "text": "This is a radiology report generation task. Here is the context:"},            {                "type": "image",                "image": "<input your image path here>",            },              {"type": "text", "text": "Given the image and the context, directly provide the report in the following format:\nFindings: [write the findings] Impression: [write the impression]\nNow write the report in the format above."},
        ],    }]
# Preparation for inferencetext = processor.apply_chat_template(    messages, tokenize=False, add_generation_prompt=True)image_inputs, video_inputs = process_vision_info(messages)inputs = processor(    text=[text],    images=image_inputs,    videos=video_inputs,    padding=True,    return_tensors="pt",)
        inputs = inputs.to(device="cuda")
# Inference: Generation of the outputgenerated_ids = model.generate(**inputs, max_new_tokens=4000)generated_ids_trimmed = [    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)]output_text = processor.batch_decode(    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False)print(output_text)

Inference with vLLM

Here we show an example of how to use UniRG-CXR with vLLM (tested with vllm==0.11.2 and transformers==4.57.1):

python

from vllm import LLM, SamplingParamsfrom transformers import AutoProcessor
min_pixels = 262144max_pixels = 262144processor = AutoProcessor.from_pretrained("microsoft/UniRG-CXR", min_pixels=min_pixels, max_pixels=max_pixels)
llm = LLM(    model="microsoft/UniRG-CXR",    trust_remote_code=True,    dtype="bfloat16",    max_model_len=8192,    tensor_parallel_size=4,    gpu_memory_utilization=0.8,    limit_mm_per_prompt={"image": 1})
# Set up sampling parameterssampling_params = SamplingParams(    temperature=0.0,    max_tokens=4000,)
image_data = []


image_data = ['Your image path']messages = [    {        "role": "user",        "content": [              {"type": "text", "text": "This is a radiology report generation task. Here is the context:"},            {                "type": "image",                "image": image_data[0],            },              {"type": "text", "text": "Given the image and the context, directly provide the report in the following format:\nFindings: [write the findings] Impression: [write the impression]\nNow write the report in the format above."},
        ],    }]
prompt = processor.apply_chat_template(    messages, tokenize=False, add_generation_prompt=True)
if image_data:    mm_prompt = {        "prompt": prompt,        "multi_modal_data": {"image": image_data}    }else:    mm_prompt = {"prompt": prompt}
# Generate responseoutputs = llm.generate([mm_prompt], sampling_params)
# Print the generated responsefor output in outputs:    prompt = output.prompt    generated_text = output.outputs[0].text    print(f"Prompt: {prompt}")    print(f"Generated text: {generated_text}")    print("-" * 50)

from vllm import LLM, SamplingParamsfrom transformers import AutoProcessor
min_pixels = 262144max_pixels = 262144processor = AutoProcessor.from_pretrained("microsoft/UniRG-CXR", min_pixels=min_pixels, max_pixels=max_pixels)
llm = LLM(    model="microsoft/UniRG-CXR",    trust_remote_code=True,    dtype="bfloat16",    max_model_len=8192,    tensor_parallel_size=4,    gpu_memory_utilization=0.8,    limit_mm_per_prompt={"image": 1})
# Set up sampling parameterssampling_params = SamplingParams(    temperature=0.0,    max_tokens=4000,)
image_data = []


image_data = ['Your image path']messages = [    {        "role": "user",        "content": [              {"type": "text", "text": "This is a radiology report generation task. Here is the context:"},            {                "type": "image",                "image": image_data[0],            },              {"type": "text", "text": "Given the image and the context, directly provide the report in the following format:\nFindings: [write the findings] Impression: [write the impression]\nNow write the report in the format above."},
        ],    }]
prompt = processor.apply_chat_template(    messages, tokenize=False, add_generation_prompt=True)
if image_data:    mm_prompt = {        "prompt": prompt,        "multi_modal_data": {"image": image_data}    }else:    mm_prompt = {"prompt": prompt}
# Generate responseoutputs = llm.generate([mm_prompt], sampling_params)
# Print the generated responsefor output in outputs:    prompt = output.prompt    generated_text = output.outputs[0].text    print(f"Prompt: {prompt}")    print(f"Generated text: {generated_text}")    print("-" * 50)

Citation

If you find our work helpful, feel free to give us a cite.

@article{liu2026scaling,  title={Scaling medical imaging report generation with multimodal reinforcement learning},  author={Liu, Qianchu and Zhang, Sheng and Qin, Guanghui and Gu, Yu and Jin, Ying and Preston, Sam and Xu, Yanbo and Kiblawi, Sid and Yim, Wen-wai and Ossowski, Tim and others},  journal={arXiv preprint arXiv:2601.17151},  year={2026}}

@article{liu2026scaling,  title={Scaling medical imaging report generation with multimodal reinforcement learning},  author={Liu, Qianchu and Zhang, Sheng and Qin, Guanghui and Gu, Yu and Jin, Ying and Preston, Sam and Xu, Yanbo and Kiblawi, Sid and Yim, Wen-wai and Ossowski, Tim and others},  journal={arXiv preprint arXiv:2601.17151},  year={2026}}

import torchfrom transformers import Qwen3VLForConditionalGeneration, AutoProcessorfrom qwen_vl_utils import process_vision_info # default: Load the model on the available device(s)model = Qwen3VLForConditionalGeneration.from_pretrained( "microsoft/UniRG-CXR", dtype=torch.bfloat16, device_map="auto") # We recommend enabling flash_attention_2 for better acceleration and memory saving.# model = Qwen3VLForConditionalGeneration.from_pretrained(# "microsoft/UniRG-CXR",# dtype=torch.bfloat16,# attn_implementation="flash_attention_2",# device_map="auto",# ) # You can set min_pixels and max_pixels according to your needs.min_pixels = 262144max_pixels = 262144processor = AutoProcessor.from_pretrained("microsoft/UniRG-CXR", min_pixels=min_pixels, max_pixels=max_pixels) messages = [ { "role": "user", "content": [ {"type": "text", "text": "This is a radiology report generation task. Here is the context:"}, { "type": "image", "image": "<input your image path here>", }, {"type": "text", "text": "Given the image and the context, directly provide the report in the following format:\nFindings: [write the findings] Impression: [write the impression]\nNow write the report in the format above."}, ], }] # Preparation for inferencetext = processor.apply_chat_template( messages, tokenize=False, add_generation_prompt=True)image_inputs, video_inputs = process_vision_info(messages)inputs = processor( text=[text], images=image_inputs, videos=video_inputs, padding=True, return_tensors="pt",) inputs = inputs.to(device="cuda") # Inference: Generation of the outputgenerated_ids = model.generate(**inputs, max_new_tokens=4000)generated_ids_trimmed = [ out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)]output_text = processor.batch_decode( generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False)print(output_text)