The benchmark include samples containing multiple images resulting in large input context, and some with ground-truth containing large numbers of items to extract resulting in large outputs. We found that the reasoning of small models significantly negatively impact their performances. The reason is that many models ended up falling in repetition loops, hitting the output tokens limit and resulting in failed requests.

Document to Markdown

NuExtract can also convert document images into clean Markdown. Output will be Markdown for text (headers etc), HTML for tables, LaTeX for math and <figure data-type="image" data-id="img_n"><img src="/NM-dev/model_card-A/resolve/main/img_n.png" alt="Detail description of the images"/>

Modern, format-agnostic benchmarks for complex document understanding are limited, so we explored a new evaluation approach. We selected 100 documents with challenging layouts and tables, asked each model to convert them into a structured representation, then used Gemini 3 Flash to compare model outputs against the source document and choose the most accurate result. The rankings aligned with human votes, suggesting this is a promising method for evaluating document-to-Markdown capabilities. More details will be shared in an upcoming technical report. Here are some results:

Using "Markdown-to-structured"

To add other evaluate references, we used our structured extraction benchmark to evaluate models in a two-step fashion: convert the benchmark inputs to Markdown, then use Qwen3.6 27B to perform the structured extraction task on them. Intuitively, it allows to evaluate how models achieve to keep the input document content and layout: good models will allow the "structured extractor" model to perform better scores.

Using NuExtract

Structured extraction

Structured extraction takes as inputs:

1. An input document, which can be text, image, or both;
2. A JSON template describing the information to extract;
3. (Optional) Instructions, allowing to specify expected output formats or values, to provide with the instructions chat template kwarg;
4. (Optional) In-Context Learning (ICL) examples.

Input JSON template

NuExtract uses a input JSON template whose structure is identical to the output JSON. Its leaf values are specify the types of the output JSON leaves. For examples:

{
  "invoice_number": "verbatim-string",
  "invoice_date": "date",
  "total_amount": "number",
  "currency": "currency",
  "line_items": [
    {
      "description": "verbatim-string",
      "item_type": ["electronics", "clothing", "vehicle", "furniture", "other"],
      "quantity": "integer",
      "unit_price": "number",
      "total": "number"
    }
  ]
}

Supported template types include:

• verbatim-string: extract text exactly as it appears in the document;
• string: generic string field, allowing abstraction or light paraphrasing;
• integer: whole number;
• number: integer or decimal number;
• date-time: ISO-8601 date, time or date-time;
• Other specific types such as data, time, country, currency, email and so on. For more details, read the complete types specifications and examples

Template constructors:

• Arrays, for example ["string"];
• Enums, for example ["yes", "no", "maybe"];
• Multi-enums (multiple possible values), for example [["A", "B", "C"]].

If the model does not find relevant information for a field, it returns null or [].

Converting JSON schema / Pydantic models to NuExtract template

Our Python SDK (pip install numind) offers a method to convert JSON schemas to NuExtract templates:

from typing import Literal

from pydantic import Field, BaseModel
from numind.nuextract_utils import convert_json_schema_to_nuextract_template


class HotelBooking(BaseModel):
    city: str
    check_in_date: str = Field(description="date")
    check_out_date: str = Field(description="date")
    number_of_guests: int
    room_type: Literal["single", "double", "suite"]


template, dropped_branches = convert_json_schema_to_nuextract_template(
    HotelBooking.model_json_schema()
)

# {'check_in_date': 'date', 'check_out_date': 'date', 'city': 'string', 'number_of_guests': 'integer', 'room_type': ['single', 'double', 'suite']}

Document-to-Markdown

NuExtract can also convert document images into clean Markdown. Output will be markdown for text (headers etc), html for tables, latex for mat and <figure data-type="image" data-id="img_n"><img src="/NM-dev/model_card-A/resolve/main/img_n.png" alt="Detail description of the images"/>

Markdown example:

<figure data-type="image" data-id="img_1">
  <img src="img_1.png" alt="Logo of Mobilier 2000 with contact information: Tél.: (418) 275-4232, 1654, boul. Marcotte, Roberval (Qc) G8H 2P2"/>
</figure>

# COMMANDE
**NUMÉRO 72259**

1

**Vendu à**
TREMBLAY ERIC
ERIC TREMBLAY
348 BOUL. DE L'ANSE
ROBERVAL
G8H 1Y9

**Livré à**
TREMBLAY ERIC
ERIC TREMBLAY
348 BOUL. DE L'ANSE
ROBERVAL
G8H 1Y9

<table>
  <thead>
    <tr>
      <th># CLIENT</th>
      <th>EXPÉDITEUR</th>
      <th>TERME DE CRÉDIT</th>
      <th>DATE</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>2753133</td>
      <td>Notre camion</td>
      <td>à la livraison</td>
      <td>22/06/2023</td>
    </tr>
  </tbody>
</table>

<table>
  <thead>
    <tr>
      <th>NOM DU VENDEUR</th>
      <th>VOTRE ÉCONOMIE !</th>
      <th># COMMANDE</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Éric</td>
      <td>0.00</td>
      <td></td>
    </tr>
  </tbody>
</table>

Reasoning and non-reasoning modes

NuExtract supports both reasoning and non-reasoning inference.

Non-thinking mode

Use this for fast and deterministic extraction or Markdown conversion.

enable_thinking = False
temperature = 0.2

Thinking mode

Use this for difficult documents, complex layouts, ambiguous fields, or cases where the document structure requires additional reasoning.

enable_thinking = True
temperature = 0.6

For production extraction workloads, we recommend starting with non-reasoning mode and enabling reasoning only for difficult examples.

vLLM deployment

NuExtract can be served with vLLM using an OpenAI-compatible API.

vllm serve numind/NuExtract3 \
  --trust-remote-code \
  --limit-mm-per-prompt '{"image": 99, "video": 0}' \
  --chat-template-content-format openai \
  --generation-config vllm \
  --max-model-len 131072 \
  --speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'

Multi Token Prediction

--speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'

MTP can improve decoding throughput without changing the OpenAI-compatible request payload. You can tune num_speculative_tokens for your hardware and workload, or remove --speculative-config if your vLLM version or environment does not support this speculative decoding method.

If you encounter memory issues, reduce the maximum model length and the maximum number of images:

vllm serve numind/NuExtract-3 \
  --trust-remote-code \
  --limit-mm-per-prompt '{"image": 6, "video": 0}' \
  --chat-template-content-format openai \
  --generation-config vllm \
  --max-model-len 16384 \
  --speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'

vLLM inference: structured extraction: text

import json
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
)

template = {
    "store": "verbatim-string",
    "date": "date-time",
    "total": "number",
    "currency": ["USD", "EUR", "GBP", "JPY", "Other"],
    "items": [
        {
            "name": "verbatim-string",
            "price": "number"
        }
    ]
}

response = client.chat.completions.create(
    model="numind/NuExtract3",
    temperature=0.2,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Yesterday I bought apples and coffee at Trader Joe's for a total of $12.40."
                }
            ],
        }
    ],
    extra_body={
        "chat_template_kwargs": {
            "template": json.dumps(template),
            "instructions": "Specify the time for the `date` entry only if it is present, otherwise only output the date component.",
            "enable_thinking": False
        }
    }
)

print(response.choices[0].message.content)

Example output:

{
  "store": "Trader Joe's",
  "date": null,
  "total": 12.40,
  "currency": "USD",
  "items": [
    {
      "name": "apples",
      "price": null
    },
    {
      "name": "coffee",
      "price": null
    }
  ]
}

vLLM inference: structured extraction: image

import json
import base64
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
)

def encode_image(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode("utf-8")

image_base64 = encode_image("receipt.png")
data_url = f"data:image/png;base64,{image_base64}"

template = {
    "store": "verbatim-string",
    "date": "date-time",
    "total": "number",
    "payment_method": "verbatim-string"
}

response = client.chat.completions.create(
    model="numind/NuExtract3",
    temperature=0.2,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {"url": data_url}
                }
            ],
        }
    ],
    extra_body={
        "chat_template_kwargs": {
            "template": json.dumps(template, indent=4),
            "enable_thinking": False
        }
    }
)

print(response.choices[0].message.content)

Example output:

{
  "store": "Trader Joe's",
  "date": "2025-04-12",
  "total": 42.85,
  "payment_method": "Visa"
}

Multiple page PDF

import base64
import json

import fitz  # pip install pymupdf
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
)

def pdf_to_png_data_urls(pdf_path, dpi=170):
    data_urls = []

    with fitz.open(pdf_path) as doc:
        for page in doc:
            pix = page.get_pixmap(dpi=dpi, alpha=False)
            png_bytes = pix.tobytes("png")
            png_base64 = base64.b64encode(png_bytes).decode("utf-8")
            data_urls.append(f"data:image/png;base64,{png_base64}")

    return data_urls

data_urls = pdf_to_png_data_urls("invoice.pdf", dpi=170)

template = {
    "invoice_number": "verbatim-string",
    "invoice_date": "date",
    "total": "number",
    "currency": "currency",
    "line_items": [
        {
            "description": "verbatim-string",
            "quantity": "number",
            "unit_price": "number",
            "total": "number"
        }
    ]
}

response = client.chat.completions.create(
    model="numind/NuExtract3",
    temperature=0.2,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {"url": data_url}
                }
                for data_url in data_urls
            ],
        }
    ],
    extra_body={
        "chat_template_kwargs": {
            "template": json.dumps(template, indent=4),
            "enable_thinking": False
        }
    }
)

print(response.choices[0].message.content)

vLLM inference: document-to-Markdown

For Markdown OCR, use mode="markdown" or mode="content" without a template.

import base64
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
)

def encode_image(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode("utf-8")

image_base64 = encode_image("document.png")
data_url = f"data:image/png;base64,{image_base64}"

response = client.chat.completions.create(
    model="numind/NuExtract3",
    temperature=1,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {"url": data_url}
                }
            ],
        }
    ],
    extra_body={
        "chat_template_kwargs": {
            "mode": "markdown",
            "enable_thinking": False
        }
    }
)

print(response.choices[0].message.content)

vLLM inference: reasoning mode

response = client.chat.completions.create(
    model="numind/NuExtract3",
    temperature=1,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {"url": data_url}
                }
            ],
        }
    ],
    extra_body={
        "chat_template_kwargs": {
            "mode": "markdown",
            "enable_thinking": True
        }
    }
)

result = response.choices[0].message.content

if "</think>" in result:
    reasoning, answer = [part.strip() for part in result.split("</think>")]
else:
    reasoning, answer = None, result

print(answer)

In-context examples for extraction

Examples are especially useful when the desired formatting is ambiguous or when the schema requires task-specific conventions. Examples can be provided by using developer messages, for which all items of the contents except the last one are the input, and the last one is the expected output.

import json
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
)

template = {
    "names": ["string"]
}

response = client.chat.completions.create(
    model="numind/NuExtract3",
    temperature=0.2,
    messages=[
        {
            "role": "developer",
            "content": [
                {
                    "type": "text",
                    "text": "Stephen is the manager at Susan's store.",
                },
                {
                    "type": "text",
                    "text": "{\"names\": [\"-STEPHEN-\", \"-SUSAN-\"]}",
                }
            ],
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "John went to the restaurant with Mary. James went to the cinema."
                }
            ],
        }
    ],
    extra_body={
        "chat_template_kwargs": {
            "template": json.dumps(template, indent=4),
            "enable_thinking": False
        }
    }
)

print(response.choices[0].message.content)

Example output:

{
  "names": ["-JOHN-", "-MARY-", "-JAMES-"]
}

vLLM inference: template generation

NuExtract can generate an extraction template from a natural language description.

from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
)

response = client.chat.completions.create(
    model="numind/NuExtract3",
    temperature=0.2,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "I want to extract the key details from a rental contract."
                }
            ],
        }
    ],
    extra_body={
        "chat_template_kwargs": {
            "mode": "template-generation"
        }
    }
)

print(response.choices[0].message.content)

Example output:

{
  "contract_title": "verbatim-string",
  "landlord": "verbatim-string",
  "tenant": "verbatim-string",
  "property_address": "verbatim-string",
  "start_date": "date-time",
  "end_date": "date-time",
  "monthly_rent": "number",
  "currency": "verbatim-string",
  "deposit": "number",
  "signatories": ["verbatim-string"]
}

Curl examples

The following examples assume that vLLM is running locally on port 8000. They use jq to build valid JSON request bodies without manually escaping the image data or template string.

Single image structured extraction

API_KEY="EMPTY"
IMAGE_BASE64_FILE=$(mktemp)
REQUEST_BODY_FILE=$(mktemp)

base64 < receipt.png | tr -d '\n' > "$IMAGE_BASE64_FILE"

TEMPLATE=$(cat <<'JSON'
{
  "store": "verbatim-string",
  "date": "date-time",
  "total": "number",
  "payment_method": "verbatim-string"
}
JSON
)

jq -n \
  --rawfile image_base64 "$IMAGE_BASE64_FILE" \
  --arg template "$TEMPLATE" \
  '{
    model: "numind/NuExtract3",
    temperature: 0.6,
    messages: [
      {
        role: "user",
        content: [
          {
            type: "image_url",
            image_url: {url: ("data:image/png;base64," + $image_base64)}
          }
        ]
      }
    ],
    chat_template_kwargs: {
      template: $template,
      enable_thinking: false
    }
  }' > "$REQUEST_BODY_FILE"

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $API_KEY" \
  --data-binary "@$REQUEST_BODY_FILE"

rm "$IMAGE_BASE64_FILE" "$REQUEST_BODY_FILE"

Single image content extraction

API_KEY="EMPTY"
IMAGE_BASE64_FILE=$(mktemp)
REQUEST_BODY_FILE=$(mktemp)

base64 < document.png | tr -d '\n' > "$IMAGE_BASE64_FILE"

jq -n \
  --rawfile image_base64 "$IMAGE_BASE64_FILE" \
  '{
    model: "numind/NuExtract3",
    temperature: 0.6,
    messages: [
      {
        role: "user",
        content: [
          {
            type: "image_url",
            image_url: {url: ("data:image/png;base64," + $image_base64)}
          }
        ]
      }
    ],
    chat_template_kwargs: {
      mode: "content",
      enable_thinking: false
    }
  }' > "$REQUEST_BODY_FILE"

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $API_KEY" \
  --data-binary "@$REQUEST_BODY_FILE"

rm "$IMAGE_BASE64_FILE" "$REQUEST_BODY_FILE"

Transformers example

import json

import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "numind/NuExtract3"

processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
).eval()

def run_nuextract(messages, **chat_template_kwargs):
    inputs = processor.apply_chat_template(
        messages,
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
        **chat_template_kwargs,
    ).to(model.device)

    with torch.inference_mode():
        generated_ids = model.generate(
            **inputs,
            max_new_tokens=4096,
            do_sample=False,
        )

    generated_ids = generated_ids[:, inputs.input_ids.shape[1]:]
    return processor.batch_decode(
        generated_ids,
        skip_special_tokens=True,
        clean_up_tokenization_spaces=False,
    )[0].strip()

# Single image structured extraction
receipt_image = Image.open("receipt.png").convert("RGB")
receipt_messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": receipt_image,
            }
        ],
    }
]

template = {
    "store": "verbatim-string",
    "date": "date-time",
    "total": "number",
    "payment_method": "verbatim-string"
}

structured_output = run_nuextract(
    receipt_messages,
    template=json.dumps(template, indent=4),
    enable_thinking=False,
)
print(structured_output)

# Single image content extraction
document_image = Image.open("document.png").convert("RGB")
document_messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": document_image,
            }
        ],
    }
]

content_output = run_nuextract(
    document_messages,
    mode="content",
    enable_thinking=False,
)
print(content_output)

Special thanks to the Lambda.ai team for the compute that made this project a success.

Citation

If you use NuExtract, please cite NuMind and link to the model page.

@misc{nuextract3,
  title = {NuExtract3},
  author = {NuMind},
  year = {2026},
  url = {https://nuextract.ai/}
}