system_prompt, user_message and assistant_response are placeholders.

We invite you to choose, depending on your use case and requirements, between keeping reasoning traces during multi-turn interactions or keeping only the final assistant response.

Please make sure to use mistral-common as the source of truth

Usage

The model can be used with the following frameworks;

Inference

• vllm (recommended): See below

In addition the community has prepared quantized versions of the model that can be used with the following frameworks (alphabetically sorted):

• llama.cpp: https://huggingface.co/mistralai/Magistral-Small-2506_gguf
• lmstudio (llama.cpp, MLX): https://lmstudio.ai/models/mistralai/magistral-small
• ollama: https://ollama.com/library/magistral
• unsloth (llama.cpp): https://huggingface.co/unsloth/Magistral-Small-2506-GGUF

Training

Fine-tuning is possible with (alphabetically sorted):

• axolotl: https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/magistral
• unsloth: https://docs.unsloth.ai/basics/magistral

Other

Also you can use Magistral with:

• kaggle: https://www.kaggle.com/models/mistral-ai/magistral-small-2506

vLLM (recommended)

We recommend using this model with the vLLM library to implement production-ready inference pipelines.

Installation

Make sure you install the latest vLLM code:

pip install -U vllm \
    --pre \
    --extra-index-url https://wheels.vllm.ai/nightly

Doing so should automatically install mistral_common >= 1.6.0.

To check:

python -c "import mistral_common; print(mistral_common.__version__)"

You can also make use of a ready-to-go docker image or on the docker hub.

Serve model as follows:

vllm serve mistralai/Magistral-Small-2506 --tokenizer_mode mistral --config_format mistral --load_format mistral --tool-call-parser mistral --enable-auto-tool-choice --tensor-parallel-size 2

Ping model as follows:

from openai import OpenAI
from huggingface_hub import hf_hub_download

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

TEMP = 0.7
TOP_P = 0.95
MAX_TOK = 40_960

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

models = client.models.list()
model = models.data[0].id

def load_system_prompt(repo_id: str, filename: str) -> str:
    file_path = hf_hub_download(repo_id=repo_id, filename=filename)
    with open(file_path, "r") as file:
        system_prompt = file.read()
    return system_prompt

SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt")

query = "Write 4 sentences, each with at least 8 words. Now make absolutely sure that every sentence has exactly one word less than the previous sentence."
# or try out other queries
# query = "Exactly how many days ago did the French Revolution start? Today is June 4th, 2025."
# query = "Think about 5 random numbers. Verify if you can combine them with addition, multiplication, subtraction or division to 133"
# query = "If it takes 30 minutes to dry 12 T-shirts in the sun, how long does it take to dry 33 T-shirts?"

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": query}
]
stream = client.chat.completions.create(
  model=model,
  messages=messages,
  stream=True,
  temperature=TEMP,
  top_p=TOP_P,
  max_tokens=MAX_TOK,
)

print("client: Start streaming chat completions...")
printed_content = False

for chunk in stream:
  content = None
  # Check the content is content
  if hasattr(chunk.choices[0].delta, "content"):
    content = chunk.choices[0].delta.content

  if content is not None:
    if not printed_content:
        printed_content = True
        print("\ncontent:", end="", flush=True)
    # Extract and print the content
    print(content, end="", flush=True)

# content:<think>
# Alright, I need to write 4 sentences where each one has at least 8 words and each subsequent sentence has one fewer word than the previous one.
# ...
# Final boxed answer (the four sentences):

# \[
# \boxed{
# \begin{aligned}
# &\text{1. The quick brown fox jumps over lazy dog and yells hello.} \\
# &\text{2. I saw the cat on the stair with my hat.} \\
# &\text{3. The man in the moon came down quickly today.} \\
# &\text{4. A cat sat on the mat today patiently.}
# \end{aligned}
# }
# \]