No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from ibm-granite/granite-4.2-3b-q4-mlx
All models by ibm-granite →Granite 4.2 language models are a family of state-of-the-art open foundation models featuring dense decoder-only architectures. They natively support multilingual capabilities, a wide range of coding tasks, retrieval-augmented generation (RAG), tool usage, structured JSON output, and extended reasoning via a built-in thinking mode.
All models are publicly released under the Apache 2.0 license, allowing free use for both research and commercial purposes. The data curation and training processes were specifically designed for enterprise scenarios and customization, incorporating governance, risk, and compliance (GRC) evaluations alongside IBM's standard data clearance and document quality review procedures.
[!NOTE] This repository contains MLX variants of an IBM Granite base model, converted using mlx-lm for native inference on Apple Silicon (M-series) hardware.
Please reference the base model's full model card here: https://huggingface.co/ibm-granite/granite-4.2-3b
mlx-lm is a Python package that provides fast LLM inference and fine-tuning on Apple Silicon using the MLX framework.
Option 1 — install into your environment (recommended for repeated use):
Option 2 — run ephemerally via uvx (no persistent mlx-lm install required):
Prerequisite:
uvxis part of uv, a fast Python package manager. Install it first by following the uv installation guide or with:
Each quantization variant is published as its own Hugging Face repository:
For example, the variants for granite-4.2-3b are published at:
Run directly from a variant repository (mlx-lm will download weights on first use):
To use a different quantization, substitute the variant suffix, for example -q8-mlx or -bf16-mlx.
You can also download the model and run from a local path:
This repository includes a generation_config.json copied from the base model. It records the recommended temperature and top_p values, but mlx_lm generate does not read them automatically — only eos_token_id is consumed by mlx-lm at inference time. Pass the values explicitly on every invocation using the flags below.
The Granite 4.2 base model recommends:
Without these flags,
mlx_lm generatedefaults to--temp 0.0(greedy / deterministic decoding) and--top-p 1.0.
Example with the recommended values applied:
In Python, always apply the chat template and pre-tokenize before calling generate():
Granite 4.2 supports built-in chain-of-thought reasoning. When thinking is active the model produces its reasoning inside a <think>…</think> block before its final answer.
Thinking is on by default. The chat template appends <think>\n automatically when add_generation_prompt=True, placing the model inside the open reasoning block so it generates content before closing it. Two template parameters let you adjust the behavior:
CLI — thinking is on by default
No extra flags are needed to enable thinking. The model reasons automatically:
CLI — disable thinking for direct answers:
Pass enable_thinking=false via --chat-template-config to skip the reasoning block entirely:
CLI — request low-effort (faster) reasoning:
Enable thinking:
Include <think> block before answer:
Disable thinking
Direct answer with no <think> block:
Low-effort (faster) reasoning:
reasoning_effort controls the depth of the thinking pass. The default is "high" (full-depth reasoning). Set it to "low" for a shorter, faster reasoning pass.
Supported Languages: English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, and Chinese. Users may finetune Granite 4.2 models for languages beyond these languages.
This model is designed to handle general instruction-following tasks and can be integrated into AI assistants across various domains, including business applications.
pip install mlx-lmpip install uv# mlx-lm is downloaded into a temporary environment on first useuvx mlx_lm generate \ --model ibm-granite/granite-4.2-3b-q4-mlx \ --prompt "Why is the sky blue according to science?" \ --temp 1.0 \ --top-p 0.95ibm-granite/granite-4.2-3b-bf16-mlxibm-granite/granite-4.2-3b-q8-mlxibm-granite/granite-4.2-3b-q4-mlxmlx_lm generate \ --model ibm-granite/granite-4.2-3b-q4-mlx \ --prompt "Why is the sky blue according to science?" \ --temp 1.0 \ --top-p 0.95uvx --from huggingface-hub hf download ibm-granite/granite-4.2-3b-q4-mlx \ --local-dir ./granite-4.2-3b-q4-mlx
mlx_lm generate \ --model ./granite-4.2-3b-q4-mlx \ --prompt "Why is the sky blue according to science?" \ --temp 1.0 \ --top-p 0.95mlx_lm generate \ --model ibm-granite/granite-4.2-3b-q4-mlx \ --prompt "Explain retrieval-augmented generation in simple terms." \ --temp 1.0 \ --top-p 0.95from mlx_lm import load, generate
model, tokenizer = load("ibm-granite/granite-4.2-3b-q4-mlx")
messages = [{"role": "user", "content": "Explain retrieval-augmented generation in simple terms."}]
# Always pre-tokenize with add_special_tokens=False — see Thinking Mode section.prompt_str = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, enable_thinking=True,)prompt = tokenizer.encode(prompt_str, add_special_tokens=False)
response = generate(model, tokenizer, prompt=prompt, temp=1.0, top_p=0.95, max_tokens=256)print(response)mlx_lm generate \ --model ibm-granite/granite-4.2-3b-q4-mlx \ --prompt "What is the difference between a mutex and a semaphore?" \ --temp 1.0 \ --top-p 0.95 \ --max-tokens 1024mlx_lm generate \ --model ibm-granite/granite-4.2-3b-q4-mlx \ --prompt "Summarize this paragraph in one sentence." \ --chat-template-config '{"enable_thinking": false}' \ --temp 1.0 \ --top-p 0.95mlx_lm generate \ --model ibm-granite/granite-4.2-3b-q4-mlx \ --prompt "What is the difference between a mutex and a semaphore?" \ --chat-template-config '{"reasoning_effort": "low"}' \ --temp 1.0 \ --top-p 0.95 \ --max-tokens 1024from mlx_lm import load, generate
model, tokenizer = load( "ibm-granite/granite-4.2-3b-q4-mlx")
messages = [{"role": "user", "content": "What is the difference between a mutex and a semaphore?"}]
# enable_thinking=True: the chat template appends <think>\n so the model reasons# before closing the block with </think> and producing its final answer.# True is the default for Granite 4.2, but passing it explicitly is good practice.## IMPORTANT: always pre-tokenize with add_special_tokens=False.# Passing a plain string to generate() triggers re-encoding with add_special_tokens=True,# which prepends a spurious BOS token and collapses the thinking block to empty.prompt_str = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, enable_thinking=True, # show Think block)prompt = tokenizer.encode(prompt_str, add_special_tokens=False)
response = generate(model, tokenizer, prompt=prompt, temp=1.0, top_p=0.95, max_tokens=1024)print(response)from mlx_lm import load, generate
model, tokenizer = load( "ibm-granite/granite-4.2-3b-q4-mlx")
messages = [{"role": "user", "content": "Summarize this paragraph in one sentence."}]
prompt_str = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, enable_thinking=False, # skip the <think> block entirely)prompt = tokenizer.encode(prompt_str, add_special_tokens=False)
response = generate(model, tokenizer, prompt=prompt, temp=1.0, top_p=0.95, max_tokens=256)print(response)# reasoning_effort="low" → shorter, faster reasoning pass# reasoning_effort="high" → full-depth reasoning (default; same as omitting the parameter)prompt_str = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, reasoning_effort="low",)prompt = tokenizer.encode(prompt_str, add_special_tokens=False)
response = generate(model, tokenizer, prompt=prompt, temp=1.0, top_p=0.95, max_tokens=512)print(response)