No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from RedHatAI/granite-4.0-h-tiny-FP8-dynamic
All models by RedHatAI →
Quantized version of ibm-granite/granite-4.0-h-tiny.
This model was obtained by quantizing the weights and activations of ibm-granite/granite-4.0-h-tiny to FP8 data type. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%. Only the weights and activations of the linear operators within transformers blocks of the language model are quantized.
This model was quantized using the llm-compressor library as shown below.
Install specific llm-compression version:
The model was evaluated on the OpenLLM leaderboard task, using lm-evaluation-harness. vLLM was used for all evaluations.
Install vLLM from main:
Openllm V1
Openllm V2
Coding Benchmarks
uv pip install -U git+https://github.com/vllm-project/vllm.git \ --extra-index-url https://wheels.vllm.ai/nightly \ --no-deps \ --no-cache
uv pip install compressed-tensors==0.12.3a20251114 --no-cacheuv pip install --upgrade torchvision --break-system-packages --no-cacheuv pip install cloudpickle msgspec zmq blake3 cachetools prometheus_client fastapi openai openai_harmony pybase64 llguidance diskcache xgrammar lm-format-enforcer partial-json-parser cbor2 einops gguf numba --no-cachevllm serve RedHatAI/granite-4.0-h-tiny-FP8-dynamic --tensor_parallel_size 1from openai import OpenAI
# Modify OpenAI's API key and API base to use vLLM's API server.openai_api_key = "EMPTY"openai_api_base = "http://<your-server-host>:8000/v1"
client = OpenAI( api_key=openai_api_key, base_url=openai_api_base,)
model = "RedHatAI/granite-4.0-h-tiny-FP8-dynamic"
messages = [ {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},]
outputs = client.chat.completions.create( model=model, messages=messages,)
generated_text = outputs.choices[0].message.contentprint(generated_text)uv pip install git+https://github.com/vllm-project/llm-compressor.git@refs/pull/2001/head --no-cacheuv pip install --upgrade torchvision --break-system-packages --no-cachefrom transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifierfrom llmcompressor.utils import dispatch_for_generationfrom llmcompressor.modeling import replace_modules_for_calibrationfrom llmcompressor.modeling.granite4 import pack_3d_experts
MODEL_ID = "ibm-granite/granite-4.0-h-tiny"
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype="auto")tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = replace_modules_for_calibration(model)
ignore_lay = ["lm_head", "re:.*block_sparse_moe.router"]
recipe = QuantizationModifier( targets=["Linear"], scheme="FP8_DYNAMIC", ignore=ignore_lay,)
oneshot(model=model, recipe=recipe)
print("========== SAMPLE GENERATION ==============")dispatch_for_generation(model)input_ids = tokenizer( "Describe Large Language Model", return_tensors="pt").input_ids.to(model.device)output = model.generate(input_ids, max_new_tokens=35)print(tokenizer.decode(output[0]))print("==========================================")
SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-FP8-dynamic"print(f"Saving to {SAVE_DIR}")
model.save_pretrained(SAVE_DIR)tokenizer.save_pretrained(SAVE_DIR)pack_3d_experts(SAVE_DIR)uv pip install -U git+https://github.com/vllm-project/vllm.git \ --extra-index-url https://wheels.vllm.ai/nightly \ --no-deps \ --no-cache
uv pip install compressed-tensors==0.12.3a20251114 --no-cacheuv pip install --upgrade torchvision --break-system-packages --no-cacheuv pip install cloudpickle msgspec zmq blake3 cachetools prometheus_client fastapi openai openai_harmony pybase64 llguidance diskcache xgrammar lm-format-enforcer partial-json-parser cbor2 einops gguf numba --no-cachelm_eval \ --model vllm \ --model_args pretrained="RedHatAI/granite-4.0-h-tiny-FP8-dynamic",dtype=auto,add_bos_token=True,max_model_len=16384,tensor_parallel_size=1,gpu_memory_utilization=0.9,enable_chunked_prefill=True,trust_remote_code=True \ --tasks openllm \ --write_out \ --batch_size auto \ --show_configlm_eval \ --model vllm \ --model_args pretrained="RedHatAI/granite-4.0-h-tiny-FP8-dynamic",dtype=auto,add_bos_token=False,max_model_len=16384,tensor_parallel_size=1,gpu_memory_utilization=0.7,disable_log_stats=True,enable_chunked_prefill=True,trust_remote_code=True \ --tasks leaderboard \ --apply_chat_template \ --fewshot_as_multiturn \ --write_out \ --batch_size auto \ --show_configevalplus.evaluate --model "RedHatAI/granite-4.0-h-tiny-FP8-dynamic" \ --dataset "humaneval" \ --backend vllm \ --tp 1 \ --greedy
evalplus.evaluate --model "RedHatAI/granite-4.0-h-tiny-FP8-dynamic" \ --dataset "mbpp" \ --backend vllm \ --tp 1 \ --greedy