No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from litert-community/granite-4.0-h-350m
All models by litert-community →Measured on device (edge-compat, granite-4.0-h-350m-fp16): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · did not run: engine_create_failed (2026-08-24); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 9.5 tok/s · prefill 9 tok/s · TTFT 25.05 s (2026-09-05); Raspberry Pi 5 · LiteRT-LM 0.16.1 · CPU, 4 threads · decode 3.8 tok/s · prefill 8 tok/s · TTFT 30.43 s (2026-09-02). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/granite-4.0-h-350m-fp16/CARD.md
Measured on device (edge-compat, granite-4.0-h-350m-int8): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · did not run: engine_create_failed (2026-08-24); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 11.5 tok/s · prefill 11 tok/s · TTFT 21.39 s (2026-09-05); Pixel 8a · LiteRT-LM 0.16.0 · CPU · decode 8.9 tok/s · prefill 8 tok/s · TTFT 28.01 s (2026-09-05). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/granite-4.0-h-350m-int8/CARD.md
ibm-granite/granite-4.0-h-350m converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15 — the Mamba2 conv/SSM state buffers are bound through 0.15's generalized state binding (ExecutorMetadata); the 0.14 engine cannot bind them. Companion to litert-community/granite-4.0-h-1b.
Granite 4.0-h is IBM's hybrid architecture: 28 Mamba2 (selective-scan) blocks interleaved with 4 grouped-query attention blocks. The Mamba2 blocks carry per-layer conv + SSM recurrent state instead of a growing KV cache, so memory stays nearly flat with context length — only the 4 attention layers keep KV.
For GPU, pick _int8_gpu. It carries the same int8 weights as _int8, re-exported with the Mamba2 selective scan written as rank-≤4 batched matmuls and with prefer_activation_type = "fp32" declared. The two older files are unchanged. It is 45 MB larger because the rewrite pads each short prefill signature up to a full 256-token chunk with a stored constant instead of a PAD op — that is the trade that lets the graph run on a GPU at all.
Pick fp16 for reference-faithful quality (greedy outputs match the PyTorch model token-for-token on our probes), int8 for size and speed. On phones we recommend int8: the CPU runtime unpacks fp16 weights to fp32 in RAM (~3.7 GB peak on iPhone vs ~2.1 GB for int8), so fp16 is best treated as the desktop/quality variant.
The converted graph is verified against the PyTorch reference at the logits level: a float export matches the HF model exactly at every decode position (8-step teacher-forced comparison, per-position max|logit diff| ≤ 3.9e-4, correlation 1.000000, top-1 identical at all positions) — both the chunked prefill path and the single-step decode path (rolled conv window + SSM state recurrence) reproduce the reference math. Both files pass an 8-question sanity gate at 8/8, matching the HF reference, and a prompt-length sweep (12–200 tokens through the engine's real prefill chunking) generates cleanly — with one engine-reuse caveat for int8 noted below.
Unlike the HF generate() default, the bundle does not prepend a BOS token — this matches Granite's official chat template, and at 350M scale it matters: a prepended <|end_of_text|> measurably degrades greedy answers on this model.
Known limitation — engine state reuse, not quantization: if one Engine object is reused across many conversations whose prompts share a growing common prefix, a reply can stop after a few tokens; on the int8 file this appears at chat-templated lengths 33–37, and on _int8_gpu at 37–41 (the folded scan changes the graph, so the band moves with it; a fresh Engine is clean at every length tested). Re-measured 2026-08-11 on litert-lm 0.15.0: a fresh Engine per conversation is clean at every tested length, and so is a reused engine that has not built up that prefix history. The cause is the runtime carrying recurrent state across conversations rather than anything in the weights (LiteRT-LM#3165); the fp16 file tolerates it, the int8 file does not. Ordinary single-conversation use is unaffected — if you sweep prompts programmatically, create a fresh Engine per prompt.
The bundle carries the tokenizer and Granite's full chat template. Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks; the KV budget for the attention layers allows up to 4096 tokens. _fp16 and _int8 are CPU files: their scan graph materialises rank-5 and rank-6 intermediates and every mobile GPU delegate refuses tensors above rank 4. _int8_gpu is the same model with those contractions rewritten as batched matmuls, and it delegates in full on Adreno and on Apple GPUs.
(The int8 Mac row was re-measured with litert-lm benchmark 0.15.0, -d 256 --runs 3, on an otherwise idle machine; the fp16 Mac row and both iPhone rows are the original cold-start single runs. Decode is dominated by the selective-scan running as generic float ops; per-parameter it is slower than a plain transformer of this size.)
granite-4.0-h-350m_int8_gpu.litertlm is the GPU file. The other two are CPU files, and the reason is in the
graph rather than in the runtime: transformers writes the Mamba2 SSD contractions as broadcast-multiply-reduce,
which materialises rank-5 and rank-6 intermediates, and every mobile GPU delegate refuses tensors above rank 4.
Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from
litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the
process high-water mark (VmHWM) sampled during that same run. The two CPU files were gated 2026-08-24 and
_int8_gpu 2026-08-27.
LiteRT-LM has no partial-GPU state. Anything short of full delegation is refused at engine creation, so a row above that says does not run is a refusal, not a slower path.
Both rows n=2 and within ±3% (GPU 486.5 / 484.0 prefill and 48.8 / 47.9 decode; CPU 305.6 / 296.8 and 97.6 / 95.2), 205-token prompt, same handset and binary, CPU taken immediately after the GPU runs with the phone already warm. Engine init is 8.6–9.6 s on GPU and is not counted in TTFT.
The GPU is a prefill win and a decode loss here. Use the GPU backend when you feed long prompts and read
short answers; use CPU when you generate a lot of text. Both backends run the same _int8_gpu file.
The two CPU files are also slow on this phone's CPU. Same binary, prompt and protocol, 2026-09-05, two cold
runs each with the phone at thermal status ≤ 1: _int8 10.6 tok/s prefill / 10.7–11.5 decode at a 2.9 GB peak;
_fp16 9.0–13.7 / 9.5–11.3 at 5.4–6.5 GB. The rank-5/6 scan graph that keeps them off the GPU also falls outside
XNNPACK's fast path, so on Android _int8_gpu is the file for both backends (97.6 tok/s decode on this CPU,
above). The desktop and iPhone CPU figures for _int8 / _fp16 in the table above are unaffected.
Pixel 8a (Tensor G3, Mali-G715, 8 GB), same binary and protocol, 2026-09-07. _int8_gpu delegates in full on this GPU as well (38313 / 38313 ops) and answers the probe, and the order is the one seen on the S26: GPU 180 tok/s prefill / 20.4–20.7 decode with a 37 s engine init and a 4.3 GB peak; CPU 103–111 / 40.1–40.7 with a 3.1 s init and a 1.1 GB peak (two cold runs per backend, thermal status 0 before every run). The _int8 file on the same CPU decodes 7.0–8.9 tok/s (2026-09-05). On an 8 GB-class phone run _int8_gpu on the CPU; this repo's litertlm_manifest.json recommends cpu for both Android device classes on these pairs.
_int8_gpu runs on both backends: 7/8 on an 8-question on-device check over Metal and 7/8 on CPU, with
byte-identical greedy output between them and matching per-question verdicts (the miss is a rhyme
question, not arithmetic or fact). Engine init 13.6 s on Metal, 1.0 s on CPU. That check is a different
harness from the desktop one above and its questions differ, so the two scores are not comparable to each
other.
Conversion script, litert-torch patch (Mamba2 export-cache layer + decode-state continuation + prefill pad guard), and gate harnesses: https://github.com/john-rocky/hf-to-litertlm (granite_work/).
Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.
litert-lm run ./granite-4.0-h-350m_fp16.litertlm --prompt "What is the capital of France? Answer in one sentence."