No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from ISTA-DASLab/Qwen3.8-27B-NVFP4-prefiller
All models by ISTA-DASLab →
Prefill and decode do not have to run the same weights. This repo holds prefillers: NVFP4 transformer blocks that are streamed off the SSD one layer at a time, paired to complement low-bit GGUF for decode.
The point is that the two phases want different things. Decode is memory-bound, has to be on device for fast generation and is accelerated by being small. Prefill is compute-bound, so there's little point in compressing it below NVFP4 (compute-native on Blackwell) as long as it doesn't occupy much memory, which we achieve via offloading.
[Image blocked: Accuracy and time-to-first-token against device weight size]
Auxiliary prefillers recover ~50% of the accuracy that aggressive quantization gives up, at close to no extra memory cost. Moreover, past a few thousand tokens, the offloading streaming disappears under compute, so it is not paid for in latency either.
[Image blocked: Prefill and decode paths]
Only the transformer stack's linear projections are disaggregated. The embedding, output head, norms and the gated-DeltaNet convolution/state parameters are shared with decode and come from the GGUF.
The checkpoint is 12.8 GiB and never becomes resident: it is read through a two-slot ring carved out of the output head's own memory, so the ring itself adds zero device memory. Reads overlap the previous layer's compute.
[Image blocked: Per-block read/compute overlap]
One directory of blocks per decode quantization, named for the GGUF it was trained against.
Each prefiller was trained to complemet a specific decode checkpoint.
Needs a Blackwell consumer GPU — GB10 (sm_121) or RTX 50-series (sm_120). The build picks the
right architecture on its own; if you pin one, it must be 120a/121a, because the
block-scaled FP4 MMA is architecture-specific PTX and plain 120/121 or the 120f family
target fail to assemble.
Both halves come from the Hub, so there is nothing to download by hand. Pass the decoder and this repo; the prefill variant follows from the decoder:
A local decoder works the same way — swap -hf ... for -m path/to/decoder.gguf.
The log tells you what it picked:
ODP has non-trivial interaction with state-space models, how they're implmeneted in llama.cpp: multiple forward passes per phase to yield intermediate states, instead of pulling all the needed states from one pass. That means that each prefill can actually run full-model SSD loading multile times unless state-space cache flags are carefully tuned to ones needs.
Each block is a real GGUF holding one layer's projections as GGML_TYPE_NVFP4, so gguf_dump and the viewer above show what is inside. The exact format is closer to vLLM's compressed_tensors (with static FP32 global absmax and dynamic FP8 mircoscales). NVFP4's per-tensor global scale has no slot in ggml's block layout and needs none: a scalar commutes with the matmul, so it rides in the KV as odp.scale.<name> and is applied to the result. llama.cpp's NVFP4 GEMM kernels are not particularly fast either --- the speedups above was obtained with custom FlashInfer kernels integration that was yet too ugly to release. The released for achieves speedups closer to 1.3x instead.
Files use llama.cpp's split-file convention, which is what lets one repo hold every variant:
git clone https://github.com/IST-DASLab/disaggregated-llama.cpp.git llama.cpp && cd llama.cppcmake -B build -DLLAMA_BUILD_BORINGSSL=ON -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Releasecmake --build build -j./build/bin/llama-server -ngl 99 \ -hf unsloth/Qwen3.8-27B-GGUF:IQ1_S \ --odp-blocks ISTA-DASLab/Qwen3.8-27B-disaggregated-NVFP4-prefillodp: taking variant 'IQ1_S' from the decoder (filename)odp: prefill variant 'IQ1_S' from ISTA-DASLab/Qwen3.8-27B-disaggregated-NVFP4-prefill (64 parts)Qwen3.8-27B-ODP-IQ1_S-00001-of-00064.gguf ... -00064-of-00064.ggufQwen3.8-27B-ODP-IQ1_M-00001-of-00064.gguf ...@misc{panferov2026disaggregatedquantizationspecializingllm, title={Disaggregated Quantization: Specializing LLM Prefill and Decode}, author={Andrei Panferov and Maximilian Kleinegger and Sweta Priyadarshi and Tijmen Blankevoort and Dan Alistarh}, year={2026}, eprint={2609.26333}, archivePrefix={arXiv}, primaryClass={cs.LG}, url={https://arxiv.org/abs/2609.26333},}