Whisper ACFT β short-window TFLite (LiteRT), tiny/base/small Γ multilingual/.en
TFLite (LiteRT) exports of the FUTO audio-context fine-tuned (ACFT) Whisper checkpoints: fixed 5 s / 10 s / 30 s window graphs split into encode and decode signatures, matching the graph interface of litert-community/whisper-tiny and litert-community/whisper-base. Six models in one repo: tiny, base, small (multilingual) and tiny.en, base.en, small.en (English-only), each in three window lengths.
No retraining β the weights are FUTO's Apache-2.0 ACFT checkpoints (futo-org/whisper-acft method and training; HF checkpoints futo-org/acft-whisper-{tiny,base,small}{,.en}). This repo only converts them: transformers TFWhisperForConditionalGeneration β two-signature graph β converter-time dynamic-range quantization (int8 weights, fp32 activations, tf.lite.Optimize.DEFAULT). Conversion by the LiteRT-LM-Unity project.
Why short windows
Whisper always encodes a 30 s window; for sub-5 s voice commands most encoder work is spent on padding. The encoder conv stack is length-agnostic and the encoder positional table is a fixed sinusoid, so slicing the first N rows of embed_positions yields a shorter window β exactly what whisper.cpp's audio_ctx parameter does at runtime. Stock Whisper decoders are trained on the full 30 s window, so shrinking it puts them out of distribution (repetition loops: stock base at a 5 s window measured CER 10.4 on a sentence clip in our evaluation); the ACFT checkpoints are fine-tuned via self-distillation (L2 on hidden states) to tolerate exactly this.
Repo layout
One folder per model; filenames encode model and window. All files are dynamic-range quantized (int8 weights, fp32 activations).
Window β encode input shape: 5 s = [1, 80, 500], 10 s = [1, 80, 1000], 30 s = [1, 80, 3000].
Integration notes
- Fixed short-window graphs: the encode signature input is
[1, 80, frames] with frames = 500 / 1000 / 3000. The runtime must size the mel window from the encode signature's input shape (compute/pad the log-mel to exactly that frame count) instead of assuming 3000 frames.
- The decode signature input order is
(mask, audio, tokens) β opposite of the stock litert-community exports β bind decode tensors by shape/name rather than position. The decoder is a fixed-length full re-run per step (no KV cache), matching the tiny/base graph interface.
- Reference implementation: the JNI bridge in LiteRT-LM-Unity, which auto-detects 500/1000/3000-frame windows from the encode signature on one code path.
Token ids β multilingual vs .en
- Multilingual (
tiny/, base/, small/): 80 mel bins / vocab 51865 β classic Whisper token layout, same as litert-community tiny/base. Decode prompt e.g. [<|startoftranscript|>=50258, <|lang|>, <|transcribe|>=50359, <|notimestamps|>=50363]. Tokenizer: tokenizer.json from the matching openai/whisper-* repo.
- English-only (
tiny.en/, base.en/, small.en/): 80 mel bins / vocab 51864 β English-only layout: <|startoftranscript|> = 50257, <|endoftext|> = 50256, <|notimestamps|> = 50362; there are no language/task tokens β the decode prompt is [50257, 50362]. Tokenizer: tokenizer.json from the matching openai/whisper-*.en repo.
Validation
Desktop probe of these files (LiteRT interpreter, XNNPACK CPU, greedy decode; CER vs punctuation-normalized references, whitespace removed). Multilingual clips: Korean sentence + English sentence + short Korean voice commands; .en clips: English sentence pair.
tiny
The short windows track the checkpoint's own 30 s behavior (same clips right/wrong at every window; residual errors are tiny's model-capacity errors, present at 30 s too) at ~14x encoder speedup for the 5 s window.
tiny.en
Transcript-identical to the 30 s window on both clips (including the same "and soul" mishearing β a tiny-capacity trait, present at 30 s too) at ~13x encoder speedup for the 5 s window.
base
~11x encoder speedup for the 5 s window; base is the recommended minimum for non-English voice commands.
base.en
small
12/12 exact across all three windows β the 5 s window is character-identical to the 30 s window at ~10x encoder speedup.
small.en
All exact. In the same evaluation pipeline on whisper-base, the ACFT 5 s graph reproduced the stock 30 s transcripts clip-for-clip at ~8x encoder speedup (f32) / ~20x (drq); stock checkpoints at a 5 s window (no ACFT) fall into a repetition loop on sentence-length clips (CER 10.4). Short-window graphs are device-validated on Android arm64 (Snapdragon 865, 0.5β0.8 s end-to-end per command clip for tiny/base 5 s) in the producing project.
Caveats
- The 5 s/10 s graphs hard-truncate longer audio β pick the window that matches your audio length; use the 30 s graph for dictation.
- Decoder is a full-sequence re-run per step (no KV cache) to match the litert-community tiny/base interface β a KV-cached runtime will be substantially faster per token.
- tiny retains its stock model-capacity limits on short noisy/quiet clips; base is the recommended minimum for non-English voice commands.
- Language forcing recommended for short clips on the multilingual models (e.g.
<|ko|> / <|en|>).
Credits and license
- ACFT method and checkpoints: FUTO (
futo-org/acft-whisper-*, Apache-2.0, derived from the MIT openai/whisper-* checkpoints). This repo contains TFLite conversions of those checkpoints, no retraining.
- Conversion and validation: LiteRT-LM-Unity.
- License: Apache-2.0, matching the source ACFT checkpoints.
Related: Korean-fine-tuned ACFT short-window models (tiny/base/medium/large-v3-turbo) live at leuconoe/whisper-acft-ko.