No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from RedHatAI/whisper-large-v3-turbo-FP8-dynamic
All models by RedHatAI →Quantized version of openai/whisper-large-v3-turbo.
This model was obtained by quantizing the weights of openai/whisper-large-v3-turbo to FP8 data type, ready for inference with vLLM >= 0.5.2.
This model can be deployed efficiently using the vLLM backend, as shown in the example below.
vLLM also supports OpenAI-compatible serving. See the documentation for more details.
This model was created with llm-compressor by running the code snippet below.
The model was evaluated on LibriSpeech and Fleurs datasets using lmms-eval, via the following commands:
Librispeech:
Fleurs:
from vllm.assets.audio import AudioAssetfrom vllm import LLM, SamplingParams
# prepare modelllm = LLM( model="neuralmagic/whisper-large-v3-turbo-FP8-Dynamic", max_model_len=448, max_num_seqs=400, limit_mm_per_prompt={"audio": 1},)
# prepare inputsinputs = { # Test explicit encoder/decoder prompt "encoder_prompt": { "prompt": "", "multi_modal_data": { "audio": AudioAsset("winning_call").audio_and_sample_rate, }, }, "decoder_prompt": "<|startoftranscript|>",}
# generate responseprint("========== SAMPLE GENERATION ==============")outputs = llm.generate(inputs, SamplingParams(temperature=0.0, max_tokens=64))print(f"PROMPT : {outputs[0].prompt}")print(f"RESPONSE: {outputs[0].outputs[0].text}")print("==========================================")python quantize.py \ --model_path openai/whisper-large-v3-turbo \ --quant_path output_dir/whisper-large-v3-turbo-FP8-Dynamicimport argparseimport torchimport osfrom datasets import load_datasetfrom transformers import WhisperProcessorfrom llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifierfrom llmcompressor.transformers.tracing import TraceableWhisperForConditionalGenerationfrom compressed_tensors.quantization import QuantizationType
# --- Args ---parser = argparse.ArgumentParser()parser.add_argument('--model_path', type=str, required=True)parser.add_argument('--quant_path', type=str, required=True)parser.add_argument('--observer', type=str, default="minmax")args = parser.parse_args()
# --- Load Model ---model = TraceableWhisperForConditionalGeneration.from_pretrained( args.model_path, device_map="auto", torch_dtype="auto",)model.config.forced_decoder_ids = Noneprocessor = WhisperProcessor.from_pretrained(args.model_path)
# --- Recipe (FP8 Dynamic) ---recipe = [ QuantizationModifier( targets="Linear", scheme="FP8_DYNAMIC", sequential_targets=["WhisperEncoderLayer", "WhisperDecoderLayer"], ignore=["re:.*lm_head"], )]
# --- Run oneshot ---oneshot( model=model, recipe=recipe, trust_remote_code_model=True,)
# --- Save ---os.makedirs(args.quant_path, exist_ok=True)model.save_pretrained(args.quant_path, save_compressed=True)processor.save_pretrained(args.quant_path)lmms-eval \ --model=whisper_vllm \ --model_args="pretrained=neuralmagic-ent/whisper-large-v3-turbo-FP8-Dynamic" \ --batch_size 64 \ --output_path <output_file_path> \ --tasks librispeechlmms-eval \ --model=whisper_vllm \ --model_args="pretrained=neuralmagic-ent/whisper-large-v3-turbo-FP8-Dynamic" \ --batch_size 64 \ --output_path <output_file_path> \ --tasks fleurs