nvidia/diar_streaming_sortformer_4spk-v2

automatic-speech-recognitionnemoUpstream checksumsnemoggufspeaker-diarizationspeaker-recognitionspeechaudio

License: cc-by-4.0
Size: not mirrored yet
Downloads in the last 30 days: 0 / 30 days

No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.

Cataloged from nvidia/diar_streaming_sortformer_4spk-v2

All models by nvidia →

A new version of streaming Sortformer v2.1 has been released, providing greater robustness for meeting speech.

Streaming Sortformer Diarizer 4spk v2

img { display: inline; }

This model is a streaming version of Sortformer diarizer. Sortformer[1] is a novel end-to-end neural model for speaker diarization, trained with unconventional objectives compared to existing end-to-end diarization models.

[Image blocked: No description]

Streaming Sortformer[2] employs an Arrival-Order Speaker Cache (AOSC) to store frame-level acoustic embeddings of previously observed speakers.

[Image blocked: No description]

Sortformer resolves permutation problem in diarization following the arrival-time order of the speech segments from each speaker.

This speaker diarization model can be used to enable the NeMo Voice Agent to recognize speakers in conversations. See the NeMo Voice Agent and the YAML configuration for more details.

Discover more from NVIDIA:

For documentation, deployment guides, enterprise-ready APIs, and the latest open models—including Nemotron and other cutting-edge speech, translation, and generative AI—visit the NVIDIA Developer Portal at developer.nvidia.com. Join the community to access tools, support, and resources to accelerate your development with NVIDIA’s NeMo, Riva, NIM, and foundation models.

Explore more from NVIDIA:

What is Nemotron?
NVIDIA Developer Nemotron
NVIDIA Riva Speech
NeMo Documentation

Model Architecture

Streaming sortformer employs pre-encode layer in the Fast-Conformer to generate speaker-cache. At each step, speaker cache is filtered to only retain the high-quality speaker cache vectors.

[Image blocked: No description]

Aside from speaker-cache management part, streaming Sortformer follows the architecture of the offline version of Sortformer. Sortformer consists of an L-size (17 layers) NeMo Encoder for Speech Tasks (NEST)[3] which is based on Fast-Conformer[4] encoder. Following that, an 18-layer Transformer[5] encoder with hidden size of 192, and two feedforward layers with 4 sigmoid outputs for each frame input at the top layer. More information can be found in the Streaming Sortformer paper[2].

[Image blocked: No description]

NVIDIA NeMo

To train, fine-tune or perform diarization with Sortformer, you will need to install NVIDIA NeMo[6]. We recommend you install it after you've installed Cython and latest PyTorch version.

Discover more from NVIDIA:

For documentation, deployment guides, enterprise-ready APIs, and the latest open models—including Nemotron and other cutting-edge speech, translation, and generative AI—visit the NVIDIA Developer Portal at developer.nvidia.com [blocked]. Join the community to access tools, support, and resources to accelerate your development with NVIDIA’s NeMo, Riva, NIM, and foundation models.

Explore more from NVIDIA:

What is Nemotron?
NVIDIA Developer Nemotron
NVIDIA Riva Speech
NeMo Documentation

🚀 Quick Start: Run Diarization Now

Here is a short example script that loads the model, runs diarization on a WAV file, and prints the results:

python

from nemo.collections.asr.models import SortformerEncLabelModeldiar_model = SortformerEncLabelModel.from_pretrained("nvidia/diar_streaming_sortformer_4spk-v2")diar_model.eval()
diar_model.sortformer_modules.chunk_len = 340diar_model.sortformer_modules.chunk_right_context = 40diar_model.sortformer_modules.fifo_len = 40diar_model.sortformer_modules.spkcache_update_period = 300
predicted_segments = diar_model.diarize(audio=["/path/to/your/audio.wav"], batch_size=1)
for segment in predicted_segments[0]:    print(segment)

How to Use this Model

There are several ways to use this model. Choose the one that fits your needs.

Run locally with NeMo-Speech.cpp

NeMo-Speech.cpp provides a lightweight native C++ runtime for local speaker diarization. After installing the runtime:

bash

hf download nvidia/diar_streaming_sortformer_4spk-v2 \  diar_streaming_sortformer_4spk-v2.q8_0.gguf \  --local-dir models
nemo-speech diarize meeting.wav \  --model models/diar_streaming_sortformer_4spk-v2.q8_0.gguf

The same model can add word-level speaker tags to a transcription:

See the NeMo-Speech.cpp diarization guide for more usage examples.

NVIDIA NeMo

The model is available for use in the NeMo Framework[6], and can be used as a pre-trained checkpoint for inference or for fine-tuning on another dataset.

Loading the Model

python

from nemo.collections.asr.models import SortformerEncLabelModel
# load model from Hugging Face model card directly (You need a Hugging Face token)diar_model = SortformerEncLabelModel.from_pretrained("nvidia/diar_streaming_sortformer_4spk-v2")
# If you have a downloaded model in "/path/to/diar_streaming_sortformer_4spk-v2.nemo", load model from a downloaded filediar_model = SortformerEncLabelModel.restore_from(restore_path="/path/to/diar_streaming_sortformer_4spk-v2.nemo", map_location='cuda', strict=False)
# switch to inference modediar_model.eval()

Input Format

Input to Sortformer can be an individual audio file:

or a list of paths to audio files:

or a numpy array (single or list):

python

import numpy as npaudio_input = np.random.randn(16000 * 10).astype(np.float32)  # 10 sec at 16kHz# or a list of arraysaudio_input = [audio_array1, audio_array2]diar_model.diarize(audio=audio_input, batch_size=2, sample_rate=16000)

Note: When using numpy arrays, you MUST specify a correct sample_rate in diar_model.diarize() function. Default sample_rate is 16000.

or a jsonl manifest file:

where each line is a dictionary containing the following fields:

yaml

# Example of a line in `multispeaker_manifest.json`{    "audio_filepath": "/path/to/multispeaker_audio1.wav",  # path to the input audio file     "offset": 0, # offset (start) time of the input audio    "duration": 600,  # duration of the audio, can be set to `null` if using NeMo main branch}{    "audio_filepath": "/path/to/multispeaker_audio2.wav",      "offset": 900,    "duration": 580,  }

Setting up Streaming Configuration

Streaming configuration is defined by the following parameters, all measured in 80ms frames:

CHUNK_SIZE: The number of frames in a processing chunk.
RIGHT_CONTEXT: The number of future frames attached after the chunk.
FIFO_SIZE: The number of previous frames attached before the chunk, from the FIFO queue.
UPDATE_PERIOD: The number of frames extracted from the FIFO queue to update the speaker cache.
SPEAKER_CACHE_SIZE: The total number of frames in the speaker cache.

Here are recommended configurations for different scenarios:

Configuration	Latency	RTF	CHUNK_SIZE	RIGHT_CONTEXT	FIFO_SIZE	UPDATE_PERIOD	SPEAKER_CACHE_SIZE
very high latency	30.4s	0.002	340	40	40	300	188
high latency	10.0s	0.005	124	1	124	124	188
low latency	1.04s	0.093	6	7	188	144	188
ultra low latency	0.32s	0.180	3	1	188	144	188

For clarity on the metrics used in the table:

Latency: Refers to Input Buffer Latency, calculated as CHUNK_SIZE + RIGHT_CONTEXT. This value does not include computational processing time.
Real-Time Factor (RTF): Characterizes processing speed, calculated as the time taken to process an audio file divided by its duration. RTF values are measured with a batch size of 1 on an NVIDIA RTX 6000 Ada Generation GPU.

To set streaming configuration, use:

python

diar_model.sortformer_modules.chunk_len = CHUNK_SIZEdiar_model.sortformer_modules.chunk_right_context = RIGHT_CONTEXTdiar_model.sortformer_modules.fifo_len = FIFO_SIZEdiar_model.sortformer_modules.spkcache_update_period = UPDATE_PERIODdiar_model.sortformer_modules.spkcache_len = SPEAKER_CACHE_SIZEdiar_model.sortformer_modules._check_streaming_parameters()

Getting Diarization Results

To perform speaker diarization and get a list of speaker-marked speech segments in the format 'begin_seconds, end_seconds, speaker_index', simply use:

To obtain tensors of speaker activity probabilities, use:

Note that if you are feeding a list of numpy arrays, you MUST provide the sample_rate in integer format.

🔬 For more detailed evaluations (DER)

If you need to perform a comprehensive evaluation and calculate the Diarization Error Rate (DER) across different parameter settings, use the NeMo example script e2e_diarize_speech.py. This script allows you to test the streaming behavior of the model by adjusting key parameters like chunk_len, fifo_len, and spkcache_update_period.

bash

python ${NEMO_ROOT}/examples/speaker_tasks/diarization/neural_diarizer/e2e_diarize_speech.py \    model_path="/path/to/diar_sortformer_4spk_v1.nemo" \    dataset_manifest="/path/to/diarization_manifest.json" \    batch_size=1 \    spkcache_len=188 \    spkcache_update_period=300 \    fifo_len=40 \    chunk_len=340 \    chunk_right_context=40

Input

This model accepts single-channel (mono) audio sampled at 16,000 Hz.

The actual input tensor is a Ns x 1 matrix for each audio clip, where Ns is the number of samples in the time-series signal.
For instance, a 10-second audio clip sampled at 16,000 Hz (mono-channel WAV file) will form a 160,000 x 1 matrix.

Output

The output of the model is an T x S matrix, where:

S is the maximum number of speakers (in this model, S = 4).
T is the total number of frames, including zero-padding. Each frame corresponds to a segment of 0.08 seconds of audio.
Each element of the T x S matrix represents the speaker activity probability in the [0, 1] range. For example, a matrix element a(150, 2) = 0.95 indicates a 95% probability of activity for the second speaker during the time range [12.00, 12.08] seconds.

Train and evaluate Sortformer diarizer using NeMo

Training

Sortformer diarizer models are trained on 8 nodes of 8×NVIDIA Tesla V100 GPUs. We use 90 second long training samples and batch size of 4. The model can be trained using this example script and base config.

Inference

Sortformer diarizer models can be performed with post-processing algorithms using inference example script. If you provide the post-processing YAML configs in post_processing folder to reproduce the optimized post-processing algorithm for each development dataset.

Technical Limitations

The model operates in a streaming mode (online mode).
It can detect a maximum of 4 speakers; performance degrades on recordings with 5 and more speakers.
While the model is designed for long-form audio and can handle recordings that are several hours long, performance may degrade on very long recordings.
The model was trained on publicly available speech datasets, primarily in English. As a result:
- Performance may degrade on non-English speech.
- Performance may also degrade on out-of-domain data, such as recordings in noisy conditions.

Datasets

Sortformer was trained on a combination of 2445 hours of real conversations and 5150 hours or simulated audio mixtures generated by NeMo speech data simulator[7]. All the datasets listed above are based on the same labeling method via RTTM format. A subset of RTTM files used for model training are processed for the speaker diarization model training purposes. Data collection methods vary across individual datasets. For example, the above datasets include phone calls, interviews, web videos, and audiobook recordings. Please refer to the Linguistic Data Consortium (LDC) website or dataset webpage for detailed data collection methods.

Training Datasets (Real conversations)

Fisher English (LDC)
AMI Meeting Corpus
VoxConverse-v0.3
ICSI
AISHELL-4
Third DIHARD Challenge Development (LDC)
2000 NIST Speaker Recognition Evaluation, split1 (LDC)
DiPCo
AliMeeting

Training Datasets (Used to simulate audio mixtures)

2004-2010 NIST Speaker Recognition Evaluation (LDC)
Librispeech

Performance

Evaluation data specifications

Dataset	Number of speakers	Number of Sessions
DIHARD III Eval <=4spk	1-4	219
DIHARD III Eval >=5spk	5-9	40
DIHARD III Eval full	1-9	259
CALLHOME-part2 2spk	2	148
CALLHOME-part2 3spk	3	74
CALLHOME-part2 4spk	4	20
CALLHOME-part2 5spk	5	5
CALLHOME-part2 6spk	6	3
CALLHOME-part2 full	2-6	250
CH109	2	109

Diarization Error Rate (DER)

All evaluations include overlapping speech.
Collar tolerance is 0s for DIHARD III Eval, and 0.25s for CALLHOME-part2 and CH109.
Post-Processing (PP) is optimized on two different held-out dataset splits.
- DIHARD III Dev Optimized Post-Processing for DIHARD III Eval
- CALLHOME-part1 Optimized Post-Processing for CALLHOME-part2 and CH109

Latency	PP	DIHARD III Eval <=4spk	DIHARD III Eval >=5spk	DIHARD III Eval full	CALLHOME-part2 2spk	CALLHOME-part2 3spk	CALLHOME-part2 4spk	CALLHOME-part2 5spk	CALLHOME-part2 6spk	CALLHOME-part2 full	CH109
30.4s	no	14.63	40.74	19.68	6.27	10.27	12.30	19.08	28.09	10.50	5.03
30.4s	yes	13.45	41.40	18.85	5.34	9.22	11.29	18.84	27.29	9.54	4.61
10.0s	no	14.90	41.06	19.96	6.96	11.05	12.93	20.47	28.10	11.21	5.28
10.0s	yes	13.75	41.41	19.10	6.05	9.88	11.72	19.66	27.37	10.15	4.80
1.04s	no	14.49	42.22	19.85	7.51	11.45	13.75	23.22	29.22	11.89	5.37
1.04s	yes	13.24	42.56	18.91	6.57	10.05	12.44	21.68	28.74	10.70	4.88
0.32s	no	14.64	43.47	20.19	8.63	12.91	16.19	29.40	30.60	13.57	6.46
0.32s	yes	13.44	43.73	19.28	6.91	10.45	13.70	27.04	28.58	11.38	5.27

NVIDIA Riva: Deployment

Streaming Sortformer is deployed via NVIDIA RIVA ASR - Speech Recognition with Speaker Diarization NVIDIA Riva, is an accelerated speech AI SDK deployable on-prem, in all clouds, multi-cloud, hybrid, on edge, and embedded. Additionally, Riva provides:

World-class out-of-the-box accuracy for the most common languages with model checkpoints trained on proprietary data with hundreds of thousands of GPU-compute hours
Best in class accuracy with run-time word boosting (e.g., brand and product names) and customization of acoustic model, language model, and inverse text normalization
Streaming speech recognition, Kubernetes compatible scaling, and enterprise-grade support For more information on NVIDIA RIVA, see the list of supported models is here.
Also check out the Riva live demo.

References

[1] Sortformer: Seamless Integration of Speaker Diarization and ASR by Bridging Timestamps and Tokens

[2] Streaming Sortformer: Speaker Cache-Based Online Speaker Diarization with Arrival-Time Ordering

[3] NEST: Self-supervised Fast Conformer as All-purpose Seasoning to Speech Processing Tasks

[4] Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition

[5] Attention is all you need

[6] NVIDIA NeMo Framework

[7] NeMo speech data simulator

Licence

License to use this model is covered by the CC-BY-4.0. By downloading the public and release version of the model, you accept the terms and conditions of the CC-BY-4.0 license.