New Speaker Diarization: 100x Faster, 5x More Accurate

TLDR: 100x more efficient, 5x more accurate

Falcon Speaker Diarization is 100x more computationally efficient than pyannote Speaker Diarization and diarizes speakers 5x more accurately than Google Speech-to-Text. Falcon Speaker Diarization is a transcription-engine-agnostic and language-independent Speaker Diarization engine with no cap on the number of speakers.

What is Speaker Diarization?

Speaker Diarization identifies “who spoke when” by segmenting audio based on voice characteristics and associating the segments with speakers. The industry practice is to embed Speaker Diarization into commercial speech-to-text systems. However, some speech-to-text solutions, such as OpenAI Whisper, do not offer embedded speaker diarization.

Advanced Speaker Diarization Benefits

Falcon is a highly accurate, computationally efficient, and modular Speaker Diarization engine powered by deep learning. What makes Falcon Speaker Diarization unique is:

Works With Any Speech-to-Text Engine

Speaker Diarization embedded in speech-to-text APIs only works with the transcription engines that they are in.

Falcon Speaker Diarization can work with any transcription engine, whether OpenAI Whisper, Google Speech-to-Text, or Amazon Transcribe.

No Limit on Number of Speakers

Most speech-to-text APIs can only perform Speaker Diarization on a limited number of speakers. Some of them require developers to input the number of speakers in advance. It’s not just time-consuming, but almost impossible when transcribing large audio archives.

Falcon Speaker Diarization identifies and tracks an uncapped number of speakers without prior input.

Multilingual Support

Speaker Diarization embedded in speech-to-text APIs performs jointly with the speech-to-text engines, meaning it’s limited to the languages supported by speech-to-text APIs.

Falcon Speaker Diarization can recognize and diarize speakers in multilingual settings i.e., it works even when speakers switch languages mid-conversation.

Intrinsic HIPAA & GDPR Compliance

Speaker Diarization embedded in most speech-to-text APIs requires uploading audio data to cloud servers, exposing sensitive conversations to potential privacy risks and compliance concerns.

Falcon Speaker Diarization processes audio locally, ensuring that sensitive conversations never leave your infrastructure and remain fully compliant with data protection regulations like HIPAA and GDPR.

Structured Output for Speaker-Labeled Transcripts

Falcon Speaker Diarization outputs timestamped speaker segments, which can be aligned with transcripts for structured, readable dialogue. Each segment includes a consistent speaker label, start time, and end time, making it easy to parse, store, or visualize.

Speaker Diarization Performance Metrics & Benchmarks

Speaker diarization engines are benchmarked across three key metrics:

Diarization Accuracy: 5x Better than Google

• Accuracy is measured using diarization error rate (DER) and jaccard error rate (JER). These are metrics that quantify speaker confusion, missed speech, and false alarms by comparing system output against ground truth. Lower error rates indicate better accuracy.
• Falcon Speaker Diarization achieves up to 5x higher accuracy than leading cloud-based diarization APIs like Google Speech-to-Text.
• It demonstrates stable performance across both short and long-form multi-speaker audio.

Diarization Speed: 100x Faster than Pyannote

• Computational speed evaluated using core-hour. It is the time required to process one hour of audio on a single CPU core. Systems are tested on identical hardware to ensure fair comparison.
• Falcon Speaker Diarization Processes speech ~100x faster than pyannote.audio in benchmark core-hour comparisons.
• It is optimized for batch inference, maintaining consistent throughput regardless of input length.

Memory Utilization: 15x Lower RAM Usage

• Memory utilization is measured by tracking peak RAM consumption during processing, which determines deployment viability on resource-constrained devices.
• Falcon Speaker Diarization Uses 15x less memory than pyannote.audio in benchmark comparisons.
• It is engineered for low memory consumption and predictable runtime behavior, even in continuous or embedded use cases.

Get Started with Falcon Speaker Diarization

Your feedback is an essential part of the process. Please create a GitHub issue to report bugs. If you enjoy building with Falcon Speaker Diarization, give it a star to help fellow developers quickly find it.

1f = pvfalcon.create(access_key)
2
3segments = f.process_file(path)

1pv_falcon_t *falcon = NULL;
2pv_falcon_init(
3  access_key,
4  model_path,
5  &falcon);
6
7int32_t num_segments = 0;
8pv_segment_t *segments = NULL;
9pv_falcon_process_file(
10  falcon,
11  path,
12  &num_segments,
13  &segments);

1const f =
2  await FalconWorker
3    .create(accessKey);
4
5const segments =
6  await f.process(pcm);

1Falcon f = new Falcon.Builder()
2  .setAccessKey(accessKey)
3  .build(appContext);
4
5FalconSegment segments =
6  f.processFile(path);

1let f = Falcon(
2  accessKey: accessKey)
3
4let segments =
5  f.processFile(path)

Frequently Asked Questions