Picovoice WordmarkPicovoice Console
Introduction
Introduction
AndroidC.NETFlutterlink to GoiOSJavaNvidia JetsonLinuxmacOSNodejsPythonRaspberry PiReact NativeRustWebWindows
AndroidC.NETFlutterlink to GoiOSJavaNodejsPythonReact NativeRustWeb
SummaryPicovoice LeopardAmazon TranscribeAzure Speech-to-TextGoogle ASRGoogle ASR (Enhanced)IBM Watson Speech-to-Text
FAQ
Introduction
AndroidC.NETFlutterlink to GoiOSJavaNodejsPythonReact NativeRustWeb
AndroidC.NETFlutterlink to GoiOSJavaNodejsPythonReact NativeRustWeb
FAQ
Introduction
AndroidCiOSLinuxmacOSPythonWebWindows
AndroidCiOSPythonWeb
SummaryOctopus Speech-to-IndexGoogle Speech-to-TextMozilla DeepSpeech
FAQ
Introduction
AndroidAngularArduinoBeagleBoneCChrome.NETEdgeFirefoxFlutterlink to GoiOSJavaNvidia JetsonLinuxmacOSMicrocontrollerNodejsPythonRaspberry PiReactReact NativeRustSafariUnityVueWebWindows
AndroidAngularC.NETFlutterlink to GoiOSJavaMicrocontrollerNodejsPythonReactReact NativeRustUnityVueWeb
SummaryPorcupineSnowboyPocketSphinx
Wake Word TipsFAQ
Introduction
AndroidAngularBeagleBoneCChrome.NETEdgeFirefoxFlutterlink to GoiOSJavaNvidia JetsonlinuxmacOSNodejsPythonRaspberry PiReactReact NativeRustSafariUnityVueWebWindows
AndroidAngularC.NETFlutterlink to GoiOSJavaNodejsPythonReactReact NativeRustUnityVueWeb
SummaryPicovoice RhinoGoogle DialogflowAmazon LexIBM WatsonMicrosoft LUIS
Expression SyntaxFAQ
Introduction
AndroidBeagleboneCiOSNvidia JetsonLinuxmacOSPythonRaspberry PiRustWebWindows
AndroidCiOSPythonRustWeb
SummaryPicovoice CobraWebRTC VAD
FAQ
Introduction
AndroidCiOSPythonWeb
AndroidCiOSPythonWeb
SummaryPicovoice KoalaMozilla RNNoise
Introduction
AndroidAngularArduinoBeagleBoneC.NETFlutterlink to GoiOSJavaNvidia JetsonMicrocontrollerNodejsPythonRaspberry PiReactReact NativeRustUnityVueWeb
AndroidAngularCMicrocontroller.NETFlutterlink to GoiOSJavaNodejsPythonReactReact NativeRustUnityVueWeb
Picovoice SDK - FAQ
IntroductionSTM32F407G-DISC1 (Arm Cortex-M4)STM32F411E-DISCO (Arm Cortex-M4)STM32F769I-DISCO (Arm Cortex-M7)IMXRT1050-EVKB (Arm Cortex-M7)
FAQGlossary

Noise Suppression Benchmark


This benchmark evaluates how Picovoice Koala compares with the popular Mozilla RNNoise noise suppression engine. Both Koala and RNNoise are lightweight platform-independent SDKs for streams of audio.

Methodology

Noisy Speech Corpus

We consider the synthetic test set of the first installment of the Microsoft DNS Challenge at Interspeech 2020, consisting of 150 noisy test files and their clean reference files. The original data is mixed at a range of various signal-to-noise ratio (SNR) levels. Furthermore, we investigate the performance at specific SNRs by separating the speech from noise and mix them back together at a custom SNR.

Metrics

Short-Term Objective Intelligibility

The performance of a Noise Suppression engine can be measured in multiple ways including Mean Opinion Score (MOS) in listening experiments, as well as objective approximations of MOS such as POLQA or PESQ. In order to make the benchmark as easily reproducible as possible, we select the Short Term Objective Intelligibility (STOI) metric that judges the intelligibility on a scale from 0 to 1, where 1 is best.

Real-Time Factor

The real-time factor is the ratio of the pure processing runtime of the Noise Suppression algorithm divided by the length of audio. The smaller this value is, the less resources are required to run the algorithm. For enhancing a stream of audio in real-time, it is important that this factor is well below 1 to avoid buffering while still leaving enough resources for other applications.

Results

Intelligibility score (STOI)

The figure below shows the average performance of each engine on the original pre-mixed dataset.

Noise Suppression performance comparisonNoise Suppression performance comparison

A more detailed view can be obtained by re-mixing the dataset at a specific noise level:

Noise Suppression performance comparison across different SNRsNoise Suppression performance comparison across different SNRs

Real-Time Factor

We measure the run times of both algorithms on an Ubuntu 20.04 machine with Intel CPU(Intel(R) Core(TM) i5-9400F CPU @ 2.90GHz), 64 GB of RAM, and NVMe storage, using a single thread.

For both engines, the real-time factor is independent of the processed data.

Noise Suppression performance comparison across different SNRsNoise Suppression performance comparison across different SNRs

Usage

The code used to create this benchmark is available on GitHub under the permissive Apache 2.0 license. Detailed instructions for benchmarking individual engines are provided in the following documents:

  • Mozilla RNNoise performance
  • Picovoice Koala performance

Was this doc helpful?

Issue with this doc?

Report a GitHub Issue
Noise Suppression Benchmark
  • Methodology
  • Noisy Speech Corpus
  • Metrics
  • Results
  • Intelligibility score (STOI)
  • Real-Time Factor
  • Usage
Platform
  • Leopard Speech-to-Text
  • Cheetah Streaming Speech-to-Text
  • Octopus Speech-to-Index
  • Porcupine Wake Word
  • Rhino Speech-to-Intent
  • Cobra Voice Activity Detection
Resources
  • Docs
  • Console
  • Blog
  • Demos
Sales
  • Pricing
  • Starter Tier
  • Enterprise
Company
  • Careers
Follow Picovoice
  • LinkedIn
  • GitHub
  • Twitter
  • Medium
  • YouTube
  • AngelList
Subscribe to our newsletter
Terms of Use
Privacy Policy
© 2019-2022 Picovoice Inc.