How to Evaluate TTS Quality: MOS Scores, Benchmarks, and Testing

Choosing the best TTS depends on the use case and business requirements. However, each benchmark offers a metric or tool to evaluate the naturalness of audio data, i.e., audio quality. Subjective listening tests remain the gold standard in the industry, but they are slow, expensive, easy to manipulate, and not reproducible. Automated proxies exist, each with trade-offs in accuracy and scope.

This guide covers the metrics that matter for TTS evaluation, from perceptual quality scores (MOS, MUSHRA) to automated estimators (UTMOS, PESQ, POLQA), speech-accuracy measures (WER, MCD), and runtime performance benchmarks (FTTS, RTF, memory). Every number cited below includes its source.

What Is a MOS Score?

The Mean Opinion Score (MOS) is a subjective quality rating defined by ITU-T Recommendation P.800 (1996). Listeners rate speech samples on a 5-point scale:

Modern neural TTS engines typically score between 4.0 and 4.5 MOS. Human speech recorded in studio conditions scores 4.5 to 4.8 (the gap below 5.0 reflects listener variability, not actual quality defects). These ranges are widely cited across TTS research literature.

P.800 imposes strict test conditions. Listeners must not have participated in subjective tests for at least six months. Ambient room noise must stay below 30 dBA. Minimum panel size is specified per test methodology (absolute category rating, degradation category rating, or comparison category rating). These requirements make MOS tests expensive and slow to run, which is why automated proxies are valuable.

5 Steps to Run a MOS Test

To run a standard MOS test, follow these five steps:

1. Select test sentences. Test sentences must be phonetically balanced material. The Harvard Sentences and TIMIT corpus, which include a mix of declarative, interrogative, and exclamatory utterances, are popular choices.

2. Synthesize audio. Generate samples from each TTS system under evaluation using identical text inputs. Normalize loudness to -23 LUFS (EBU R128).

3. Recruit listeners. P.800 requires naive listeners (no prior subjective testing for 6 months). A minimum of 24 listeners reduces confidence-interval width to practical levels.

4. Randomize presentation. Each listener rates samples in a different random order. Include anchor samples (natural speech, low-quality vocoder) to calibrate the scale.

5. Collect and analyze. Compute per-system mean and 95% confidence interval. Report sample size, listener demographics, and playback equipment.

For faster iteration, MUSHRA tests (ITU-R BS.1534) let listeners compare multiple systems on the same screen. MUSHRA uses a 0-100 continuous scale and requires a hidden reference and a low anchor. It produces finer-grained comparisons than MOS but demands more listener training.

Automated TTS Quality Metrics

Subjective tests are the ground truth, but they take weeks and cost thousands of dollars per round. Automated metrics fill the gap for continuous integration, A/B testing, and model development.

UTMOS (Automatic MOS Prediction)

UTMOS is an automatic MOS estimator developed by UTokyo-SaruLab. It won the VoiceMOS Challenge 2022, achieving the highest score on both the main track and the out-of-domain track. The system extracts SSL representations from wav2vec 2.0 and HuBERT, then predicts MOS through a lightweight regression head.

Verified correlation with human MOS ratings: LCC 0.842, SRCC 0.809 (VoiceMOS Challenge 2022 results). These numbers mean UTMOS captures roughly 84% of the variance in human judgments, making it a practical automated proxy. It is reference-free: no clean ground-truth audio is needed, which makes it applicable to zero-shot and cross-lingual TTS.

Limitations: UTMOS was trained on English-language MOS data. Performance on tonal languages (Mandarin, Vietnamese) or heavily accented speech degrades. It also cannot assess factual accuracy or semantic appropriateness of generated speech.

PESQ and POLQA (Reference-Based Metrics)

PESQ (ITU-T P.862, 2001) is a reference-based metric: it compares degraded audio against a clean reference signal and produces a predicted MOS on a 1-to-5 scale. PESQ was originally designed for telephony codec evaluation, not TTS. It works best for measuring signal-level distortions (compression artifacts, packet loss) and is less sensitive to prosodic or stylistic quality differences.

POLQA (ITU-T P.863, 2011; 3rd edition 2018) is the successor to PESQ. It supports super-wideband and fullband audio, handles time-varying degradations better, and includes a narrowband mode for backward compatibility. For modern TTS evaluation, POLQA is the better choice when a reference signal is available.

Both PESQ and POLQA require a reference recording, which limits their use in production TTS pipelines where no ground-truth audio exists. They are most useful during model development, when comparing a synthesized utterance against a target recording from a voice actor.

Word Error Rate (WER)

WER measures intelligibility: what fraction of words does an ASR system misrecognize when transcribing synthesized speech? A WER below 3% on standard test sets indicates high intelligibility. WER is especially useful for catching pronunciation errors, skipped words, and hallucinated phonemes that perceptual scores might average out.

To compute WER, run synthesized audio through a production ASR engine (Whisper, Google Speech-to-Text, or similar) and compare the transcript against the input text using standard edit-distance alignment. Report the ASR engine and model version alongside WER, since different ASR systems produce different baselines.

Quality Metrics Comparison

While automated metrics like UTMOS are best for rapid CI/CD testing, subjective MOS tests still remain a reliable metric for final production go-live. Since they're subjective, MOS tests better run with the target audience.

Next Steps

Evaluating TTS quality requires both perceptual metrics and runtime benchmarks. Picovoice publishes independently verified TTS benchmarks covering FTTS, RTF, memory, and model size across multiple engines. These benchmarks run on standardized hardware with open methodology, making direct comparison straightforward.

To start building with a TTS engine optimized for on-device deployment, explore the Orca documentation or visit the Orca product page.

FAQ