Evaluating text-to-speech models

During the training time, text-to-speech models optimize for the mean-square error loss (or mean absolute error) between the predicted spectrogram values and the generated ones. Both MSE and MAE encourage the model to minimize the difference between the predicted and target spectrograms. However, since TTS is a one-to-many mapping problem, i.e. the output spectrogram for a given text can be represented in many different ways, the evaluation of the resulting text-to-speech (TTS) models is much more difficult.

Unlike many other computational tasks that can be objectively measured using quantitative metrics, such as accuracy or precision, evaluating TTS relies heavily on subjective human analysis.

One of the most commonly employed evaluation methods for TTS systems is conducting qualitative assessments using mean opinion scores (MOS). MOS is a subjective scoring system that allows human evaluators to rate the perceived quality of synthesized speech on a scale from 1 to 5. These scores are typically gathered through listening tests, where human participants listen to and rate the synthesized speech samples.

One of the main reasons why objective metrics are challenging to develop for TTS evaluation is the subjective nature of speech perception. Human listeners have diverse preferences and sensitivities to various aspects of speech, including pronunciation, intonation, naturalness, and clarity. Capturing these perceptual nuances with a single numerical value is a daunting task. At the same time, the subjectivity of the human evaluation makes it challenging to compare and benchmark different TTS systems.

Furthermore, this kind of evaluation may overlook certain important aspects of speech synthesis, such as naturalness, expressiveness, and emotional impact. These qualities are difficult to quantify objectively but are highly relevant in applications where the synthesized speech needs to convey human-like qualities and evoke appropriate emotional responses.

In summary, evaluating text-to-speech models is a complex task due to the absence of one truly objective metric. The most common evaluation method, mean opinion scores (MOS), relies on subjective human analysis. While MOS provides valuable insights into the quality of synthesized speech, it also introduces variability and subjectivity.