The Word Error Rate (WER) score is a common metric used to evaluate the performance of a machine translation or speech recognition system. It measures the number of errors a system produces against a reference transcript of the total words spoken, which provides an overall indication of the system’s accuracy. Generally, a lower WER score implies more accuracy in recognizing speech.
How is WER calculated?
The formula for calculating the WER metric can be summarized as follows,
WER equation (Source: author)
Where,
- S – Total number of words substituted (ex:- System recognizes “dog” as “dot”)
- D – Total number of words deleted/disregarded (ex:- System converts the actual phrase “The brown fox jumps over the dog” to “The fox jumps over the dog.”)
- I – Total number of words inserted by a system that were absent in the reference transcript. (ex:- “The brown fox jumps over the dog” to “The quick brown fox jumps over the lazy dog”)
- N – Total number of words in the reference transcript.
If we consider one of our previous examples, “The brown fox jumps over the dog,” and if a system interprets it as “The fox jumps over the dog,” here we can observe that there’s one deletion error as the system has disregarded the word “brown.” Thereby, the WER score would then be,
WER = (0+1+0)/7
= 0.143
A WER of 0.143 or 14.3% indicates the system recognized 85.7% of the words accurately with one deletion error and no substitution or insertion errors.
How is WER used?
WER is a crucial performance metric to assess the accuracy of automatic speech recognition (ASR) systems. It’s particularly useful to evaluate and compare the efficiency and performance of different ASR systems or track the improvements of a single system over time. It’s a very straightforward metric that provides a quick indication of how well a system is performing in regard to recognizing speech and converting spoken words into text.
WER is simple and treats all words equally. This means it doesn’t consider the context or the importance of a word in a sentence. It also doesn’t differentiate between minor or major words, as it doesn’t understand the meaning change between character differences in a word. Nevertheless, WER remains a widely used metric as it is quite simple and straightforward and an easy way to evaluate ASR systems.
Factors that can influence WER
ASR systems are prone to substitution, insertion, and deletion errors due to certain key factors, which in turn increases the WER score. Listed below are a few of them,
- Training datasets: Datasets used to train the ASR system can often directly influence the WER as they learn to recognize the sound of words through these training datasets. Generally, larger and more diverse datasets are preferred as they can contain a wider range of noise levels and speech patterns.
- Acoustic variability: Varying audio quality through factors such as microphone quality, background noise, pitch, and voice can impact WER. The same words recorded in different scenarios can differentiate the WER score. For example, WER would be much higher for an audio clip recorded in a soccer stadium than in a music studio.
- Speaker variability: Dialects, pronunciations, speed, and voice quality can also impact WER. For example, the WER score might be slightly higher for the pronunciation of English words by a person with a Japanese accent rather than that of an American accent.
- Language complexity: Language variations, vocabulary changes, lexical complexity, and homophones can have varying impacts on the WER score. Languages that emphasize tone for comprehension might have higher WER scores; for example, the Thai language is more tonal than English, which is non-tonal.
Benefits of using WER
- Standardized quantitative metric: WER is a widely accepted quantitative measure for the accuracy of an ASR system. It’s a clear numerical value that can be used to compare different ASR systems.
- Easy to understand: The concept and the calculation of WER is simple and straightforward. It simply measures the number of errors (substitutions, deletions, and insertions) in the interpreted text compared to the reference transcript and produces a score.
Challenges of using WER
- Equal weighting of errors: WER treats all types of errors equally, without taking into consideration their impact on the meaning of the overall transcribed text. Minor errors, such as missing a conjunction, are given the same weight as missing a noun or a verb in a sentence. Errors in crucial words that can change the meaning of a sentence are not highlighted in any way, as all errors are treated equally.
- Not a comprehensive measure: As WER can vary based on external factors such as background noise, accents, audio quality, and crosstalk, it cannot be used as a comprehensive metric that fully demonstrates the capability of an ASR system. Furthermore, issues such as crosstalk cannot be handled by an ASR system, which in turn drastically affects the WER score.
- Not the only metric: While WER is a useful metric to get a quick overview of the performance of an ASR system and track its accuracy over time, it should not be used as the sole metric to evaluate it. It is important to experiment and test the system with a variety of audio clips recorded in varying environmental conditions to fully understand its capabilities and limitations.
Conclusion
WER remains a fundamental metric for evaluating the accuracy of ASR systems. Regardless of its limitations, such as equal weighting of errors and sensitivity to external factors, WER offers a simple, standardized, and widely accepted quantitative measure for evaluating ASR system performance.
Developers and researchers can gain deeper insights into the strengths and weaknesses of ASR systems by understanding the factors influencing WER and combining them with additional evaluation metrics. This ensures continuous improvement and adaptation of ASR systems to diverse real-world scenarios.
