How it works
The analysis starts from the sounds recognised in the audio, before any word is transcribed, so that no language model can correct what the speaker said.
- The sounds actually said. A Wav2Vec2 model fine-tuned on phonetic labels (
facebook/wav2vec2-lv-60-espeak-cv-ft) recognises sounds straight from the audio. - The sounds expected. The sentence is turned into sounds with the CMU pronouncing dictionary for English, espeak-ng for the other languages, and aligned on the recording. Every expected sound gets a time span and measures of how well it was said: how far it sits from the best candidate on its frames, how much the sentence's likelihood drops when it is removed or replaced by a close sound, its duration and context.
- The words to flag. A small model trained on 5,000 utterances rated sound by sound and word by word turns those measures, with the recogniser's internal representation of each sound, into the probability that a rater would reject the word. A word is reported above 0.35.
- The four scores. Gradient-boosted trees map 53 measures of the whole recording (how well the sounds were said, pace, pauses, rhythm, pitch, energy) to the raters' four scores. Wrong and half-read sentences were added as examples so that they score low.
- Free speech. The same path, on the transcript instead of the text. In English, the trees are averaged with a linear model on the recogniser's internal state over the whole recording.
- Other languages. Generic models, trained without the identity of the sounds, and a native reference per language that removes what the recogniser hears differently there.
Every model is a plain JSON file in the package, retrainable with the scripts under benchmarks/, and every figure on this site comes with the script that produced it.
The benchmark
speechocean762: 5,000 English utterances by non-native speakers, half of them children, each rated by five experts on the whole sentence, every word and every sound. 2,500 train the models; the 2,500 others, from different speakers, measure them.
| Pearson with the raters | Total | Accuracy | Fluency | Prosody |
|---|---|---|---|---|
| Sentence known | 0.79 | 0.74 | 0.82 | 0.81 |
| No text | 0.71 | 0.68 | 0.79 | 0.78 |
Words: one in twenty is flagged; the raters had rejected 53 to 54 in 100 of those, and the flags catch about half of the words they reject.
Every figure, with the script that reproduces it: Benchmarks.
References
- wav2vec 2.0, Baevski et al., 2020
- speechocean762, Zhang et al., 2021