How it works

The analysis starts from the sounds recognised in the audio, before any word is transcribed, so that no language model can correct what the speaker said.

  1. The sounds actually said. A Wav2Vec2 model fine-tuned on phonetic labels (facebook/wav2vec2-lv-60-espeak-cv-ft) recognises sounds straight from the audio.
  2. The sounds expected. The sentence is turned into sounds with the CMU pronouncing dictionary for English, espeak-ng for the other languages, and aligned on the recording. Every expected sound gets a time span and measures of how well it was said: how far it sits from the best candidate on its frames, how much the sentence's likelihood drops when it is removed or replaced by a close sound, its duration and context.
  3. The words to flag. A small model trained on 5,000 utterances rated sound by sound and word by word turns those measures, with the recogniser's internal representation of each sound, into the probability that a rater would reject the word. A word is reported above 0.35.
  4. The four scores. Gradient-boosted trees map 53 measures of the whole recording (how well the sounds were said, pace, pauses, rhythm, pitch, energy) to the raters' four scores. Wrong and half-read sentences were added as examples so that they score low.
  5. Free speech. The same path, on the transcript instead of the text. In English, the trees are averaged with a linear model on the recogniser's internal state over the whole recording.
  6. Other languages. Generic models, trained without the identity of the sounds, and a native reference per language that removes what the recogniser hears differently there.

Every model is a plain JSON file in the package, retrainable with the scripts under benchmarks/, and every figure on this site comes with the script that produced it.

The benchmark

speechocean762: 5,000 English utterances by non-native speakers, half of them children, each rated by five experts on the whole sentence, every word and every sound. 2,500 train the models; the 2,500 others, from different speakers, measure them.

Pearson with the ratersTotalAccuracyFluencyProsody
Sentence known0.790.740.820.81
No text0.710.680.790.78

Words: one in twenty is flagged; the raters had rejected 53 to 54 in 100 of those, and the flags catch about half of the words they reject.

Every figure, with the script that reproduces it: Benchmarks.

References