Benchmarks

Every figure on this site, the data it was measured on and the script that produces it. The scripts are in the repository under benchmarks/, and so are the models they write.

The corpus

speechocean762 holds 5,000 English sentences read by 250 Mandarin speakers, half of them children, each rated by five experts on the sentence, every word and every sound. The models are trained on the 2,500 sentences of the train split. Every figure below comes from the other 2,500, read by different speakers, and nothing was tuned on them.

The score

Pearson correlation with the raters, test split.

SystemTotalAccuracyFluencyProsody
Sentence known (1.0)0.7900.7420.8240.807
No text (1.0)0.7080.6810.7870.777
Hand-tuned formula (0.3)0.6330.6030.5950.618

With the sentence known, the score misses the raters by 0.71 points on their 0 to 10 scale on average. Published supervised systems reach 0.74 to 0.80 on the total of this corpus; those figures come from their papers and were not re-run here.

The correlation depends on who speaks. With the sentence: adults 0.85, children of 12 and under 0.67, teenagers 0.46 (12 speakers whose ratings vary little). Without it: adults 0.82, children 0.42.

Mispronounced words

A word counts as mispronounced when the raters gave it less than 5 out of 10, which is the case for 5.4 in 100 words of the test split (15,967 words).

SystemAUCPrecisionRecallWords flagged
Trained model (1.0)0.9240.530.515.1 %
Edit-distance rule (0.3)0.8040.200.7519.9 %

Words not said and speech in excess are built from the texts, since the speakers of the corpus read what they were given. A word added to the text is reported as not said 83 times in 100, and 0.7 in 100 words really read are reported as missing by mistake. A word removed from the text is located as extra speech 63 times in 100, and 2.8 in 100 untouched sentences get a false one.

Other conditions

  • Phone line. The test split through a simulated land line (8 kHz, 300 to 3,400 Hz, G.711): 0.778 with the sentence instead of 0.813, 0.636 without it instead of 0.684, on 500 sentences. A single score moves by 3.5 points on average.
  • Long free speech. Six sentences of one speaker glued into 22 seconds and compared with the mean of their six ratings: 0.74 to 0.77 for 17 adults. Too few speakers for more than an order of magnitude.
  • Other languages. No rated learner corpus exists for them, so they are checked on native speakers. The figures are on Languages. On English, the generic models those languages use give 0.771 on the total and an AUC of 0.911 on the words.

Speed

On a 16-thread CPU, models loaded: an 18-second recording takes 3.3 seconds with --fast, 6.2 seconds with both speech models, and 3 minutes of free speech take 52 seconds. Both models together use 3.6 GB of memory.

Reproducing the figures

The corpus is downloaded from the Hugging Face Hub into ~/.cache/openpronounce/speechocean762/. Extracting the speech model's outputs takes about 30 minutes per split on CPU and 1.4 GB of disk; the rest runs in minutes.

$ python benchmarks/word_model.py --extract train && python benchmarks/word_model.py --extract test
$ python benchmarks/word_model.py --features train && python benchmarks/word_model.py --features test
$ python benchmarks/word_model.py --fit --out openpronounce/models/word_mdd_en.json
$ python benchmarks/scorer.py --features train && python benchmarks/scorer.py --features test
$ python benchmarks/scorer.py --fit --out openpronounce/models/scorer_en.json

The other scripts follow the same pattern: free.py for free speech, groups.py for age and gender, missing_extra.py, telephone.py and native_calibration.py for a new language. The repository's benchmarks/README.md has every table in full, with what was tried and did not help.

Training on your own ratings

The models are plain JSON files, and OPENPRONOUNCE_MODELS_DIR loads yours in their place. The scripts read speechocean762; training on your own rated recordings means adapting the step that loads them. Before training anything, measure the shipped models on your data: if they already follow your raters closely enough, there is nothing to gain. That work can also be ordered as a service.