Measure it on your data
The published figures come from one corpus of non-native speakers reading short English sentences. This page explains how to measure the agreement between OpenPronounce and your own assessors, on your own recordings.
Prepare a CSV of rated recordings
audio,score,text,level recordings/0001.wav,7,"I would like a glass of water",B1 recordings/0002.wav,4,"I would like a glass of water",A2 recordings/0003.wav,8,,C1
audio: a path, relative to the CSV.score: your assessors' rating, on any scale (0 to 10, 1 to 5, levels mapped to numbers).text: what the speaker had to say. Leave it empty for free speech.- Any other column becomes a breakdown: level, first language, age group, microphone.
A hundred recordings give a usable figure. With fewer, the confidence interval gets wide.
Run the command
$ openpronounce-evaluate recordings.csv --group-by levelIt scores the recordings one by one into recordings.results.csv and resumes where it stopped if interrupted. --lang sets the language, --report-only reprints the report without scoring anything.
Read the report
On 150 rated recordings of the benchmark it printed:
150 recordings: 150 compared with a rating, 0 could not be scored, 0 without a rating flagged unreliable: 1, poor recording quality: 0 all n= 150 Pearson 0.840 [0.78, 0.88] Spearman 0.737 error 0.60 to read our score on your scale: yours = 0.1001 x ours + 0.031
- Pearson is the agreement, from 0 to 1, and the brackets are its 95 % interval. Read the interval, not the point.
- Error is how far its score falls from your rating on average, in your units, once mapped on your scale.
- The last line converts its 0 to 100 into your scale, if you want to show your own.
If you can share the report, please send it. It is the only way to learn how OpenPronounce does on speakers nobody has measured.