Evaluating OpenPronounce for a product
This page answers the questions a technical or product lead usually asks before choosing a pronunciation engine, and describes how to test it on your own recordings at no cost.
Contents: agreement, data, errors, maintenance, evaluation plan, purchasing.
Does it agree with our own raters?
On the public benchmark, 2,500 recordings rated by five experts, the overall score correlates at 0.85 with the raters for adults reading a sentence and 0.82 for adults speaking without a text. Averaged over the ten recordings each speaker made, the free-speech score ranks speakers at 0.88.
They are non-native speakers reading short English sentences, so the figure that matters is the one measured on your own speakers. The openpronounce-evaluate command produces it. If you already have it, enter it here to see how to use the score.
Where do the recordings go?
Nowhere. They are processed on the machine that runs OpenPronounce and are not sent to the project or to any third party. The software has no account, no licence server and no telemetry.
For a company, that changes the paperwork more than the code. A voice recording is personal data. Sent to a cloud API, each one goes to a vendor, who becomes a processor to declare, with a contract to sign and sometimes a transfer outside the European Union to justify. Kept on your servers, the recordings stay under the policies you already have.
Two network accesses exist by default, and both can be removed:
| What is sent | When | How to remove it |
|---|---|---|
| Download of the speech models from Hugging Face | First run | Build the models into your image and set HF_HUB_OFFLINE=1 |
| Text of the expected sentence, never the audio, to Google's speech synthesis | First analysis of each sentence | OPENPRONOUNCE_TTS=piper, or quick mode |
What happens when the score is wrong?
Every result carries reliable and a list of reasons: the recording does not match the sentence, words are missing, there is too little speech, or the audio is clipped, noisy or too quiet. Your application decides what the user sees in those cases.
The known weaknesses are published with figures in the list of limits. If your use needs it, the scoring models can be retrained on your own ratings: they are JSON files, the training scripts are in the repository, and OPENPRONOUNCE_MODELS_DIR loads your versions in place of the shipped ones.
What if the project stops?
Nothing you ship would break. The code, the models and the scripts that trained them are public under the AGPL, nothing in the software calls home or expires, and your team could keep maintaining the version you ship. Commercial licences pay for maintenance and the next languages; Governance describes how the project is run.
Each model file states its licence, its training data and its measured agreement:
$ openpronounce --models en free speech scorer AGPL-3.0-or-later, total pearson 0.708 trained on speechocean762 train split, 2500 utterances, prompt hidden en utterance scorer AGPL-3.0-or-later, total pearson 0.790 trained on speechocean762 train split, 2500 utterances + 742 synthetic negatives en word model AGPL-3.0-or-later, word_auc_lt5 0.924 fr native speakers of the language against those of English AGPL-3.0-or-later trained on FLEURS fr_fr dev, 120 native recordings, generic models ...
A two-week evaluation
Everything below is allowed by the free licence. A commercial licence is only needed when a closed product reaches its users.
- 1
Day 1: install it
An engineer runs the Docker image on a spare server or a laptop, in about an hour. See the installation guide.
- 2
Days 2 to 5: collect about 100 rated recordings
Use recordings your assessors have already scored, on any scale, or have two assessors rate 100 new ones independently. Cover the levels, ages and devices of your product, and add a column for each breakdown you want: level, first language, microphone.
- 3
Day 6: measure
$ openpronounce-evaluate recordings.csv --group-by level --group-by first_languageThe report gives the correlation with your assessors, with its confidence interval, overall and for each group, and a formula that maps the score onto your scale.
- 4
Second week: try it with real users
Enable it for a small share of your users behind a feature flag. Show a band rather than a number and hide unreliable results. Ask your assessors whether the flagged words are the ones they would have chosen.
- 5
Decide
If the figure and your assessors' opinion are good enough, choose the licence that fits and ship. If not, the evaluation cost two weeks of work. In both cases, the report is welcome: it is how the project learns how it does on speakers like yours.
Purchasing
Which licence do we need?
One per product that includes OpenPronounce, priced by your company's revenue in the last fiscal year. Two questions on the pricing page give the answer.
Do we need a licence to evaluate it?
No. Installing, testing and measuring it on your own data are covered by the AGPL, without time limit.
Can we pay by bank transfer against a purchase order?
Yes for Enterprise licences; ask for a quote and say what your purchasing department needs. Small business and Business licences are paid online by card.
What hardware does it need?
A recent CPU server with 4 GB of free memory. A two-second sentence takes under half a second in quick mode. A GPU is used when present but is not required.
What about the other languages?
French, Spanish, German, Italian, Portuguese and Dutch are included in every licence. Native speakers score in them as they do in English, but no rated corpus of non-native speech exists yet to measure agreement with human raters. If you have rated recordings, the evaluation above would provide the first figures. See Languages.