Our approach to benchmarking

Reviewed as of 19 August 2026. Modulate publishes results that a third party can reproduce. Every figure in this tab is measured on a public dataset, by a public harness, against a metric defined before the run.

​

Why third-party benchmarks

A vendor-run evaluation selects its own test set, and that selection is not visible in the published number. A leaderboard fixes the datasets, the protocol files, the thresholds, and the scoring code, identically for every entrant. Internal evaluation covers conditions no public benchmark measures, notably telephony-bandwidth audio and the conversational domains Modulate’s customers run. Results in this tab that are internal rather than third-party say so on the page.

​

Selecting datasets

Three properties determine whether a benchmark result is informative:

• Attack and acoustic coverage. A deepfake score measured only on text-to-speech does not describe voice conversion, neural codec resynthesis, or diffusion synthesis. A transcription score measured only on read audiobooks does not describe meetings.
• Out-of-domain datasets. Speech DF Arena’s paper reports many systems posting low error in domain and high error out of domain. Modulate reports per-dataset spread alongside the average for that reason.
• No overlap with training data. Modulate does not train on the evaluation splits of the benchmarks it submits to.

Datasets lists every corpus behind the published results and what each one stresses.

​

Metrics

​

EER and accuracy are one measurement

Equal Error Rate is the false-positive rate at the threshold where false positives and false negatives are equal. One threshold, one number. Detection accuracy as quoted on a deepfake leaderboard is 100% - EER. Modulate’s 1.104% average EER and its 98.9% average accuracy are the same result stated two ways, not two pieces of evidence.

​

Pooled EER and average EER

Speech DF Arena reports both. They are not interchangeable. The two can order the same systems differently. In the 19 August 2026 snapshot, Resemble Detect 3B Omni posts a better pooled EER than Hiya Authenticity Verification (2.099% against 2.324%) and a worse average EER (2.570% against 2.113%). A system with a strong average and a weak pooled score is calibrated per dataset but needs a different threshold for each, which a production deployment cannot supply. Modulate reports both on every deepfake result.

​

WER

Word Error Rate is the sum of substitutions, insertions, and deletions over reference word count. It weights every word equally. A transcript that renders a sentence correctly except the account number scores better than one that drops two filler words. WER is sensitive to text normalisation, so cross-vendor comparison is valid only within a single harness. The Open ASR Leaderboard applies one normaliser to every entrant. Figures taken from two vendors’ own published numbers are not comparable, and this tab does not mix them.

​

Failure modes

• Benchmark overfitting. Repeated submission against a fixed public test set converges on that test set. Out-of-domain spread is the check: tight per-dataset results across unseen corpora indicate generalisation, an average carried by two strong datasets does not.
• Cherry-picked subsets. An average across a chosen pair of datasets is not a leaderboard average. Subset figures in this tab name their datasets in the same sentence as the number.
• Stale standings. A leaderboard position is a snapshot. Standings and their measurement dates are on Benchmarks.
• Size and latency omitted. A system needing 30 seconds of speech cannot gate a live authentication flow whatever its EER. Results pages report parameter count, minimum audio duration, and price alongside accuracy.
• Derived comparisons presented as measurements. Converting an EER into missed detections per thousand calls is arithmetic on a published figure. Pages that do it show the calculation.

​

Reproducing a result