> ## Documentation Index
> Fetch the complete documentation index at: https://docs.modulate.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Our approach to benchmarking
> How Modulate selects benchmarks and datasets, which metrics it reports, and the failure modes that make a leaderboard score misleading.
*Reviewed as of 19 August 2026.*
Modulate publishes results that a third party can reproduce. Every figure in this tab is measured on a public dataset, by a public harness, against a metric defined before the run.
## Why third-party benchmarks
A vendor-run evaluation selects its own test set, and that selection is not visible in the published number. A leaderboard fixes the datasets, the protocol files, the thresholds, and the scoring code, identically for every entrant.
| Property | Third-party leaderboard | Vendor-run evaluation |
| ------------------ | -------------------------------------- | --------------------------------------------------------- |
| Test set selection | Fixed by the maintainer | Chosen by the vendor |
| Scoring code | Published, shared across entrants | Unpublished |
| Competitor scores | Measured under the same protocol | Quoted from competitor marketing, or re-run by the vendor |
| Reproducibility | Any third party can re-run the harness | None |
Internal evaluation covers conditions no public benchmark measures, notably telephony-bandwidth audio and the conversational domains Modulate's customers run. Results in this tab that are internal rather than third-party say so on the page.
## Selecting datasets
Three properties determine whether a benchmark result is informative:
* **Attack and acoustic coverage.** A deepfake score measured only on text-to-speech does not describe voice conversion, neural codec resynthesis, or diffusion synthesis. A transcription score measured only on read audiobooks does not describe meetings.
* **Out-of-domain datasets.** Speech DF Arena's paper reports many systems posting low error in domain and high error out of domain. Modulate reports per-dataset spread alongside the average for that reason.
* **No overlap with training data.** Modulate does not train on the evaluation splits of the benchmarks it submits to.
[Datasets](/benchmarks/datasets) lists every corpus behind the published results and what each one stresses.
## Metrics
### EER and accuracy are one measurement
Equal Error Rate is the false-positive rate at the threshold where false positives and false negatives are equal. One threshold, one number.
Detection accuracy as quoted on a deepfake leaderboard is `100% - EER`. Modulate's 1.104% average EER and its 98.9% average accuracy are the same result stated two ways, not two pieces of evidence.
### Pooled EER and average EER
Speech DF Arena reports both. They are not interchangeable.
| Metric | Computation | Property measured |
| ----------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| Average EER | Per-dataset EER, then the mean across datasets | Consistency. Each dataset gets an equal vote and its own threshold. |
| Pooled EER | One global threshold across all scores from every dataset in the run | Deployability. Whether a single production threshold holds everywhere at once. |
The two can order the same systems differently. In the 19 August 2026 snapshot, Resemble Detect 3B Omni posts a better pooled EER than Hiya Authenticity Verification (2.099% against 2.324%) and a worse average EER (2.570% against 2.113%).
A system with a strong average and a weak pooled score is calibrated per dataset but needs a different threshold for each, which a production deployment cannot supply. Modulate reports both on every deepfake result.
### WER
Word Error Rate is the sum of substitutions, insertions, and deletions over reference word count. It weights every word equally. A transcript that renders a sentence correctly except the account number scores better than one that drops two filler words.
WER is sensitive to text normalisation, so cross-vendor comparison is valid only within a single harness. The Open ASR Leaderboard applies one normaliser to every entrant. Figures taken from two vendors' own published numbers are not comparable, and this tab does not mix them.
## Failure modes
* **Benchmark overfitting.** Repeated submission against a fixed public test set converges on that test set. Out-of-domain spread is the check: tight per-dataset results across unseen corpora indicate generalisation, an average carried by two strong datasets does not.
* **Cherry-picked subsets.** An average across a chosen pair of datasets is not a leaderboard average. Subset figures in this tab name their datasets in the same sentence as the number.
* **Stale standings.** A leaderboard position is a snapshot. Standings and their measurement dates are on [Benchmarks](/benchmarks/overview).
* **Size and latency omitted.** A system needing 30 seconds of speech cannot gate a live authentication flow whatever its EER. Results pages report parameter count, minimum audio duration, and price alongside accuracy.
* **Derived comparisons presented as measurements.** Converting an EER into missed detections per thousand calls is arithmetic on a published figure. Pages that do it show the calculation.
## Reproducing a result
Speech DF Arena publishes its toolkit at [github.com/Speech-Arena/speech\_df\_arena](https://github.com/Speech-Arena/speech_df_arena). The Open ASR Leaderboard publishes its evaluation code alongside the [leaderboard space](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard).
Each benchmark defines its own protocol format. Speech DF Arena expects a `protocol.csv` per dataset carrying absolute file paths and a `spoof` or `bonafide` label per utterance.
Call the model per utterance and write its score in the format the harness expects. Both leaderboards score utterance-level output, so the per-frame results from [Deepfake Detection](/get-started/deepfake) need aggregating to one score per file.
Report pooled and average EER, or per-dataset and average WER, with the snapshot date of the leaderboard being compared against.
**Worth knowing:** a full harness run makes one API call per evaluation utterance across corpora totalling tens of thousands of files. [Contact support](/support) about evaluation credits before starting one.