> ## Documentation Index
> Fetch the complete documentation index at: https://docs.modulate.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Transcription

> Three transcription models, batch and streaming, with speaker labels, emotion, accent, deepfake, and PII/PHI signals.

Modulate has three transcription models.

* **Multilingual Transcription** carries per-utterance timing, speaker labels, and optional emotion, accent, deepfake, and PII/PHI signals. Batch and streaming.
* **English Fast Transcription** is English-only and tuned for throughput and latency. Batch and streaming.
* **Multilingual Fast Transcription** returns a transcript in any supported language with no metadata. Batch only.

| | Multilingual (batch) | Multilingual (streaming) | English Fast (batch) | English Fast (streaming) | Multilingual Fast (batch) |
| - | - | - | - | - | - |
| Protocol | HTTP POST | WebSocket | HTTP POST | WebSocket | HTTP POST |
| Languages | Multilingual | Multilingual | English only | English only | Multilingual |
| Utterance-level output | ✓ | ✓ | ✓ (opt-in) | ✓ (opt-in `endpointing`) | — |
| Speaker diarization | ✓ | ✓ | ✓ (opt-in) | — | — |
| Emotion signal | ✓ | ✓ | — | — | — |
| Accent signal | ✓ | ✓ | — | — | — |
| Deepfake signal | ✓ | ✓ | — | — | — |
| PII/PHI tagging | ✓ | ✓ | — | — | — |
| Language hint | ✓ | ✓ | — | — | ✓ |
| Custom vocabulary | ✓ | ✓ | — | — | — |
| Partial transcripts | — | ✓ (opt-in `partial_results`) | — | ✓ (every \~1.5 s) | — |

Every signal is off by default except speaker diarization, which defaults to `true` on Multilingual Transcription and to `false` on English Fast Transcription.

## Multilingual Transcription (batch)

Returns `application/json`. `text`, `duration_ms`, and `utterances` are always present. `text` may be an empty string and `utterances` an empty array when no speech was recognized.

| Field | Type | Contents |
| - | - | - |
| `text` | string | The full transcript, all utterances concatenated. |
| `duration_ms` | integer | Total audio duration. |
| `utterances` | array | Utterances ordered by start time. |
| `utterances[].utterance_uuid` | string | Identifier for the utterance. |
| `utterances[].text` | string | Transcribed text for this utterance. |
| `utterances[].start_ms` | integer | Start relative to the beginning of the file. |
| `utterances[].duration_ms` | integer | Utterance length. |
| `utterances[].speaker` | integer | Speaker number, 1-indexed. |
| `utterances[].language` | string | Language detected for this utterance. |
| `utterances[].emotion` | string or null | Null unless `emotion_signal` is enabled. |
| `utterances[].accent` | string or null | Null unless `accent_signal` is enabled. |
| `utterances[].deepfake_score` | number or null | Null unless `deepfake_signal` is enabled, or the utterance is under 0.5 seconds. |

Language is detected per utterance, so a file where speakers switch languages reports each utterance in the language actually spoken.

### Try it

<CodeGroup>
  ```bash curl + jq theme={null}
  curl -sS -X POST https://platform.modulate.ai/api/velma-2-stt-batch \
    -H "X-API-Key: $MODULATE_API_KEY" \
    -F "upload_file=@audio.mp3" \
    -F "speaker_diarization=true" \
    -F "emotion_signal=true" \
    -F "accent_signal=true" \
  | jq -r 'if .detail then "error: \(.detail)" else
      .text, (.utterances[]
      | "  [\(.start_ms)ms] Speaker \(.speaker) (\(.language)) \(.emotion)/\(.accent): \(.text)")
    end'
  ```

  ```python Python theme={null}
  import os, requests

  response = requests.post(
      "https://platform.modulate.ai/api/velma-2-stt-batch",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
      data={
          "speaker_diarization": "true",
          "emotion_signal": "true",
          "accent_signal": "true",
      },
      files={"upload_file": open("audio.mp3", "rb")},
  )
  response.raise_for_status()
  result = response.json()

  print(result["text"])
  for u in result["utterances"]:
      print(f"  [{u['start_ms']}ms] Speaker {u['speaker']} ({u['language']}) "
            f"{u['emotion']}/{u['accent']}: {u['text']}")
  ```

  ```javascript JavaScript theme={null}
  import fs from "fs";
  import FormData from "form-data";

  const form = new FormData();
  form.append("upload_file", fs.createReadStream("audio.mp3"), { filename: "audio.mp3" });
  form.append("speaker_diarization", "true");
  form.append("emotion_signal", "true");
  form.append("accent_signal", "true");

  const response = await fetch("https://platform.modulate.ai/api/velma-2-stt-batch", {
    method: "POST",
    headers: { "X-API-Key": process.env.MODULATE_API_KEY, ...form.getHeaders() },
    body: form,
  });

  const result = await response.json();

  if (!response.ok) {
    console.error(`error: ${result.detail}`);
    process.exit(1);
  }

  console.log(result.text);
  for (const u of result.utterances) {
    console.log(
      `  [${u.start_ms}ms] Speaker ${u.speaker} (${u.language}) ` +
      `${u.emotion}/${u.accent}: ${u.text}`
    );
  }
  ```
</CodeGroup>

```text Output theme={null}
Hello, how are you? Bonjour, ça va?
  [0ms] Speaker 1 (en) Neutral/American: Hello, how are you?
  [2500ms] Speaker 2 (fr) Happy/British: Bonjour, ça va?
```

<Accordion title="Response">
  ```json theme={null}
  {
    "text": "Hello, how are you? Bonjour, ça va?",
    "duration_ms": 5000,
    "utterances": [
      {
        "utterance_uuid": "e5f6a7b8-c9d0-1234-efab-345678901234",
        "text": "Hello, how are you?",
        "start_ms": 0,
        "duration_ms": 2000,
        "speaker": 1,
        "language": "en",
        "emotion": "Neutral",
        "accent": "American",
        "deepfake_score": null
      },
      {
        "utterance_uuid": "f6a7b8c9-d0e1-2345-fabc-456789012345",
        "text": "Bonjour, ça va?",
        "start_ms": 2500,
        "duration_ms": 2500,
        "speaker": 2,
        "language": "fr",
        "emotion": "Happy",
        "accent": "British",
        "deepfake_score": null
      }
    ]
  }
  ```
</Accordion>

The example above enables emotion and accent. With every signal off, the response shape is identical and `emotion`, `accent`, and `deepfake_score` are `null`.

### What you can configure

| Form field | Default | Effect |
| - | - | - |
| `upload_file` | *required* | The audio to transcribe. |
| `speaker_diarization` | `true` | Adds a 1-indexed `speaker` to each utterance. |
| `emotion_signal` | `false` | Adds an `emotion` label per utterance. |
| `accent_signal` | `false` | Adds an `accent` label per utterance. |
| `deepfake_signal` | `false` | Adds a `deepfake_score` per utterance, 0.0 natural to 1.0 synthetic. |
| `pii_phi_tagging` | `false` | Wraps sensitive spans in the transcript text with entity tags. |
| `language` | auto-detect | ISO 639-1 hint such as `en` or `fr`. BCP 47 subtags like `en-US` are accepted; only the primary subtag is used. Invalid codes are rejected with `400`. |
| `config` | — | A JSON-encoded object carrying the same flags plus `custom_terms`. A field set here overrides the matching top-level form field. |

`custom_terms` travels only inside the JSON `config` field. There is no top-level form field for it.

### Audio formats

Accepted extensions: `.aac`, `.aiff`, `.flac`, `.mov`, `.mp3`, `.mp4`, `.ogg`, `.opus`, `.wav`, `.webm`.

Maximum file size is 100 MB. Files above it are rejected with `413`. Empty files are rejected with `400`.

## Multilingual Transcription (streaming)

Utterances arrive as JSON text messages while audio streams in.

| Message | Payload |
| - | - |
| `utterance` | `utterance` object, same fields as the batch response. |
| `partial_utterance` | In-progress text. Emitted only with `partial_results=true`. |
| `done` | `duration_ms`. The server closes after this. |
| `error` | `error` string. The server closes after this. |

Each `partial_utterance` carries `text`, `start_ms`, `speaker`, `emotion`, `accent`, and `deepfake_score`. Any of those except `text` may be null before a value is available. Each partial replaces the previous one for the active utterance, and the finalized `utterance` supersedes all partials before it.

Send audio as binary WebSocket frames in any chunk size. Send an empty text frame (`""`) to end the stream.

### Try it

<CodeGroup>
  ```python Python theme={null}
  import os, asyncio, json, websockets

  API_KEY = os.environ["MODULATE_API_KEY"]
  AUDIO_FILE = "audio.mp3"
  CHUNK_SIZE = 4096

  async def stream():
      url = (
          f"wss://platform.modulate.ai/api/velma-2-stt-streaming"
          f"?api_key={API_KEY}&speaker_diarization=true"
      )
      async with websockets.connect(url) as ws:
          async def send():
              with open(AUDIO_FILE, "rb") as f:
                  while chunk := f.read(CHUNK_SIZE):
                      await ws.send(chunk)
              await ws.send("")  # end of stream

          async def receive():
              async for message in ws:
                  msg = json.loads(message)
                  if msg["type"] == "utterance":
                      u = msg["utterance"]
                      print(f"  [{u['start_ms']}ms] Speaker {u['speaker']} ({u['language']}): {u['text']}")
                  elif msg["type"] == "done":
                      print(f"done: {msg['duration_ms']}ms of audio")
                      break
                  elif msg["type"] == "error":
                      raise RuntimeError(msg["error"])

          await asyncio.gather(send(), receive())

  asyncio.run(stream())
  ```

  ```javascript JavaScript theme={null}
  import { WebSocket } from "ws";
  import { createReadStream } from "fs";

  const url =
    `wss://platform.modulate.ai/api/velma-2-stt-streaming` +
    `?api_key=${process.env.MODULATE_API_KEY}&speaker_diarization=true`;

  const ws = new WebSocket(url);

  ws.on("open", () => {
    const stream = createReadStream("audio.mp3", { highWaterMark: 4096 });
    stream.on("data", (chunk) => ws.send(chunk));
    stream.on("end", () => ws.send(""));
  });

  ws.on("message", (data) => {
    const msg = JSON.parse(data);
    if (msg.type === "utterance") {
      const u = msg.utterance;
      console.log(`  [${u.start_ms}ms] Speaker ${u.speaker} (${u.language}): ${u.text}`);
    } else if (msg.type === "done") {
      console.log(`done: ${msg.duration_ms}ms of audio`);
      ws.close();
    } else if (msg.type === "error") {
      console.error(`error: ${msg.error}`);
    }
  });

  // A rejected handshake closes without sending an error frame first.
  ws.on("close", (code) => {
    if (code !== 1000) console.error(`error: connection closed ${code}`);
  });
  ws.on("error", (err) => console.error(`error: ${err.message}`));
  ```
</CodeGroup>

```text Output theme={null}
  [0ms] Speaker 1 (fr): Bonjour, comment allez-vous?
  [3100ms] Speaker 2 (en): Doing well, thanks for asking.
done: 45000ms of audio
```

<Accordion title="Messages received">
  ```json theme={null}
  { "type": "utterance", "utterance": { "utterance_uuid": "b2c3d4e5-f6a7-8901-bcde-f12345678901", "text": "Bonjour, comment allez-vous?", "start_ms": 0, "duration_ms": 2800, "speaker": 1, "language": "fr", "emotion": null, "accent": null, "deepfake_score": null } }
  { "type": "utterance", "utterance": { "utterance_uuid": "c3d4e5f6-a7b8-9012-cdef-234567890123", "text": "Doing well, thanks for asking.", "start_ms": 3100, "duration_ms": 2200, "speaker": 2, "language": "en", "emotion": null, "accent": null, "deepfake_score": null } }
  { "type": "done", "duration_ms": 45000 }
  ```
</Accordion>

A stream ends with an empty text frame. A shell pipe sends binary frames only, so `curl` and `websocat` cannot close a stream cleanly. Use one of the clients above.

### What you can configure

| Query parameter | Default | Effect |
| - | - | - |
| `api_key` | *required* | The API key. WebSocket connections authenticate through the query string, not a header. |
| `speaker_diarization` | `true` | Adds a 1-indexed `speaker` to each utterance. |
| `emotion_signal` | `false` | Adds an `emotion` label per utterance. |
| `accent_signal` | `false` | Adds an `accent` label per utterance. |
| `deepfake_signal` | `false` | Adds a `deepfake_score` per utterance. |
| `pii_phi_tagging` | `false` | Wraps sensitive spans in the transcribed text with entity tags. |
| `partial_results` | `false` | Emits `partial_utterance` messages with in-progress text. |
| `language` | auto-detect | ISO 639-1 hint. An invalid code closes the connection with `1003`. |
| `audio_format` | auto-detect | Optional for self-describing containers. Required for raw formats. |
| `sample_rate` | *required for raw* | Sample rate in Hz. Must not be sent for self-describing formats. |
| `num_channels` | *required for raw* | 1 to 8. Must not be sent for self-describing formats. |

An optional JSON configuration can be sent as the **first text frame**, before any audio. Any field there overrides the matching query parameter, and it is the only way to pass `custom_terms` on this endpoint. A binary first frame is treated as audio and the query-parameter defaults apply, so binary-first clients keep working unchanged.

### Audio formats

**Self-describing formats.** `wav`, `mp3`, `ogg`, `flac`, `webm`, `aac`, `aiff`. Auto-detected from headers when `audio_format` is omitted. `sample_rate` and `num_channels` must not be sent for these.

**Raw formats.** `s8`, `s16le`, `s16be`, `s24le`, `s24be`, `s32le`, `s32be`, `u8`, `u16le`, `u16be`, `u24le`, `u24be`, `u32le`, `u32be`, `f32le`, `f32be`, `f64le`, `f64be`, `mulaw`, `alaw`. Headerless, so `audio_format`, `sample_rate`, and `num_channels` are all required.

`sample_rate` accepts `8000`, `11025`, `16000`, `22050`, `32000`, `44100`, `48000`, `96000`.

## English Fast Transcription (batch)

Returns `application/json`. `text` and `duration_ms` are always present. Two independent opt-in flags each add one key: `time_stamps` adds `words`, and `speaker_diarization` adds `utterances`. An optional key is absent rather than `null` when its flag is off, so the response schema does not change shape as flags are toggled. When `time_stamps=true` and timings cannot be produced, `words_unavailable` is returned in place of `words`. Exactly one of the two is present on a `time_stamps=true` request.

| Field | Type | Contents |
| - | - | - |
| `text` | string | The full transcript, auto-capitalized and auto-punctuated. |
| `duration_ms` | integer | Total audio duration. |
| `words` | array | Word timings covering `text`, in document order. Present only with `time_stamps=true`. |
| `words[].word` | string | The word as it appears in `text`, including attached punctuation and capitalization. |
| `words[].start` | number | Start time in seconds from the start of the file, to three decimals. |
| `words[].end` | number | End time in seconds from the start of the file, to three decimals. |
| `words[].timing` | string | How the span was obtained: `aligned` when every audio-carrying character was placed, `partial` when only a prefix was measured, `estimated` when the span was bracketed between timed neighbors. |
| `words[].confidence` | number | Alignment confidence from 0 to 1, to four decimals. Measures how well the word was located in the audio, not whether it was transcribed correctly. Absent on an `estimated` word, where nothing was measured. |
| `words_unavailable` | object | Returned in place of `words` when `time_stamps=true` but no timings could be produced. The transcript and any speaker turns are unaffected. |
| `words_unavailable.reason` | string | `audio_too_long_for_alignment`, `alignment_disabled`, `alignment_failed`, or `empty_transcript`. Branch on this value rather than on `message`. |
| `words_unavailable.message` | string | One human-readable sentence, safe to display. The wording is not stable. |
| `words_unavailable.audio_seconds` | number | Duration of the submitted audio, to one decimal. |
| `words_unavailable.max_audio_seconds` | number | Longest audio for which this deployment produces word timings on the non-diarized path, to one decimal. Configured per deployment, so read this value rather than hard-coding a limit. |
| `words_unavailable.alternative` | string | The available fallback, or an empty string when there is none. |
| `utterances` | array | Speaker turns. Present only with `speaker_diarization=true`. May be empty when no transcribable speech is found. |
| `utterances[].utterance_uuid` | string | Identifier for the utterance. |
| `utterances[].text` | string | Transcribed text for this turn. |
| `utterances[].start_ms` | integer | Start within the audio, in milliseconds. |
| `utterances[].duration_ms` | integer | Turn length, in milliseconds. |
| `utterances[].speaker` | integer | Speaker number, 1-indexed. |
| `utterances[].words` | array | Word timings for this turn only. Present only when both flags are enabled. |

<Warning>
  The two arrays carry different units. `words` times are fractional **seconds**; `utterances` times are integer **milliseconds**. Both are measured from the start of the file. That includes `utterances[].words`, whose times are file-relative rather than utterance-relative, so they compare directly against the top-level `words` array with no offset arithmetic.
</Warning>

With diarization enabled, `text` becomes the time-ordered concatenation of the utterance texts.

Word timings come from aligning the finished transcript against the audio, so every entry in `words` matches a word in `text`. Every whitespace-separated token of `text` appears in `words` exactly once, in document order. A word whose characters fall outside the aligner's vocabulary is still returned rather than dropped: `timing` reports `partial` when only a prefix of the word was measured, or `estimated` when the span was bracketed between its timed neighbors. On a `partial` word, `end` is extended in proportion to the untimed characters and clamped to the next word's start, so treat it as approximate at the tail.

On the non-diarized path, word timings are produced up to a maximum audio length configured per deployment, 3600 seconds by default. Above that length the transcript is returned as normal and `words_unavailable` carries the `audio_too_long_for_alignment` reason together with the deployment's `max_audio_seconds`. Setting `speaker_diarization=true` removes the limit, because that path aligns each speaker turn separately.

This model produces no emotion, accent, deepfake, PII/PHI, or custom vocabulary output.

### Try it

<CodeGroup>
  ```bash curl + jq theme={null}
  curl -sS -X POST https://platform.modulate.ai/api/velma-2-stt-batch-english-vfast \
    -H "X-API-Key: $MODULATE_API_KEY" \
    -F "upload_file=@audio.mp3" \
    -F "time_stamps=true" \
    -F "speaker_diarization=true" \
  | jq -r 'if .detail then "error: \(.detail)" else
      .text,
      (.utterances[]? | "  [\(.start_ms)ms] Speaker \(.speaker): \(.text)"),
      (.words[:5][]? | "  \(.start)s - \(.end)s  \(.word)")
    end'
  ```

  ```python Python theme={null}
  import os, requests

  response = requests.post(
      "https://platform.modulate.ai/api/velma-2-stt-batch-english-vfast",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
      data={"time_stamps": "true", "speaker_diarization": "true"},
      files={"upload_file": open("audio.mp3", "rb")},
  )
  response.raise_for_status()
  result = response.json()

  print(result["text"])

  for u in result.get("utterances", []):
      print(f"  [{u['start_ms']}ms] Speaker {u['speaker']}: {u['text']}")

  # words[] times are seconds; utterances[] times are milliseconds
  for w in result.get("words", [])[:5]:
      print(f"  {w['start']}s - {w['end']}s  {w['word']}")
  ```
</CodeGroup>

```text Output theme={null}
Thanks for calling. My account got locked.
  [0ms] Speaker 1: Thanks for calling.
  [4400ms] Speaker 2: My account got locked.
  0.2s - 0.61s  Thanks
  0.64s - 0.79s  for
  0.82s - 1.35s  calling.
  4.51s - 4.68s  My
  4.7s - 5.18s  account
```

Only the first five word timings are printed above. `words` covers every token of `text`.

<Accordion title="Response">
  ```json theme={null}
  {
    "text": "Thanks for calling. My account got locked.",
    "duration_ms": 9000,
    "words": [
      { "word": "Thanks", "start": 0.2, "end": 0.61, "timing": "aligned", "confidence": 0.9881 },
      { "word": "for", "start": 0.64, "end": 0.79, "timing": "aligned", "confidence": 0.9799 },
      { "word": "calling.", "start": 0.82, "end": 1.35, "timing": "aligned", "confidence": 0.9744 },
      { "word": "My", "start": 4.51, "end": 4.68, "timing": "aligned", "confidence": 0.9702 },
      { "word": "account", "start": 4.7, "end": 5.18, "timing": "aligned", "confidence": 0.9835 },
      { "word": "got", "start": 5.2, "end": 5.39, "timing": "aligned", "confidence": 0.9781 },
      { "word": "locked.", "start": 5.42, "end": 5.94, "timing": "aligned", "confidence": 0.9668 }
    ],
    "utterances": [
      {
        "utterance_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
        "text": "Thanks for calling.",
        "start_ms": 0,
        "duration_ms": 1500,
        "speaker": 1,
        "words": [
          { "word": "Thanks", "start": 0.2, "end": 0.61, "timing": "aligned", "confidence": 0.9881 },
          { "word": "for", "start": 0.64, "end": 0.79, "timing": "aligned", "confidence": 0.9799 },
          { "word": "calling.", "start": 0.82, "end": 1.35, "timing": "aligned", "confidence": 0.9744 }
        ]
      },
      {
        "utterance_uuid": "b2c3d4e5-f6a7-8901-bcde-f23456789012",
        "text": "My account got locked.",
        "start_ms": 4400,
        "duration_ms": 1700,
        "speaker": 2,
        "words": [
          { "word": "My", "start": 4.51, "end": 4.68, "timing": "aligned", "confidence": 0.9702 },
          { "word": "account", "start": 4.7, "end": 5.18, "timing": "aligned", "confidence": 0.9835 },
          { "word": "got", "start": 5.2, "end": 5.39, "timing": "aligned", "confidence": 0.9781 },
          { "word": "locked.", "start": 5.42, "end": 5.94, "timing": "aligned", "confidence": 0.9668 }
        ]
      }
    ]
  }
  ```
</Accordion>

The second speaker's word times are file-relative: `4.51` seconds into the file, not `0.11` seconds into their turn.

With both flags off, the response is `text` and `duration_ms` only. The `words` and `utterances` keys are absent rather than empty.

### What you can configure

| Form field | Default | Effect |
| - | - | - |
| `upload_file` | *required* | The audio to transcribe. |
| `time_stamps` | `false` | Adds the `words` array, and `words` inside each utterance when diarization is also on. Costs latency on long audio (see Behavior below). |
| `speaker_diarization` | `false` | Adds the `utterances` array. Note this defaults off here, unlike Multilingual Transcription. |

The two flags are independent: enable either, both, or neither.

### Audio formats

Accepted extensions include `.mp3`, `.wav`, `.flac`, `.m4a`, `.mp4`, `.ogg`, `.opus`, `.webm`, `.aac`, `.aiff`, and `.mov`, plus 85 others.

<Accordion title="All 96 accepted extensions">
  `.3g2`, `.3ga`, `.3gp`, `.3gpp`, `.8svx`, `.aa3`, `.aac`, `.ac3`, `.act`, `.adts`, `.aif`,
  `.aifc`, `.aiff`, `.alac`, `.amb`, `.amr`, `.ape`, `.asf`, `.at3`, `.au`, `.avi`, `.avr`,
  `.awb`, `.bwf`, `.c2`, `.caf`, `.dss`, `.dts`, `.dtshd`, `.eac3`, `.ec3`, `.f4a`, `.f4b`,
  `.flac`, `.flv`, `.gsm`, `.iff`, `.m2a`, `.m2ts`, `.m4a`, `.m4b`, `.m4r`, `.m4v`, `.mka`,
  `.mkv`, `.mlp`, `.mmf`, `.mov`, `.mp+`, `.mp1`, `.mp2`, `.mp3`, `.mp4`, `.mpa`, `.mpc`, `.mpeg`,
  `.mpg`, `.mpga`, `.mpp`, `.mts`, `.mxf`, `.nist`, `.oga`, `.ogg`, `.ogx`, `.oma`, `.omg`,
  `.opus`, `.paf`, `.pvf`, `.qcp`, `.ra`, `.rf64`, `.rka`, `.rm`, `.rmvb`, `.sf`, `.shn`, `.snd`,
  `.sph`, `.svx`, `.tak`, `.thd`, `.ts`, `.tta`, `.vob`, `.voc`, `.vqf`, `.w64`, `.wav`,
  `.wave`, `.weba`, `.webm`, `.wma`, `.wmv`, `.wv`
</Accordion>

Maximum file size is 100 MB. Files above it are rejected with `413`. Empty files are rejected with `400`.

### Behavior

`time_stamps` adds a forced-alignment pass over the audio. On short audio that pass overlaps transcription and is effectively free. Its cost grows faster than linearly with audio length, so on long recordings it can add substantial latency. Benchmark with the flag set to its production value, not with it off.

`speaker_diarization` transcribes each speaker turn separately rather than the file as a whole. It is slower than the default path, and the transcript it returns can differ from the same audio submitted without the flag. Do not treat a diarized `text` as byte-identical to an undiarized one.

With diarization enabled, quiet background speech on far-field or multi-party audio can surface as additional utterances. Overlapping speech is attributed to a single speaker per time span.

<Note>
  `503` carries a second meaning on this endpoint: besides a temporary outage, it is returned when `speaker_diarization` is requested while diarization is disabled on the deployment serving the request. Retrying will not clear that case. Drop the flag, or ask about having diarization enabled.
</Note>

## English Fast Transcription (streaming)

The lowest-latency option, and the one for cases where a person is waiting on the text.

| Message | Payload |
| - | - |
| `partial_utterance` | `text` and `is_final: false`. Emitted roughly every 1.5 seconds. |
| `utterance` | `text`, `is_final: true`, `start_ms`, `duration_ms`. With `diarize=true`, also `speaker`, `n_speakers`, `speakers_capped`, and `speaker_purity`. |
| `done` | `duration_ms`. The server closes after this. |
| `error` | `error` string. The server closes after this. |

Text in both message types is already auto-capitalized and auto-punctuated. Render `text` directly.

<Warning>
  Each `partial_utterance` contains the complete transcript of its scope so far, not a delta. Replace the displayed text on every partial. Never append. The last few words, including their capitalization and punctuation, can be revised in the next partial as the model receives more context.
</Warning>

Every connection ends with at least one final `utterance`. A stream containing no speech yields a single final with empty `text`.

### Try it

<CodeGroup>
  ```python Python theme={null}
  import asyncio, json, os, websockets

  API_KEY = os.environ["MODULATE_API_KEY"]
  AUDIO_FILE = "audio.ogg"
  CHUNK_SIZE = 8192

  async def transcribe():
      url = (
          f"wss://platform.modulate.ai/api/velma-2-stt-streaming-english-v2"
          f"?api_key={API_KEY}&audio_format=ogg&endpointing=true"
      )
      async with websockets.connect(url, max_size=None) as ws:
          async def send():
              with open(AUDIO_FILE, "rb") as f:
                  while chunk := f.read(CHUNK_SIZE):
                      await ws.send(chunk)
              await ws.send("")  # end of stream

          send_task = asyncio.create_task(send())
          try:
              async for message in ws:
                  msg = json.loads(message)
                  if msg["type"] == "partial_utterance":
                      # Replace the displayed partial, do not append.
                      print(f"\r[partial] {msg['partial_utterance']['text']}", end="", flush=True)
                  elif msg["type"] == "utterance":
                      print(f"\n[final]   {msg['utterance']['text']}")
                  elif msg["type"] == "done":
                      print(f"\ndone: {msg['duration_ms']}ms of audio")
                      break
                  elif msg["type"] == "error":
                      raise RuntimeError(msg["error"])
          finally:
              if not send_task.done():
                  send_task.cancel()

  asyncio.run(transcribe())
  ```
</CodeGroup>

```text Output theme={null}
[partial] Hello, how are you doing
[final]   Hello, how are you doing today?

done: 14253ms of audio
```

Each partial overwrites the one before it on the same terminal line, so the running text is visible in place and only the last one remains once the final arrives.

<Accordion title="Messages received">
  ```json theme={null}
  {"type": "partial_utterance", "partial_utterance": {"text": "Hello, how are you", "is_final": false}}
  {"type": "partial_utterance", "partial_utterance": {"text": "Hello, how are you doing", "is_final": false}}
  {"type": "utterance", "utterance": {"text": "Hello, how are you doing today?", "is_final": true, "start_ms": 0, "duration_ms": 2360}}
  {"type": "done", "duration_ms": 14253}
  ```
</Accordion>

A stream ends with an empty text frame. A shell pipe sends binary frames only, so `curl` and `websocat` cannot close a stream cleanly. Use one of the clients above.

### What you can configure

| Query parameter | Default | Effect |
| - | - | - |
| `api_key` | *required* | The API key. |
| `audio_format` | *required* | How the server decodes the bytes sent. Required on every connection. |
| `sample_rate` | *required for raw* | Sample rate in Hz. Ignored for container formats. |
| `num_channels` | *required for raw* | 1 to 8. Ignored for container formats. |
| `endpointing` | `false` | Segments speech at pauses. |
| `diarize` | `false` | Labels which speaker produced each final `utterance`. Independent of `endpointing`. |

With `endpointing=false`, the connection produces a single final `utterance` at end of stream covering the whole stream with `start_ms=0`, and each partial reflects the whole stream so far.

With `endpointing=true`, trailing silence closes the current utterance and starts a new one. The server emits one final `utterance` per speech segment, each with its own `start_ms` and `duration_ms`, and each partial reflects the current segment. The full transcript is the concatenation of the final texts. This is what makes the endpoint usable for live conversation.

### Speaker labels

With `diarize=true`, each final `utterance` carries `speaker`, a zero-based index identifying who produced it. Partials never carry it. A label is assigned when an utterance is finalized and is never revised.

The index is stable for one connection and meaningless outside it: the same person on a new connection may get a different index, so it is not an identity to store or match across calls. `speaker` is `null` when diarization was requested but no speaker could be determined, which happens on very short segments and on a voice the model has not heard enough of yet. Absent is different again, and means `diarize=true` was not sent.

Three fields qualify the label, and two of them are the reason to read more than `speaker` alone:

| Field | Meaning |
| - | - |
| `n_speakers` | Distinct speakers detected on the connection so far. |
| `speakers_capped` | `true` once all ten speaker slots are in use. |
| `speaker_purity` | Proportion of this utterance's speech that came from `speaker`, from 0 to 1. |

<Warning>
  **Worth knowing:** ten distinct speakers are labeled per connection. Beyond that, further speakers are merged into existing labels — their speech is still transcribed, but `speaker` is wrong for some utterances, and `speakers_capped` is the only signal that it happened.
</Warning>

`speaker_purity` below `1.0` means the utterance spans a speaker change. Segments are cut at pauses whether or not `diarize` is set, so a fast exchange with no pause between turns stays in one utterance and gets the label of whoever spoke most of it.

With `diarize=true&endpointing=false` there is only one final `utterance` for the whole stream, so one `speaker` cannot describe it. Read `n_speakers` there and treat `speaker` as indicative.

A diarized connection has a lower concurrency ceiling than plain transcription, so it can be refused with close code `1013` while a plain connection succeeds. Close code `1003` covers both an invalid `diarize` value and `diarize=true` on an instance where speaker labelling is not enabled.

### Audio formats

`audio_format` is required on every connection, including containers. Omitting it closes the connection with `1003`.

**Container formats.** Accepted values include `mp3`, `wav`, `flac`, `m4a`, `mp4`, `ogg`, `opus`, `webm`, `aac`, `aiff`, `wma`, `amr`, and `au`, plus 66 others. The header determines sample rate and channel count, and `sample_rate` and `num_channels` are ignored.

<Accordion title="All 79 accepted container values">
  `3g2`, `3ga`, `3gp`, `3gpp`, `8svx`, `aa3`, `aac`, `ac3`, `act`, `adts`, `aif`, `aifc`, `aiff`,
  `amb`, `amr`, `asf`, `at3`, `au`, `avr`, `awb`, `bwf`, `c2`, `caf`, `dss`, `dts`, `dtshd`,
  `eac3`, `ec3`, `f4a`, `f4b`, `flac`, `gsm`, `iff`, `m2a`, `m2ts`, `m4a`, `m4b`, `m4r`, `m4v`,
  `mka`, `mkv`, `mlp`, `mp+`, `mp1`, `mp2`, `mp3`, `mp4`, `mpa`, `mpc`, `mpga`, `mpp`, `mts`,
  `oga`, `ogg`, `ogx`, `oma`, `omg`, `opus`, `paf`, `pvf`, `qcp`, `ra`, `rf64`, `rm`, `rmvb`,
  `snd`, `svx`, `thd`, `ts`, `tta`, `voc`, `vqf`, `w64`, `wav`, `wave`, `weba`, `webm`, `wma`,
  `wmv`
</Accordion>

The MP4-family values (`mp4`, `m4a`, `m4b`, `m4r`, `m4v`, `3gp`, `3gpp`, `3ga`, `3g2`, `f4a`, `f4b`) must be sent in a streamable layout. Anything else ends the connection with an audio-processing error.

**Raw formats.** `s8`, `s16le`, `s16be`, `s24le`, `s24be`, `s32le`, `s32be`, `u8`, `u16le`, `u16be`, `u24le`, `u24be`, `u32le`, `u32be`, `f32le`, `f32be`, `f64le`, `f64be`, `mulaw`, `alaw`, `g722`, `vox`. `sample_rate` and `num_channels` are both required. `g722` and `vox` are mono-only: `num_channels` must be `1`.

`sample_rate` accepts `8000`, `11025`, `16000`, `22050`, `32000`, `44100`, `48000`, `96000`.

For the lowest end-to-end latency use `audio_format=s16le&sample_rate=16000&num_channels=1`, which bypasses the server's audio decoder.

Sending container bytes while declaring a raw PCM format closes the connection with `4002`.

## Multilingual Fast Transcription (batch)

Returns `application/json`. All three fields are always present.

| Field | Type | Contents |
| - | - | - |
| `text` | string | The full transcript, auto-capitalized and auto-punctuated. |
| `duration_ms` | integer | Total audio duration. |
| `language` | string | The language the transcript is in, as a short code. Echoes a declared `language`, otherwise reports the detected one. |

For audio containing more than one language, `language` reports the predominant one. The transcript still reflects each part in the language spoken there.

This model produces no diarization, utterance breakdown, or enrichment signals.

### Try it

<CodeGroup>
  ```bash curl + jq theme={null}
  curl -sS -X POST https://platform.modulate.ai/api/velma-2-stt-batch-multilingual-vfast \
    -H "X-API-Key: $MODULATE_API_KEY" \
    -F "upload_file=@audio.mp3" \
    -F "language=es" \
  | jq -r 'if .detail then "error: \(.detail)" else
      "[\(.language)] \(.text)"
    end'
  ```

  ```python Python theme={null}
  import os, requests

  response = requests.post(
      "https://platform.modulate.ai/api/velma-2-stt-batch-multilingual-vfast",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
      data={"language": "es"},
      files={"upload_file": open("audio.mp3", "rb")},
  )
  response.raise_for_status()
  result = response.json()
  print(f"[{result['language']}] {result['text']}")
  ```
</CodeGroup>

```text Output theme={null}
[es] Hola a todos. Bienvenidos a la junta semanal.
```

<Accordion title="Response">
  ```json theme={null}
  {
    "text": "Hola a todos. Bienvenidos a la junta semanal.",
    "duration_ms": 4200,
    "language": "es"
  }
  ```
</Accordion>

### What you can configure

| Form field | Default | Effect |
| - | - | - |
| `upload_file` | *required* | The audio to transcribe. |
| `language` | auto-detect | A short code such as `en`, `es`, `fr`, `ja`. Declaring it takes the fastest direct path and the value is echoed back. Omit to detect automatically. |

### Audio formats

Accepted extensions: `.aac`, `.aiff`, `.flac`, `.mov`, `.mp3`, `.mp4`, `.ogg`, `.opus`, `.wav`, `.webm`.

Maximum file size is 100 MB. Files above it are rejected with `413`. Empty files are rejected with `400`.

### Error codes

This endpoint returns `401` for a missing or invalid key, where the other transcription endpoints return `403`. It also returns `502` when the request cannot be completed and `504` on timeout.

## Multilingual Fast Transcription (streaming)

The streaming counterpart to Multilingual Fast Transcription. The spoken language is identified automatically, so no language is declared and there is no per-language endpoint.

The 25 supported languages are Bulgarian, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hungarian, Italian, Latvian, Lithuanian, Maltese, Polish, Portuguese, Romanian, Russian, Slovak, Slovenian, Spanish, Swedish, Ukrainian.

<Warning>
  **Worth knowing:** these 25 are all European. Multilingual Transcription covers 100 languages, so audio in Chinese, Japanese, Arabic, Hindi or Korean belongs on that endpoint rather than this one.
</Warning>

| Message | Payload |
| - | - |
| `partial_utterance` | `text` and `is_final: false`. The complete transcript so far for the current scope. |
| `utterance` | `text`, `is_final: true`, `start_ms`, `duration_ms`. With `diarize=true`, also `speaker`, `n_speakers`, `speakers_capped`, and `speaker_purity`. |
| `done` | `duration_ms`. The server closes after this. |
| `error` | `error` string. The server closes after this. |

Text is auto-capitalized and auto-punctuated in the language spoken. Render it directly.

<Warning>
  **Worth knowing:** no message reports which language was detected. The batch endpoint returns a `language` field; this one does not, so a client cannot branch on the language. Call [Language Detection](/get-started/language-detection) when the code itself is needed.
</Warning>

Each `partial_utterance` carries the complete transcript of its scope, not a delta. Replace the displayed text on every partial rather than appending.

### What you can configure

| Query parameter | Default | Effect |
| - | - | - |
| `api_key` | *required* | The API key. |
| `audio_format` | *required* | How the server decodes the bytes sent. Required on every connection. |
| `sample_rate` | *required for raw* | Sample rate in Hz. Ignored for container formats. |
| `num_channels` | *required for raw* | 1 to 8. Ignored for container formats. |
| `endpointing` | `false` | Segments speech at pauses into one final `utterance` per segment. |
| `diarize` | `false` | Labels which speaker produced each final `utterance`. Independent of `endpointing`. |

### Speaker labels

`diarize=true` behaves as it does on English Fast Transcription (streaming), with one difference that matters when sizing a conversation: **four** concurrent speakers are labeled here, not ten.

| Field | Meaning |
| - | - |
| `n_speakers` | Distinct speakers detected on the connection so far. |
| `speakers_capped` | `true` once all four speaker slots are in use. |
| `speaker_purity` | Proportion of this utterance's speech that came from `speaker`, from 0 to 1. |

<Warning>
  **Worth knowing:** beyond four speakers, further speakers are merged into existing labels. Their speech is still transcribed, but `speaker` is wrong for some utterances, and `speakers_capped` is the only signal it happened. A four-person call has no headroom.
</Warning>

Labels appear on final utterances only, never on partials, and are never revised once assigned. `speaker` is stable within one connection and meaningless outside it. `speaker: null` means diarization ran but could not assign that utterance, which is different from the field being absent.

With `diarize=true` and `endpointing=false` there is one whole-stream utterance, so read `n_speakers` and treat `speaker` as indicative.

### Audio formats

`audio_format` is required on every connection, including containers.

**Container formats.** Accepted values include `mp3`, `wav`, `flac`, `m4a`, `mp4`, `ogg`, `opus`, `webm`, `aac`, `aiff`, `wma`, `amr`, and `au`, plus 66 others. The header determines sample rate and channel count, and `sample_rate` and `num_channels` are ignored.

<Accordion title="All 79 accepted container values">
  `3g2`, `3ga`, `3gp`, `3gpp`, `8svx`, `aa3`, `aac`, `ac3`, `act`, `adts`, `aif`, `aifc`, `aiff`,
  `amb`, `amr`, `asf`, `at3`, `au`, `avr`, `awb`, `bwf`, `c2`, `caf`, `dss`, `dts`, `dtshd`,
  `eac3`, `ec3`, `f4a`, `f4b`, `flac`, `gsm`, `iff`, `m2a`, `m2ts`, `m4a`, `m4b`, `m4r`, `m4v`,
  `mka`, `mkv`, `mlp`, `mp+`, `mp1`, `mp2`, `mp3`, `mp4`, `mpa`, `mpc`, `mpga`, `mpp`, `mts`,
  `oga`, `ogg`, `ogx`, `oma`, `omg`, `opus`, `paf`, `pvf`, `qcp`, `ra`, `rf64`, `rm`, `rmvb`,
  `snd`, `svx`, `thd`, `ts`, `tta`, `voc`, `vqf`, `w64`, `wav`, `wave`, `weba`, `webm`, `wma`,
  `wmv`
</Accordion>

The MP4-family values (`mp4`, `m4a`, `m4b`, `m4r`, `m4v`, `3gp`, `3gpp`, `3ga`, `3g2`, `f4a`, `f4b`) must be sent in a streamable layout. Anything else ends the connection with an audio-processing error.

**Raw formats.** `s8`, `s16le`, `s16be`, `s24le`, `s24be`, `s32le`, `s32be`, `u8`, `u16le`, `u16be`, `u24le`, `u24be`, `u32le`, `u32be`, `f32le`, `f32be`, `f64le`, `f64be`, `mulaw`, `alaw`, `g722`, `vox`. `sample_rate` and `num_channels` are both required. `g722` and `vox` are mono-only: `num_channels` must be `1`.

`sample_rate` accepts `8000`, `11025`, `16000`, `22050`, `32000`, `44100`, `48000`, `96000`.

## Signals

### Speaker diarization

`speaker_diarization`, boolean. Identifies distinct speakers and assigns each utterance a 1-indexed `speaker`. Numbers are consistent within one request: speaker 1 in one utterance is the same person as speaker 1 in another from the same audio. They are not consistent across separate requests or files.

Diarization is independent of language detection. In a multilingual conversation a speaker keeps one label across a language switch.

### Emotion

`emotion_signal`, boolean. Classifies the emotional tone of each utterance from the voice signal, into `emotion`. Classification is acoustic, so two utterances with identical text can receive different labels if the delivery differs.

`Neutral`, `Calm`, `Happy`, `Amused`, `Excited`, `Proud`, `Affectionate`, `Interested`, `Hopeful`, `Frustrated`, `Angry`, `Contemptuous`, `Concerned`, `Afraid`, `Sad`, `Ashamed`, `Bored`, `Tired`, `Surprised`, `Anxious`, `Stressed`, `Disgusted`, `Disappointed`, `Confused`, `Relieved`, `Confident`.

For a whole-file label with no transcript, see [Emotion Detection](/get-started/emotion).

### Accent

`accent_signal`, boolean. Classifies the regional or national accent of each utterance's speaker, into `accent`. Results are typically stable for a speaker with a consistent accent, and vary more on short or acoustically difficult segments.

`American`, `British`, `Australian`, `Southern`, `Indian`, `Irish`, `Scottish`, `Eastern_European`, `African`, `Asian`, `Latin_American`, `Middle_Eastern`, `Unknown`.

For a whole-file label with no transcript, see [Accent Detection](/get-started/accent).

### Deepfake score

`deepfake_signal`, boolean. Scores each utterance for the likelihood it contains AI-generated speech, into `deepfake_score`.

| Value | Meaning |
| - | - |
| `0.0` | Likely natural human speech. |
| `1.0` | Likely synthetic speech. |
| `null` | The signal is disabled, or the utterance is under 0.5 seconds. |

This is one score per utterance. For frame-level verdicts across a whole file, explicit `no-content` handling for silence, or verdicts on live audio without a transcript, see [Deepfake Detection](/get-started/deepfake).

### PII/PHI tagging

`pii_phi_tagging`, boolean. Wraps personally identifiable and personal health information in the transcript text with entity tags. Transcript content is preserved and only markup is added.

To also silence the corresponding audio ranges, use [PII/PHI Redaction](/get-started/pii) instead.

### Language hint

`language`, ISO 639-1 code. By default the language is detected per utterance. Passing a code hints the expected language. BCP 47 region and script subtags such as `en-US` are accepted, but only the primary subtag is used. Case-insensitive.

Transport differs by endpoint, and getting it wrong produces a `400` or a silently ignored value:

| Endpoint | Where `language` goes |
| - | - |
| Multilingual Transcription (batch) | Top-level form field, or inside the JSON `config` field |
| Multilingual Transcription (streaming) | Query parameter, or inside the first-frame configuration |
| Multilingual Fast Transcription (batch) | Top-level form field |
| PII/PHI Redaction (batch) | Top-level form field |
| PII/PHI Redaction (streaming) | Query parameter |
| Velma Triage (batch and streaming) | The `stt` block of the config |

An invalid code is rejected with `400` on batch endpoints, and closes streaming connections with `1003`. English Fast Transcription accepts no language parameter in either mode.

### Custom vocabulary

`custom_terms`, array. Biases transcription toward domain terms and names that would otherwise be mistranscribed. Each entry is a plain string or an object:

| Field | Type | Contents |
| - | - | - |
| `term` | string | Required. The term to bias toward. |
| `definition` | string | Optional, 256 characters maximum. |
| `pronunciations` | array of strings | Optional, 16 maximum, 128 characters each. X-SAMPA notation, or hyphen-separated ASCII respelling with the stressed syllable capitalized, such as `GOO-guhl`. Printable ASCII only. |

At most 1000 entries, and the term strings serialized together as JSON must total under 8000 characters. Blank terms and blank pronunciations are dropped.

```json theme={null}
{
  "custom_terms": [
    "Modulate",
    { "term": "Velma", "definition": "Modulate's conversation intelligence model", "pronunciations": ["VEL-muh"] }
  ]
}
```

| Endpoint | Where `custom_terms` goes |
| - | - |
| Multilingual Transcription (batch) | Inside the JSON `config` form field |
| Multilingual Transcription (streaming) | Inside the first-frame configuration |
| Velma Triage (batch and streaming) | The `stt` block of the config |

The other transcription endpoints and both redaction endpoints do not accept custom vocabulary.

## API reference

* [Multilingual Transcription Batch](/api-reference/stt/batch)
* [Multilingual Transcription Streaming](/api-reference/stt/streaming)
* [English Fast Transcription Batch](/api-reference/stt/batch-english-vfast)
* [English Fast Transcription Streaming](/api-reference/stt/streaming-vfast)
* [Multilingual Fast Transcription Batch](/api-reference/stt/batch-multilingual-vfast)
* [Multilingual Fast Transcription Streaming](/api-reference/stt/streaming-multilingual-vfast)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.