> ## Documentation Index
> Fetch the complete documentation index at: https://docs.modulate.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Speech-to-Text Streaming English
> Low-latency English speech-to-text over WebSocket. Emits rolling partial transcripts during streaming, with optional utterance segmentation at pauses (endpointing) and optional speaker labels (diarize). No other enrichments.
Low-latency English speech-to-text over WebSocket. Emits a rolling partial transcript every \~1.5 seconds while audio streams in, then delivers one complete final transcript at end-of-stream — or, with [endpointing enabled](#utterance-segmentation-endpointing), a final transcript for each utterance as the speaker pauses. With [`diarize=true`](#speaker-labels), each final utterance also carries a speaker label. No emotion detection, accent detection, or PII/PHI tagging.
## Endpoint
```text theme={null}
wss://platform.modulate.ai/api/velma-2-stt-streaming-english-v2
```
## Authentication
Pass your API key as a query parameter when opening the connection.
```text theme={null}
wss://platform.modulate.ai/api/velma-2-stt-streaming-english-v2?api_key=YOUR_API_KEY&audio_format=ogg
```
Unlike the batch endpoints, this API does not use an `X-API-Key` header. The key must be in the query string at connection time.
See [Authentication and rate limits](/guides/authentication) for how to obtain and manage API keys.
## Supported audio formats
**Container formats** — `sample_rate` and `num_channels` are ignored if supplied:
`mp3`, `wav`, `flac`, `m4a`, `mp4`, `ogg`, `opus`, `webm`, `aac`, `aiff`, `wma`, `amr`, and `au`, plus 66 others
`3g2`, `3ga`, `3gp`, `3gpp`, `8svx`, `aa3`, `aac`, `ac3`, `act`, `adts`, `aif`, `aifc`, `aiff`,
`amb`, `amr`, `asf`, `at3`, `au`, `avr`, `awb`, `bwf`, `c2`, `caf`, `dss`, `dts`, `dtshd`,
`eac3`, `ec3`, `f4a`, `f4b`, `flac`, `gsm`, `iff`, `m2a`, `m2ts`, `m4a`, `m4b`, `m4r`, `m4v`,
`mka`, `mkv`, `mlp`, `mp+`, `mp1`, `mp2`, `mp3`, `mp4`, `mpa`, `mpc`, `mpga`, `mpp`, `mts`,
`oga`, `ogg`, `ogx`, `oma`, `omg`, `opus`, `paf`, `pvf`, `qcp`, `ra`, `rf64`, `rm`, `rmvb`,
`snd`, `svx`, `thd`, `ts`, `tta`, `voc`, `vqf`, `w64`, `wav`, `wave`, `weba`, `webm`, `wma`,
`wmv`
The MP4-family values (`mp4`, `m4a`, `m4b`, `m4r`, `m4v`, `3gp`, `3gpp`, `3ga`, `3g2`, `f4a`, `f4b`) must be sent in a streamable layout; otherwise the connection ends with an audio-processing error.
Opus audio is accepted either as `opus` or, for Opus-in-Ogg streams, as `ogg`.
**Raw PCM formats** — `sample_rate` and `num_channels` are required:
`s8`, `s16le`, `s16be`, `s24le`, `s24be`, `s32le`, `s32be`, `u8`, `u16le`, `u16be`, `u24le`, `u24be`, `u32le`, `u32be`, `f32le`, `f32be`, `f64le`, `f64be`, `mulaw`, `alaw`, `g722`, `vox`
`g722` and `vox` are mono-only: `num_channels` must be `1`.
**Valid sample rates:** 8000, 11025, 16000, 22050, 32000, 44100, 48000, 96000
For lowest end-to-end latency, send `audio_format=s16le&sample_rate=16000&num_channels=1`. This matches the model's native input format and bypasses the server's audio decoder entirely.
## Query parameters
| Parameter | Required | Description |
| - | - | - |
| `api_key` | Yes | Your API key |
| `audio_format` | Yes | Container or raw PCM format of the audio you will stream. See supported formats above. |
| `sample_rate` | Raw PCM only | Source sample rate in Hz. Required when `audio_format` is a raw PCM type; ignored for container formats. |
| `num_channels` | Raw PCM only | Number of audio channels (1–8). Required when `audio_format` is a raw PCM type; ignored for container formats. The server downmixes to mono internally. |
| `endpointing` | No | `true` or `false` (default `false`). When `true`, speech is segmented into utterances at pauses — see [Utterance segmentation](#utterance-segmentation-endpointing). Any other value is rejected with close code `1003`. |
| `diarize` | No | `true` or `false` (default `false`). When `true`, each final `utterance` carries speaker fields — see [Speaker labels](#speaker-labels). Any other value is rejected with close code `1003`. |
`diarize` and `endpointing` are independent; all four combinations are valid.
This endpoint accepts no other enrichment toggles. Emotion, accent, and PII parameters are not recognized and have no effect. Use [STT Streaming](/api-reference/stt/streaming) if you need those.
## Utterance segmentation (endpointing)
By default, the connection produces one final `utterance` at end-of-stream covering everything you sent, and each `partial_utterance` reflects the whole stream so far.
With `endpointing=true`, speech is segmented at pauses:
* Each `partial_utterance` contains the complete transcript of the **current segment** so far (still replace, never append).
* Each time a pause ends a segment, you immediately receive a final `utterance` for it — final text arrives shortly after the speaker stops, instead of at end-of-stream. Each final carries `start_ms` and `duration_ms` for its speech.
* The full transcript is the concatenation of the final `utterance` texts, in order.
* Every connection still ends with at least one final `utterance` (a stream containing no speech yields one with empty `text`), followed by `done`.
Use endpointing for live conversations — voice agents, assistants, meeting captions — where you want final text per utterance as it happens. Leave it off for whole-recording transcription where one final transcript is simpler to consume.
## Speaker labels
Set `diarize=true` to label which speaker produced each utterance. Labels are attached to final `utterance` messages only, never to `partial_utterance`.
`speaker` is a zero-based index, stable for the lifetime of one connection and meaningless outside it. The same person on a new connection may receive a different index, so the value identifies a speaker within a conversation and is not an identity that can be matched across connections or sessions.
A label is assigned when an utterance is finalized and is never revised. There is no back-patching, so an early utterance keeps whatever label it was given.
`speaker` is `null` when diarization was requested but no speaker could be determined for that utterance, which happens on very short or silent segments and on a brand-new voice that has not yet accumulated enough evidence. That is distinct from the field being absent, which means `diarize=true` was not requested.
### Speaker capacity
Up to ten distinct speakers are labeled per connection. Past that, further speakers are merged into existing labels rather than reported separately: their speech is still transcribed, but `speaker` is wrong for some utterances.
**Worth knowing:** `speakers_capped` is the only signal that this happened. Nothing in the transcript or in `speaker` itself indicates a merged speaker, so a client acting on speaker identity should check `speakers_capped` alongside `n_speakers`.
### Utterances that span a speaker change
Segments are cut at pauses regardless of `diarize`. An utterance that spans a fast exchange with no pause between turns carries the speaker who produced most of it, and `speaker_purity` reports what proportion that was. `1.0` means the utterance was cleanly single-speaker; a lower value means the single label describes only the majority of it.
### With endpointing off
`diarize=true&endpointing=false` produces one final `utterance` for the whole stream, so a single `speaker` cannot describe it. Read `n_speakers` in that mode and treat `speaker` as indicative only.
## Connection flow
1. Connect to the WebSocket endpoint with `api_key`, `audio_format`, (for raw PCM) `sample_rate` and `num_channels`, and optionally `endpointing` and `diarize`.
2. Stream audio as **binary** WebSocket frames. Frames can be any size; 4–64 KB is typical.
3. Receive `partial_utterance` JSON messages every \~1.5 seconds. Each contains the **complete transcript of its scope so far** — replace any previously displayed partial, do not append.
4. With `endpointing=true`, receive a final `utterance` message each time a pause ends a segment.
5. Send an **empty text frame** (`""`) to signal end of audio.
6. Receive the final `utterance` for any content not already finalized at a pause (the whole stream with endpointing off; skipped when the last segment was already finalized mid-stream).
7. Receive a `done` message with total audio duration.
8. The connection closes automatically.
## Server messages
### `partial_utterance`
Sent roughly every 1.5 seconds while audio is streaming. Each message contains the **complete transcript built so far for the current scope** — the whole connection by default, or the current speech segment when `endpointing=true`. It is not a delta from the previous message: replace your displayed partial text with each new value — never append.
```json theme={null}
{
"type": "partial_utterance",
"partial_utterance": {
"text": "Hello, how are you",
"is_final": false
}
}
```
| Field | Type | Description |
| - | - | - |
| `type` | string | Always `"partial_utterance"` |
| `partial_utterance.text` | string | Complete transcript so far. Replace, don't append. |
| `partial_utterance.is_final` | boolean | Always `false` |
Partials never carry speaker labels, even with `diarize=true`. Speaker fields appear on final `utterance` messages only.
### `utterance`
The final transcript for one utterance. Will not be revised. With endpointing off, sent exactly once at end-of-stream, covering the entire audio stream. With `endpointing=true`, sent each time a pause ends a segment; the full transcript is the concatenation of every `utterance` text in order.
```json theme={null}
{
"type": "utterance",
"utterance": {
"text": "Hello, how are you doing today?",
"is_final": true,
"start_ms": 0,
"duration_ms": 2360
}
}
```
With `diarize=true`, the same message carries four additional fields:
```json theme={null}
{
"type": "utterance",
"utterance": {
"text": "Good morning, thanks for calling.",
"is_final": true,
"start_ms": 8120,
"duration_ms": 4440,
"speaker": 0,
"n_speakers": 2,
"speakers_capped": false,
"speaker_purity": 1.0
}
}
```
| Field | Type | Description |
| - | - | - |
| `type` | string | Always `"utterance"` |
| `utterance.text` | string | Final transcript for this utterance. Will not be revised. |
| `utterance.is_final` | boolean | Always `true` |
| `utterance.start_ms` | integer | Start of this utterance's speech, in milliseconds from the beginning of the stream. Approximate. `0` with endpointing off. |
| `utterance.duration_ms` | integer | Duration of this utterance's speech in milliseconds, excluding the pause that ended it. Approximate. With endpointing off, covers the whole stream including any leading and trailing silence. |
| `utterance.speaker` | integer \| null | Present only with `diarize=true`. Zero-based index of the speaker who produced this utterance. `null` when no speaker could be determined. |
| `utterance.n_speakers` | integer | Present only with `diarize=true`. Distinct speakers detected on the connection so far. |
| `utterance.speakers_capped` | boolean | Present only with `diarize=true`. `true` once every speaker slot is in use. |
| `utterance.speaker_purity` | number | Present only with `diarize=true`. Proportion of this utterance's speech attributed to `speaker`, from 0 to 1. |
With `diarize=false` no speaker field appears in any message, and the payloads are byte-identical to a non-diarized connection.
### `done`
Sent immediately after the final `utterance`. Signals stream completion. The connection closes shortly after.
```json theme={null}
{
"type": "done",
"duration_ms": 14253
}
```
| Field | Type | Description |
| - | - | - |
| `type` | string | Always `"done"` |
| `duration_ms` | integer | Total audio duration in milliseconds |
### `error`
Sent if something goes wrong. The connection closes after this message. No further messages follow an `error`.
```json theme={null}
{
"type": "error",
"error": "Invalid audio_format='xyz'."
}
```
| Field | Type | Description |
| - | - | - |
| `type` | string | Always `"error"` |
| `error` | string | Human-readable description of the error |
Diarization adds two error texts, both closing with `1003`:
```json theme={null}
{ "type": "error", "error": "Invalid diarize='yes'. Expected 'true' or 'false'." }
```
```json theme={null}
{ "type": "error", "error": "diarize=true is not available on this endpoint." }
```
The second means speaker labelling is not enabled on the serving instance. Retrying does not clear it; reconnect without `diarize`.
## WebSocket close codes
| Code | Meaning |
| - | - |
| `1000` | Normal closure after the `done` message |
| `1003` | Invalid query parameters — missing `audio_format`, invalid `sample_rate`, invalid `endpointing`, invalid `diarize`, unsupported value, etc. Also returned when `diarize=true` is requested but speaker labelling is not enabled on the serving instance. |
| `1011` | Internal server error. An `error` message is sent before the close. |
| `1013` | The service is temporarily at capacity. Not an account or billing condition — retry the connection. Connections requesting `diarize=true` have a lower concurrency ceiling than plain transcription, so a diarized connection can be refused while a plain one succeeds. |
| `4001` | The `api_key` query parameter is missing or invalid |
| `4002` | Audio bytes did not match the declared raw PCM format, or the audio could not be decoded mid-stream |
| `4003` | The request is not permitted |
| `4004` | The API key does not have access to this model |
| `4029` | Insufficient credits |
| `4030` | Concurrent request limit reached |
| `4031` | Monthly usage limit reached |
## Rate limits
* Concurrent connection limits apply per model; exceeding them rejects the handshake with close code `4030`.
See [Authentication and rate limits](/guides/authentication) for retry guidance.
## Examples
```python theme={null}
import asyncio
import json
import os
import websockets
API_KEY = os.environ["MODULATE_API_KEY"]
AUDIO_FILE = "recording.ogg"
CHUNK_SIZE = 8192
async def transcribe():
url = (
f"wss://platform.modulate.ai/api/velma-2-stt-streaming-english-v2"
f"?api_key={API_KEY}&audio_format=ogg"
)
async with websockets.connect(url, max_size=None) as ws:
async def send_audio():
with open(AUDIO_FILE, "rb") as f:
while chunk := f.read(CHUNK_SIZE):
await ws.send(chunk)
await ws.send("") # signal end-of-stream
send_task = asyncio.create_task(send_audio())
try:
async for msg in ws:
data = json.loads(msg)
if data["type"] == "partial_utterance":
# Replace any previously displayed partial — not a delta
print(f"\r[partial] {data['partial_utterance']['text']}", end="", flush=True)
elif data["type"] == "utterance":
u = data["utterance"]
# "speaker" is present only with diarize=true; None means undetermined
who = f" speaker {u['speaker']}" if u.get("speaker") is not None else ""
print(f"\n[final{who}] {u['text']}")
elif data["type"] == "done":
print(f"\nDone. Duration: {data['duration_ms']}ms")
break
elif data["type"] == "error":
print(f"\nError: {data['error']}")
break
finally:
if not send_task.done():
send_task.cancel()
asyncio.run(transcribe())
```
```javascript theme={null}
const WebSocket = require("ws");
const fs = require("fs");
const API_KEY = process.env.MODULATE_API_KEY;
const AUDIO_FILE = "recording.ogg";
const CHUNK_SIZE = 8192;
const url = new URL("wss://platform.modulate.ai/api/velma-2-stt-streaming-english-v2");
url.searchParams.set("api_key", API_KEY);
url.searchParams.set("audio_format", "ogg");
const ws = new WebSocket(url.toString());
ws.on("open", () => {
const stream = fs.createReadStream(AUDIO_FILE, { highWaterMark: CHUNK_SIZE });
stream.on("data", (chunk) => ws.send(chunk));
stream.on("end", () => ws.send("")); // signal end-of-stream
});
ws.on("message", (data) => {
const msg = JSON.parse(data.toString());
if (msg.type === "partial_utterance") {
// Each partial is the full transcript so far — replace, don't append
process.stdout.write(`\r[partial] ${msg.partial_utterance.text}`);
} else if (msg.type === "utterance") {
// "speaker" is present only with diarize=true; null means undetermined
const who = msg.utterance.speaker != null ? ` speaker ${msg.utterance.speaker}` : "";
console.log(`\n[final${who}] ${msg.utterance.text}`);
} else if (msg.type === "done") {
console.log(`\nDone. Duration: ${msg.duration_ms}ms`);
ws.close();
} else if (msg.type === "error") {
console.error("\nError:", msg.error);
ws.close();
}
});
ws.on("error", (err) => console.error("WebSocket error:", err.message));
ws.on("close", (code) => console.log(`Connection closed: ${code}`));
```
The examples above work unchanged with endpointing and diarization: append `&endpointing=true` to the URL and the same message loop prints one `[final]` line per pause instead of a single one at the end; add `&diarize=true` as well and each of those lines is prefixed with its speaker label.
WebSocket APIs cannot be tested with cURL. For command-line testing, use [`websocat`](https://github.com/vi/websocat).
## Related
* [Which API should I use?](/guides/which-api) — when Streaming v2 is the right choice vs STT Streaming or batch
* [STT Streaming](/api-reference/stt/streaming) — multilingual streaming with speaker diarization and enrichments
* [Authentication and rate limits](/guides/authentication)
## AsyncAPI
````yaml api/velma_2_stt_streaming_english_v2.yaml velma2SttStreamingEnglishV2
id: velma2SttStreamingEnglishV2
title: Velma2 stt streaming english v2
description: |
Establish a WebSocket connection for low-latency English transcription.
The server emits partial transcripts as audio streams in, and a final
`utterance` at end-of-stream. With `endpointing=true`, speech is
additionally segmented at pauses into one final `utterance` per segment.
With `diarize=true`, each final `utterance` also carries a numerical
speaker ID for its dominant speaker.
## Connection Flow
1. Connect with `api_key`, `audio_format`, (for raw formats) `sample_rate` +
`num_channels`, and optionally `endpointing` and `diarize`
2. Connection accepted if authentication succeeds and limits are not
exceeded
3. Stream audio data as binary WebSocket frames
4. Receive `partial_utterance` results every ~1.5 seconds while transcription
is in progress (each is the complete transcript of the current segment
so far - REPLACE, don't append)
5. With `endpointing=true`, receive a final `utterance` message each time a
segment of speech ends (at a pause), carrying that segment's text, start
time, and duration
6. Send empty text frame (`""`) when the audio stream ends
7. Receive the final `utterance` for any content not already finalized
at a pause (the whole stream with endpointing off; with
`endpointing=true`, this step is skipped when the last segment was
already finalized mid-stream)
8. Receive `done` message with total audio duration
9. Connection closes automatically
## Partial and Segment Semantics
By default the connection produces a single final `utterance` at
end-of-stream, covering the whole stream (`start_ms=0`), and each
`partial_utterance` reflects the whole stream so far. With
`endpointing=true`, speech is segmented into utterances at pauses: each
`partial_utterance` reflects the CURRENT segment, and a final `utterance`
is emitted each time a segment ends, after which the following
`partial_utterance` messages belong to the next segment; the full
transcript is the concatenation of the final `utterance` texts. In both
modes a `partial_utterance` is the complete transcript of its scope so far
(not a delta), and the last few words may be revised in subsequent partials
as the model sees more audio context, so clients should replace the
displayed partial text rather than append.
## Speaker Labels
With `diarize=true`, each final `utterance` carries `speaker` - a
zero-based index that is stable for the lifetime of the connection.
It identifies speakers only WITHIN one connection: the same person on
a new connection may get a different index, and indices are not
identities that can be matched across connections or sessions.
Up to ten distinct speakers are labeled per connection. Past that,
further speakers are merged into existing labels rather than reported
separately - their speech is still transcribed, but its `speaker` will
be wrong. `speakers_capped` is `true` once all slots are in use, which
is the only signal that labels may be unreliable; clients doing
anything consequential with speaker identity should check it, alongside
`n_speakers` for how many distinct speakers the connection has
detected so far.
Segments are cut at pauses regardless of `diarize`: a segment that
spans a fast speaker exchange with no pause carries the speaker who
spoke most of it, and `speaker_purity` below `1.0` is the signal that
the single label describes only the majority of the segment.
Speaker labels are attached to final `utterance` messages only, never
to `partial_utterance`. Diarization has no back-patching: a label is
assigned when the utterance is finalized and is not revised
afterwards. Very early utterances in a conversation may report
`speaker: null` while a new voice has not yet accumulated enough
evidence to be assigned a label.
With `diarize=true` and `endpointing=false` there is only one final
`utterance` for the whole stream, so a single `speaker` cannot
describe it. In that mode read `n_speakers` - how many distinct
speakers were detected on the connection - and treat `speaker` as
indicative only.
## Text Formatting
Both `partial_utterance` and `utterance` payloads contain text that
has already been auto-capitalized and auto-punctuated by the model.
Clients can render the `text` field directly without additional
casing or punctuation normalization. Note that in `partial_utterance`
messages, both capitalization and punctuation in the final words may
be revised in subsequent partials as the model sees more audio
context.
## Error Handling
If an error occurs during transcription, the server sends an error
message and closes the connection. Connection rejections during
handshake use WebSocket close codes (see below).
## WebSocket Close Codes
- `1000`: Normal closure after the `done` message, or after the
connection completes.
- `1003`: Invalid query parameters (missing `audio_format`, invalid
`sample_rate`, invalid `endpointing`, invalid `diarize`, unsupported
value, etc.). Also returned when `diarize=true` is requested but
speaker labelling is not enabled on the serving instance.
- `1011`: Internal server error. An `error` message is sent before the
close.
- `1013`: The service is temporarily at capacity and could not accept
the connection. An `error` message is sent before the close with the
text "The service is temporarily unavailable. Please try again later."
This is not an account or billing condition - retry the connection.
Note that connections requesting `diarize=true` have a lower
concurrency ceiling than plain transcription, so a `diarize=true`
connection may be refused while a plain one succeeds.
- `4001`: The `api_key` query parameter is missing or invalid.
- `4002`: Audio data didn't match the declared raw format (sent
container bytes but declared a raw type), or the audio could not
be decoded mid-stream. The `error` message is
"Audio data does not match the declared audio_format." for a declared
raw format mismatch, or
"The audio could not be processed. It may be corrupted or in an unsupported format."
for a mid-stream decode failure.
- `4003`: The request is not permitted.
- `4004`: The API key does not have access to this model.
- `4029`: The request could not be completed due to insufficient
credits.
- `4030`: Concurrent request limit reached.
- `4031`: Monthly usage limit reached.
servers:
- id: production
protocol: wss
host: platform.modulate.ai
bindings: []
variables: []
address: /api/velma-2-stt-streaming-english-v2
parameters: []
bindings:
- protocol: ws
version: latest
value:
query:
type: object
required:
- api_key
- audio_format
properties:
api_key:
type: string
description: |
Your API key for authentication. Must be provided as a query
parameter when establishing the WebSocket connection.
audio_format:
type: string
description: |
Container or raw format of the audio data the client will send.
Required.
For container formats, the audio header determines the sample rate
and channel count, and `sample_rate` / `num_channels` query
parameters are ignored. The MP4-family values (`mp4`, `m4a`,
`m4b`, `m4r`, `m4v`, `3gp`, `3gpp`, `3ga`, `3g2`, `f4a`,
`f4b`) must be sent in a streamable layout;
otherwise the connection ends with an audio-processing error.
For raw formats, `sample_rate` and `num_channels` are required.
The raw `g722` and `vox` formats are additionally mono-only
(`num_channels` must be `1`).
For lowest end-to-end latency, send
`audio_format=s16le&sample_rate=16000&num_channels=1`.
enum:
- s8
- s16le
- s16be
- s24le
- s24be
- s32le
- s32be
- u8
- u16le
- u16be
- u24le
- u24be
- u32le
- u32be
- f32le
- f32be
- f64le
- f64be
- mulaw
- alaw
- g722
- vox
- 3g2
- 3ga
- 3gp
- 3gpp
- 8svx
- aa3
- aac
- ac3
- act
- adts
- aif
- aifc
- aiff
- amb
- amr
- asf
- at3
- au
- avr
- awb
- bwf
- c2
- caf
- dss
- dts
- dtshd
- eac3
- ec3
- f4a
- f4b
- flac
- gsm
- iff
- m2a
- m2ts
- m4a
- m4b
- m4r
- m4v
- mka
- mkv
- mlp
- mp+
- mp1
- mp2
- mp3
- mp4
- mpa
- mpc
- mpga
- mpp
- mts
- oga
- ogg
- ogx
- oma
- omg
- opus
- paf
- pvf
- qcp
- ra
- rf64
- rm
- rmvb
- snd
- svx
- thd
- ts
- tta
- voc
- vqf
- w64
- wav
- wave
- weba
- webm
- wma
- wmv
sample_rate:
type: integer
description: |
Source sample rate of the audio data in Hz. Required when
`audio_format` is a raw type; ignored for container formats.
enum:
- 8000
- 11025
- 16000
- 22050
- 32000
- 44100
- 48000
- 96000
num_channels:
type: integer
description: |
Number of audio channels in the source data (1-8). Required when
`audio_format` is a raw type; ignored for container formats.
minimum: 1
maximum: 8
endpointing:
type: boolean
default: false
description: >
Whether to segment speech into utterances at pauses. Optional;
defaults to `false`.
When `false` (the default), the connection produces a single
final `utterance` at end-of-stream covering the whole stream
(`start_ms=0`), and each `partial_utterance` reflects the whole
stream so far.
When `true`, trailing silence after speech closes the current
utterance and starts a new one: the server emits one final
`utterance` per speech segment (each with its own `start_ms` /
`duration_ms`), and each `partial_utterance` reflects the current
segment. The full transcript is the concatenation of the final
`utterance` texts.
In both modes every connection ends with at least one final
`utterance`; a stream containing no speech yields a single
final with empty `text`.
Independent of `diarize`; all combinations are valid.
Example:
`audio_format=s16le&sample_rate=16000&num_channels=1&endpointing=true`.
diarize:
type: boolean
default: false
description: >
Whether to label which speaker produced each utterance.
Optional; defaults to `false`.
When `false` (the default), no speaker fields appear in any
message and the payloads are byte-identical to a
non-diarized connection.
When `true`, each final `utterance` additionally carries
`speaker`, `n_speakers`, `speakers_capped`, and
`speaker_purity`; see "Speaker Labels" above for the speaker
capacity and the limits of the `speaker` index, and the
close-code list for the handshake rejections and capacity
behavior specific to diarized connections.
Independent of `endpointing`; all combinations are valid.
Speaker labels are only meaningful per utterance, however,
so `diarize=true&endpointing=false` yields one whole-stream
utterance where `n_speakers` is the useful field rather
than `speaker`.
Example:
`audio_format=s16le&sample_rate=16000&num_channels=1&endpointing=true&diarize=true`.
schemaProperties:
- name: query
type: object
required: false
properties:
- name: api_key
type: string
description: |
Your API key for authentication. Must be provided as a query
parameter when establishing the WebSocket connection.
required: true
- name: audio_format
type: string
description: |
Container or raw format of the audio data the client will send.
Required.
For container formats, the audio header determines the sample rate
and channel count, and `sample_rate` / `num_channels` query
parameters are ignored. The MP4-family values (`mp4`, `m4a`,
`m4b`, `m4r`, `m4v`, `3gp`, `3gpp`, `3ga`, `3g2`, `f4a`,
`f4b`) must be sent in a streamable layout;
otherwise the connection ends with an audio-processing error.
For raw formats, `sample_rate` and `num_channels` are required.
The raw `g722` and `vox` formats are additionally mono-only
(`num_channels` must be `1`).
For lowest end-to-end latency, send
`audio_format=s16le&sample_rate=16000&num_channels=1`.
enumValues:
- s8
- s16le
- s16be
- s24le
- s24be
- s32le
- s32be
- u8
- u16le
- u16be
- u24le
- u24be
- u32le
- u32be
- f32le
- f32be
- f64le
- f64be
- mulaw
- alaw
- g722
- vox
- 3g2
- 3ga
- 3gp
- 3gpp
- 8svx
- aa3
- aac
- ac3
- act
- adts
- aif
- aifc
- aiff
- amb
- amr
- asf
- at3
- au
- avr
- awb
- bwf
- c2
- caf
- dss
- dts
- dtshd
- eac3
- ec3
- f4a
- f4b
- flac
- gsm
- iff
- m2a
- m2ts
- m4a
- m4b
- m4r
- m4v
- mka
- mkv
- mlp
- mp+
- mp1
- mp2
- mp3
- mp4
- mpa
- mpc
- mpga
- mpp
- mts
- oga
- ogg
- ogx
- oma
- omg
- opus
- paf
- pvf
- qcp
- ra
- rf64
- rm
- rmvb
- snd
- svx
- thd
- ts
- tta
- voc
- vqf
- w64
- wav
- wave
- weba
- webm
- wma
- wmv
required: true
- name: sample_rate
type: integer
description: |
Source sample rate of the audio data in Hz. Required when
`audio_format` is a raw type; ignored for container formats.
enumValues:
- 8000
- 11025
- 16000
- 22050
- 32000
- 44100
- 48000
- 96000
required: false
- name: num_channels
type: integer
description: |
Number of audio channels in the source data (1-8). Required when
`audio_format` is a raw type; ignored for container formats.
required: false
- name: endpointing
type: boolean
description: >
Whether to segment speech into utterances at pauses. Optional;
defaults to `false`.
When `false` (the default), the connection produces a single
final `utterance` at end-of-stream covering the whole stream
(`start_ms=0`), and each `partial_utterance` reflects the whole
stream so far.
When `true`, trailing silence after speech closes the current
utterance and starts a new one: the server emits one final
`utterance` per speech segment (each with its own `start_ms` /
`duration_ms`), and each `partial_utterance` reflects the current
segment. The full transcript is the concatenation of the final
`utterance` texts.
In both modes every connection ends with at least one final
`utterance`; a stream containing no speech yields a single
final with empty `text`.
Independent of `diarize`; all combinations are valid.
Example:
`audio_format=s16le&sample_rate=16000&num_channels=1&endpointing=true`.
required: false
- name: diarize
type: boolean
description: >
Whether to label which speaker produced each utterance.
Optional; defaults to `false`.
When `false` (the default), no speaker fields appear in any
message and the payloads are byte-identical to a
non-diarized connection.
When `true`, each final `utterance` additionally carries
`speaker`, `n_speakers`, `speakers_capped`, and
`speaker_purity`; see "Speaker Labels" above for the speaker
capacity and the limits of the `speaker` index, and the
close-code list for the handshake rejections and capacity
behavior specific to diarized connections.
Independent of `endpointing`; all combinations are valid.
Speaker labels are only meaningful per utterance, however,
so `diarize=true&endpointing=false` yields one whole-stream
utterance where `n_speakers` is the useful field rather
than `speaker`.
Example:
`audio_format=s16le&sample_rate=16000&num_channels=1&endpointing=true&diarize=true`.
required: false
operations:
- &ref_2
id: streamAudio
title: Stream audio
description: Stream audio for low-latency English transcription
type: send
messages:
- &ref_7
id: audioChunk
contentType: application/octet-stream
payload:
- type: string
format: binary
x-parser-schema-id:
name: Audio Data Chunk
description: |
Raw audio data chunk for transcription. Sent zero or more times
by the client during the stream lifetime. Each chunk's bytes are
decoded by the server according to the connection's
`audio_format` query parameter.
headers: []
jsonPayloadSchema:
type: string
format: binary
x-parser-schema-id:
title: Audio Data Chunk
description: |
Raw audio data chunk for transcription. Sent zero or more times
by the client during the stream lifetime. Each chunk's bytes are
decoded by the server according to the connection's
`audio_format` query parameter.
example: '{}'
bindings: []
extensions:
- id: x-parser-unique-object-id
value: audioChunk
- &ref_8
id: endOfStream
contentType: text/plain
payload:
- type: string
const: ''
x-parser-schema-id:
name: End of Audio Stream
description: |
Empty text frame signaling end of audio. Sent exactly once by the
client after the last audio chunk. The server responds with a
final `utterance` message followed by a `done` message, then
closes the connection.
headers: []
jsonPayloadSchema:
type: string
const: ''
x-parser-schema-id:
title: End of Audio Stream
description: |
Empty text frame signaling end of audio. Sent exactly once by the
client after the last audio chunk. The server responds with a
final `utterance` message followed by a `done` message, then
closes the connection.
example: '""'
bindings: []
extensions:
- id: x-parser-unique-object-id
value: endOfStream
bindings: []
extensions: &ref_0
- id: x-parser-unique-object-id
value: velma2SttStreamingEnglishV2
- &ref_1
id: receiveTranscription
title: Receive transcription
description: Receive partial and final transcription results from the server
type: receive
messages:
- &ref_3
id: partialUtteranceResult
contentType: application/json
payload:
- name: Partial Utterance Result
description: >
In-progress transcript snapshot for the CURRENT speech segment.
Sent
zero or more times per segment, roughly every 1.5 seconds. The
`text`
field contains the current segment's transcript so far - not a
delta,
and not the whole stream - with auto-capitalization and
auto-punctuation already applied. Clients should REPLACE the
current
segment's previously displayed partial text, not append. Partials
never carry speaker labels, even with `diarize=true`.
type: object
properties:
- name: type
type: string
description: partial_utterance
required: true
- name: partial_utterance
type: object
required: true
properties:
- name: text
type: string
description: >
The complete transcript built so far for the current scope
-
the whole connection by default, or the current speech
segment
when `endpointing=true` - with auto-capitalization
(sentence
case, proper nouns, "I", etc.) and auto-punctuation
(commas,
periods, question marks, etc.) already applied by the
model.
This is NOT a delta from the previous partial - clients
should
display this as a replacement for any previously displayed
partial text. The last few words, including their
capitalization and punctuation, may be revised in
subsequent
partials as the model sees more audio context.
required: true
- name: is_final
type: boolean
description: |
Always `false` for `partial_utterance` messages. Use the
message `type` field to distinguish partial from final.
required: true
headers: []
jsonPayloadSchema:
type: object
required:
- type
- partial_utterance
properties:
type:
type: string
const: partial_utterance
x-parser-schema-id:
partial_utterance:
type: object
required:
- text
- is_final
properties:
text:
type: string
description: >
The complete transcript built so far for the current scope -
the whole connection by default, or the current speech
segment
when `endpointing=true` - with auto-capitalization (sentence
case, proper nouns, "I", etc.) and auto-punctuation (commas,
periods, question marks, etc.) already applied by the model.
This is NOT a delta from the previous partial - clients
should
display this as a replacement for any previously displayed
partial text. The last few words, including their
capitalization and punctuation, may be revised in subsequent
partials as the model sees more audio context.
example: Hello, how are you
x-parser-schema-id:
is_final:
type: boolean
const: false
description: |
Always `false` for `partial_utterance` messages. Use the
message `type` field to distinguish partial from final.
x-parser-schema-id:
x-parser-schema-id: StreamingPartialUtterance
x-parser-schema-id:
title: Partial Utterance Result
description: |
In-progress transcript snapshot for the CURRENT speech segment. Sent
zero or more times per segment, roughly every 1.5 seconds. The `text`
field contains the current segment's transcript so far - not a delta,
and not the whole stream - with auto-capitalization and
auto-punctuation already applied. Clients should REPLACE the current
segment's previously displayed partial text, not append. Partials
never carry speaker labels, even with `diarize=true`.
example: |-
{
"type": "partial_utterance",
"partial_utterance": {
"text": "Hello, how are you",
"is_final": false
}
}
bindings: []
extensions:
- id: x-parser-unique-object-id
value: partialUtteranceResult
- &ref_4
id: utteranceResult
contentType: application/json
payload:
- name: Final Utterance Result
description: >
The final transcript for one segment of speech, with
auto-capitalization and auto-punctuation applied, plus the
segment's
start time and duration. With `diarize=true` it also carries the
speaker who produced it. Sent once per segment (when the segment
ends
at a pause, and once more for the last segment after
end-of-stream),
and supersedes the `partial_utterance` messages for that segment.
The
full transcript is the concatenation of these texts in order.
type: object
properties:
- name: type
type: string
description: utterance
required: true
- name: utterance
type: object
required: true
properties:
- name: text
type: string
description: >
The final transcribed text for one segment of speech, with
auto-capitalization (sentence case, proper nouns, "I",
etc.) and
auto-punctuation (commas, periods, question marks, etc.)
already
applied by the model. This is the authoritative transcript
for the
segment and will not be revised. The full transcript is
the
concatenation of the `text` of every `utterance` in order.
required: true
- name: is_final
type: boolean
description: >
Always `true` for `utterance` messages. Use the message
`type`
field to distinguish partial from final.
required: true
- name: start_ms
type: integer
description: >
Start time of this segment, in milliseconds from the
beginning of
the audio stream. Approximate: speech onset is detected at
the
server's internal processing granularity, so the start may
differ
from the first spoken word by a fraction of a second.
required: true
- name: duration_ms
type: integer
description: >
Duration of this segment's speech, in milliseconds,
excluding the
trailing pause that ended the segment. Approximate, with
the same
boundary granularity as `start_ms`. With endpointing off,
the
single final utterance instead covers the whole stream,
including
any leading and trailing silence.
required: true
- name: speaker
type: integer
description: >
Present only when `diarize=true`. Zero-based index of the
speaker who produced this utterance, stable for the
lifetime of
the connection but meaningless outside it - the same
person on a
new connection may receive a different index.
`null` means diarization was requested but no speaker
could be
determined for this utterance, which happens for very
short or
silent segments and for a brand-new voice that has not yet
accumulated enough evidence to be assigned a label. That
is
distinct from the field being absent, which means
diarization
was not requested at all.
With `endpointing=false` there is a single whole-stream
utterance that may contain several speakers, so this field
is
indicative only in that mode - read `n_speakers` instead.
required: false
- name: n_speakers
type: integer
description: >
Present only when `diarize=true`. Count of distinct
speakers
detected on the connection so far. This is the meaningful
speaker signal when `endpointing=false`, where one
`speaker`
index cannot describe a whole multi-speaker stream.
required: false
- name: speakers_capped
type: boolean
description: >
Present only when `diarize=true`. `true` once every
speaker slot
is in use, meaning the connection may contain more
speakers than
can be labeled. Beyond the cap, additional speakers are
merged
into existing labels rather than reported separately -
their
speech is still transcribed, but `speaker` will be wrong
for
some utterances and there is no other signal that this
happened.
Clients that act on speaker identity should check this
field.
required: false
- name: speaker_purity
type: number
description: >
Present only when `diarize=true`. The proportion of this
utterance's speech attributed to the speaker named in
`speaker`. `1.0` means the utterance was cleanly
single-speaker.
A lower value means the utterance spans a speaker change
that
the pause-based segmentation did not split, so its single
`speaker` label describes only the majority of it.
required: false
headers: []
jsonPayloadSchema:
type: object
required:
- type
- utterance
properties:
type:
type: string
const: utterance
x-parser-schema-id:
utterance:
type: object
required:
- text
- is_final
- start_ms
- duration_ms
properties:
text:
type: string
description: >
The final transcribed text for one segment of speech, with
auto-capitalization (sentence case, proper nouns, "I", etc.)
and
auto-punctuation (commas, periods, question marks, etc.)
already
applied by the model. This is the authoritative transcript
for the
segment and will not be revised. The full transcript is the
concatenation of the `text` of every `utterance` in order.
example: Hello, how are you doing today?
x-parser-schema-id:
is_final:
type: boolean
const: true
description: >
Always `true` for `utterance` messages. Use the message
`type`
field to distinguish partial from final.
x-parser-schema-id:
start_ms:
type: integer
minimum: 0
description: >
Start time of this segment, in milliseconds from the
beginning of
the audio stream. Approximate: speech onset is detected at
the
server's internal processing granularity, so the start may
differ
from the first spoken word by a fraction of a second.
example: 0
x-parser-schema-id:
duration_ms:
type: integer
minimum: 0
description: >
Duration of this segment's speech, in milliseconds,
excluding the
trailing pause that ended the segment. Approximate, with the
same
boundary granularity as `start_ms`. With endpointing off,
the
single final utterance instead covers the whole stream,
including
any leading and trailing silence.
example: 2360
x-parser-schema-id:
speaker:
type: integer
nullable: true
minimum: 0
description: >
Present only when `diarize=true`. Zero-based index of the
speaker who produced this utterance, stable for the lifetime
of
the connection but meaningless outside it - the same person
on a
new connection may receive a different index.
`null` means diarization was requested but no speaker could
be
determined for this utterance, which happens for very short
or
silent segments and for a brand-new voice that has not yet
accumulated enough evidence to be assigned a label. That is
distinct from the field being absent, which means
diarization
was not requested at all.
With `endpointing=false` there is a single whole-stream
utterance that may contain several speakers, so this field
is
indicative only in that mode - read `n_speakers` instead.
example: 0
x-parser-schema-id:
n_speakers:
type: integer
minimum: 0
description: |
Present only when `diarize=true`. Count of distinct speakers
detected on the connection so far. This is the meaningful
speaker signal when `endpointing=false`, where one `speaker`
index cannot describe a whole multi-speaker stream.
example: 2
x-parser-schema-id:
speakers_capped:
type: boolean
description: >
Present only when `diarize=true`. `true` once every speaker
slot
is in use, meaning the connection may contain more speakers
than
can be labeled. Beyond the cap, additional speakers are
merged
into existing labels rather than reported separately - their
speech is still transcribed, but `speaker` will be wrong for
some utterances and there is no other signal that this
happened.
Clients that act on speaker identity should check this
field.
example: false
x-parser-schema-id:
speaker_purity:
type: number
minimum: 0
maximum: 1
description: >
Present only when `diarize=true`. The proportion of this
utterance's speech attributed to the speaker named in
`speaker`. `1.0` means the utterance was cleanly
single-speaker.
A lower value means the utterance spans a speaker change
that
the pause-based segmentation did not split, so its single
`speaker` label describes only the majority of it.
example: 1
x-parser-schema-id:
x-parser-schema-id: StreamingUtterance
x-parser-schema-id:
title: Final Utterance Result
description: |
The final transcript for one segment of speech, with
auto-capitalization and auto-punctuation applied, plus the segment's
start time and duration. With `diarize=true` it also carries the
speaker who produced it. Sent once per segment (when the segment ends
at a pause, and once more for the last segment after end-of-stream),
and supersedes the `partial_utterance` messages for that segment. The
full transcript is the concatenation of these texts in order.
example: |-
{
"type": "utterance",
"utterance": {
"text": "Hello, how are you doing today?",
"is_final": true,
"start_ms": 0,
"duration_ms": 2360
}
}
bindings: []
extensions:
- id: x-parser-unique-object-id
value: utteranceResult
- &ref_5
id: errorResult
contentType: application/json
payload:
- name: Error
description: |
An error occurred during transcription. Sent at most once per
connection; the WebSocket closes shortly after. No further
messages follow an `error`.
type: object
properties:
- name: type
type: string
description: error
required: true
- name: error
type: string
description: Human-readable error message
required: true
headers: []
jsonPayloadSchema:
type: object
required:
- type
- error
properties:
type:
type: string
const: error
x-parser-schema-id:
error:
type: string
description: Human-readable error message
x-parser-schema-id:
x-parser-schema-id:
title: Error
description: |
An error occurred during transcription. Sent at most once per
connection; the WebSocket closes shortly after. No further
messages follow an `error`.
example: |-
{
"type": "error",
"error": "Internal server error"
}
bindings: []
extensions:
- id: x-parser-unique-object-id
value: errorResult
- &ref_6
id: doneResult
contentType: application/json
payload:
- name: Transcription Complete
description: >
Transcription completed successfully. Sent exactly once,
immediately
after the final `utterance` message. After this message the server
closes the WebSocket.
type: object
properties:
- name: type
type: string
description: done
required: true
- name: duration_ms
type: integer
description: Total duration of audio processed in milliseconds
required: true
headers: []
jsonPayloadSchema:
type: object
required:
- type
- duration_ms
properties:
type:
type: string
const: done
x-parser-schema-id:
duration_ms:
type: integer
minimum: 0
description: Total duration of audio processed in milliseconds
x-parser-schema-id:
x-parser-schema-id:
title: Transcription Complete
description: |
Transcription completed successfully. Sent exactly once, immediately
after the final `utterance` message. After this message the server
closes the WebSocket.
example: |-
{
"type": "done",
"duration_ms": 14253
}
bindings: []
extensions:
- id: x-parser-unique-object-id
value: doneResult
bindings: []
extensions: *ref_0
sendOperations:
- *ref_1
receiveOperations:
- *ref_2
sendMessages:
- *ref_3
- *ref_4
- *ref_5
- *ref_6
receiveMessages:
- *ref_7
- *ref_8
extensions:
- id: x-parser-unique-object-id
value: velma2SttStreamingEnglishV2
securitySchemes: []
````
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.