> ## Documentation Index > Fetch the complete documentation index at: https://docs.modulate.ai/llms.txt > Use this file to discover all available pages before exploring further. # Multilingual Fast Transcription Streaming > Low-latency multilingual speech-to-text over WebSocket. Covers 25 languages with automatic language identification, with optional speaker labels. Low-latency multilingual speech-to-text over WebSocket. The spoken language is identified automatically, so the client declares no language and uses no per-language endpoint. Interim partial transcripts stream while audio arrives, and a final `utterance` is emitted for each segment of speech. This endpoint produces no emotion, accent, or PII/PHI data. Use [STT Streaming](/api-reference/stt/streaming) if you need those. ## Endpoint ```text theme={null} wss://platform.modulate.ai/api/velma-2-stt-streaming-multilingual-vfast ``` ## Authentication The API key goes in the query string at connection time, not in a header. See [Authentication and rate limits](/guides/authentication). ## Supported audio formats **Container formats** — `sample_rate` and `num_channels` are ignored if supplied: `mp3`, `wav`, `flac`, `m4a`, `mp4`, `ogg`, `opus`, `webm`, `aac`, `aiff`, `wma`, `amr`, and `au`, plus 66 others `3g2`, `3ga`, `3gp`, `3gpp`, `8svx`, `aa3`, `aac`, `ac3`, `act`, `adts`, `aif`, `aifc`, `aiff`, `amb`, `amr`, `asf`, `at3`, `au`, `avr`, `awb`, `bwf`, `c2`, `caf`, `dss`, `dts`, `dtshd`, `eac3`, `ec3`, `f4a`, `f4b`, `flac`, `gsm`, `iff`, `m2a`, `m2ts`, `m4a`, `m4b`, `m4r`, `m4v`, `mka`, `mkv`, `mlp`, `mp+`, `mp1`, `mp2`, `mp3`, `mp4`, `mpa`, `mpc`, `mpga`, `mpp`, `mts`, `oga`, `ogg`, `ogx`, `oma`, `omg`, `opus`, `paf`, `pvf`, `qcp`, `ra`, `rf64`, `rm`, `rmvb`, `snd`, `svx`, `thd`, `ts`, `tta`, `voc`, `vqf`, `w64`, `wav`, `wave`, `weba`, `webm`, `wma`, `wmv` The MP4-family values (`mp4`, `m4a`, `m4b`, `m4r`, `m4v`, `3gp`, `3gpp`, `3ga`, `3g2`, `f4a`, `f4b`) must be sent in a streamable layout; otherwise the connection ends with an audio-processing error. **Raw PCM formats** — `sample_rate` and `num_channels` are required: `s8`, `s16le`, `s16be`, `s24le`, `s24be`, `s32le`, `s32be`, `u8`, `u16le`, `u16be`, `u24le`, `u24be`, `u32le`, `u32be`, `f32le`, `f32be`, `f64le`, `f64be`, `mulaw`, `alaw`, `g722`, `vox` `g722` and `vox` are mono-only: `num_channels` must be `1`. **Valid sample rates:** 8000, 11025, 16000, 22050, 32000, 44100, 48000, 96000 For lowest end-to-end latency, send `audio_format=s16le&sample_rate=16000&num_channels=1`. ## Query parameters | Parameter | Required | Description | | - | - | - | | `api_key` | Yes | Your API key | | `audio_format` | Yes | Container or raw PCM format of the audio you will stream | | `sample_rate` | Raw PCM only | Source sample rate in Hz. Required for a raw format; ignored for container formats. | | `num_channels` | Raw PCM only | Number of audio channels (1–8). Required for a raw format; ignored for container formats. | | `endpointing` | No | `true` or `false` (default `false`). When `true`, speech is segmented into utterances at pauses. | | `diarize` | No | `true` or `false` (default `false`). When `true`, each final `utterance` carries speaker fields — see [Speaker labels](#speaker-labels). | `diarize` and `endpointing` are independent; all four combinations are valid. **Worth knowing:** `diarize=true` substantially increases time-to-first-text. Speaker decisions are made over a multi-second window, so the first `partial_utterance` arrives several seconds later than on a non-diarized connection, and the gap widens under concurrency. Transcript accuracy is unaffected - the cost is latency, not quality. If you are rendering live captions, either size the interface for this or run a second, non-diarized connection for the display text. ## Language identification The spoken language is identified automatically across these 25 languages: Bulgarian, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hungarian, Italian, Latvian, Lithuanian, Maltese, Polish, Portuguese, Romanian, Russian, Slovak, Slovenian, Spanish, Swedish, Ukrainian. Each `utterance` is transcribed in the language spoken, with auto-capitalization and auto-punctuation already applied. There is no `language` query parameter, and recognition quality varies by language. **Worth knowing:** this set is smaller and narrower than it sounds. Multilingual Transcription covers 100 languages; these 25 are all European, so Chinese, Japanese, Arabic, Hindi and Korean are not among them. Audio outside the set is still transcribed, but not reliably. **Worth knowing:** no message carries a language field. The transcript is in the detected language, but the detection itself is not reported, so a client cannot branch on it. Multilingual Fast Transcription (batch) does return `language`; the streaming endpoint does not. Use [Language Detection](/get-started/language-detection) when the code itself is needed. ## Speaker labels With `diarize=true`, each final `utterance` carries `speaker`, a zero-based index identifying who produced it. Partials never carry it, and a label is assigned when an utterance is finalized and never revised. The index is stable for one connection and meaningless outside it: the same person on a new connection may receive a different index, so it is not an identity that can be matched across connections. `speaker` is `null` when diarization was requested but no speaker could be determined for that utterance. That is distinct from the field being absent, which means `diarize=true` was not requested. ### Speaker capacity Up to **four** concurrent speakers are labeled per connection. Beyond four, additional speakers are merged into existing labels: their speech is still transcribed, but `speaker` is wrong for some utterances. **Worth knowing:** `speakers_capped` and `speaker_purity` are the two signals that a label may be unreliable, and a client acting on speaker identity should check both. The four-speaker ceiling is lower than English Fast Transcription (streaming), which labels ten, so a conversation that diarizes cleanly there can exceed capacity here. ### Utterances that span a speaker change Segments are cut at pauses regardless of `diarize`. An utterance spanning a fast exchange with no pause carries the speaker who produced most of it, and `speaker_purity` reports what proportion that was. `1.0` is cleanly single-speaker; lower means the label describes only the majority. ### With endpointing off `diarize=true&endpointing=false` produces one final `utterance` for the whole stream, so a single `speaker` cannot describe it. Read `n_speakers` and treat `speaker` as indicative only. ## Connection flow 1. Connect with `api_key`, `audio_format`, and (for raw formats) `sample_rate` and `num_channels`. 2. Stream audio as **binary** WebSocket frames. 3. Receive `partial_utterance` messages while audio arrives. 4. Send an **empty text frame** (`""`) to signal end of audio. 5. Receive the final `utterance`(s), then a `done` message. 6. The connection closes. **Worth knowing:** step 4 must be a **text** frame containing the empty string. An empty *binary* frame is treated as an audio chunk and does not end the stream, so the connection hangs until it times out with no final `utterance`. ## Server messages ### `partial_utterance` The complete transcript so far for the current scope — the whole connection by default, or the current segment when `endpointing=true`. Not a delta: replace the displayed text on each message, never append. ```json theme={null} { "type": "partial_utterance", "partial_utterance": { "text": "Buenos días, gracias por", "is_final": false } } ``` Partials never carry speaker fields, even with `diarize=true`. ### `utterance` The final transcript for one segment. Will not be revised. ```json theme={null} { "type": "utterance", "utterance": { "text": "Buenos días, gracias por llamar.", "is_final": true, "start_ms": 0, "duration_ms": 2360, "speaker": 0, "n_speakers": 2, "speakers_capped": false, "speaker_purity": 1.0 } } ``` | Field | Type | Description | | - | - | - | | `type` | string | Always `"utterance"` | | `utterance.text` | string | Final transcript for this segment, in the language spoken. | | `utterance.is_final` | boolean | Always `true` | | `utterance.start_ms` | integer | Start of this segment, in milliseconds from the beginning of the stream. Approximate. | | `utterance.duration_ms` | integer | Duration of this segment's speech, excluding the pause that ended it. Approximate. | | `utterance.speaker` | integer \| null | Present only with `diarize=true`. Zero-based speaker index, `null` when none could be determined. | | `utterance.n_speakers` | integer | Present only with `diarize=true`. Distinct speakers detected on the connection so far. | | `utterance.speakers_capped` | boolean | Present only with `diarize=true`. `true` once all four speaker slots are in use. | | `utterance.speaker_purity` | number | Present only with `diarize=true`. Proportion of this utterance attributed to `speaker`, from 0 to 1. | With `diarize=false` no speaker field appears in any message. ### `done` Sent once, immediately after the final `utterance`. The server closes after it. ```json theme={null} { "type": "done", "duration_ms": 14253 } ``` ### `error` Sent if something goes wrong. The connection closes after it; no further messages follow. ```json theme={null} { "type": "error", "error": "diarize=true is not available on this endpoint." } ``` That second message means speaker labelling is not enabled on the instance that served the connection. Retry once — instances can differ while a deployment is rolling — and if it persists, reconnect without `diarize`. ## WebSocket close codes | Code | Meaning | | - | - | | `1000` | Normal closure after the `done` message | | `1003` | Invalid query parameters — missing `audio_format`, invalid `sample_rate`, invalid `endpointing`, invalid `diarize`, unsupported value. Also returned when `diarize=true` is requested but speaker labelling is not enabled on the serving instance; retry once before treating this as permanent. | | `1011` | Internal server error. An `error` message is sent before the close. | | `1013` | The service is temporarily at capacity. Not an account or billing condition — retry the connection. Diarized connections have a lower concurrency ceiling than plain transcription, so a `diarize=true` connection can be refused while a plain one succeeds. | | `4001` | The `api_key` query parameter is missing or invalid | | `4002` | Audio bytes did not match the declared raw format, or the audio could not be decoded mid-stream | | `4003` | The request is not permitted | | `4004` | The API key does not have access to this model | | `4029` | Insufficient credits | | `4030` | Concurrent request limit reached — an account-level limit on simultaneous connections, distinct from the service-capacity `1013`. Contact support to raise it. | | `4031` | Monthly usage limit reached | ## Related * [Transcription](/get-started/stt) — parameters, response shape, and audio formats for every transcription endpoint * [Multilingual Fast Transcription Batch](/api-reference/stt/batch-multilingual-vfast) — the file-upload counterpart, which does return `language` * [Which API should I use?](/guides/which-api) ## AsyncAPI ````yaml api/stt_streaming_multilingual_vfast.yaml velma2SttStreamingMultilingualVfast id: velma2SttStreamingMultilingualVfast title: Velma2 stt streaming multilingual vfast description: | Establish a WebSocket connection for low-latency multilingual transcription. The server emits partial transcripts as audio streams in, and a final `utterance` at end-of-stream. With `endpointing=true`, speech is additionally segmented at pauses into one final `utterance` per segment. With `diarize=true`, each final `utterance` also carries the speaker who produced it. With `time_stamps=true`, each final `utterance` also carries the start and end time of every word. ## Supported Languages Bulgarian, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hungarian, Italian, Latvian, Lithuanian, Maltese, Polish, Portuguese, Romanian, Russian, Slovak, Slovenian, Spanish, Swedish, Ukrainian. The language is detected from the audio automatically. There is no language query parameter, and the detected language is NOT reported in the response - transcript text is returned in the language spoken. Recognition quality varies by language. ## Connection Flow 1. Connect with `api_key`, `audio_format`, (for raw formats) `sample_rate` + `num_channels`, and optionally `endpointing`, `diarize`, and `time_stamps` 2. Connection accepted if authentication succeeds and limits are not exceeded 3. Stream audio data as binary WebSocket frames 4. Receive `partial_utterance` results every ~1.5 seconds while transcription is in progress (each is the complete transcript of the current segment so far - REPLACE, don't append) 5. With `endpointing=true`, receive a final `utterance` message each time a segment of speech ends, carrying that segment's text, start time, and duration 6. Send empty text frame (`""`) when the audio stream ends 7. Receive the final `utterance` for any content not already finalized mid-stream (the whole stream with endpointing off; with `endpointing=true`, this step is skipped when the last segment was already finalized mid-stream) 8. Receive `done` message with total audio duration 9. Connection closes automatically ## Partial and Segment Semantics By default the connection produces a single final `utterance` at end-of-stream, covering the whole stream (`start_ms=0`), and each `partial_utterance` reflects the whole stream so far. With `endpointing=true`, speech is segmented into utterances: each `partial_utterance` reflects the CURRENT segment, and a final `utterance` is emitted each time a segment ends, after which the following `partial_utterance` messages belong to the next segment; the full transcript is the concatenation of the final `utterance` texts. In both modes a `partial_utterance` is the complete transcript of its scope so far (not a delta), and the last few words may be revised in subsequent partials as the model sees more audio context, so clients should replace the displayed partial text rather than append. With `endpointing=true` AND `diarize=true`, a segment ends either at a pause OR when the speaker changes, whichever comes first. This generally produces more segments than endpointing alone: in fast back-and-forth conversation the pauses between turns are often shorter than the silence threshold, so speaker changes are what separate the turns. ## Speaker Labels With `diarize=true`, each final `utterance` carries `speaker` - a zero-based index that is stable for the lifetime of the connection. It identifies speakers only WITHIN one connection: the same person on a new connection may get a different index, and indices are not identities that can be matched across connections or sessions. A maximum of four concurrent speakers is supported. Past that, further speakers are merged into existing labels rather than reported separately - their speech is still transcribed, but its `speaker` will be wrong. `speakers_capped` is `true` once all four slots are in use, which is the only signal that labels may be unreliable; clients doing anything consequential with speaker identity should check it. Speaker labels are attached to final `utterance` messages only, never to `partial_utterance`. Diarization has no back-patching: a label is assigned when the utterance is finalized and is not revised afterwards. With `diarize=true` and `endpointing=false` there is only one final `utterance` for the whole stream, so a single `speaker` cannot describe it. In that mode read `n_speakers` - how many distinct speakers were detected on the connection - and treat `speaker` as indicative only. ## Word Timestamps With `time_stamps=true`, each final `utterance` carries `words` - one entry per word of its `text`, in order, each with `start_ms` and `end_ms`. Word times are measured on the same clock as the utterance's `start_ms`: milliseconds from the beginning of the audio stream, not from the start of the utterance. Word timestamps are attached to final `utterance` messages only, never to `partial_utterance`. They are computed alongside the transcript, so requesting them does not change the transcript text, its latency, or any other field of the payload. `time_stamps` is independent of `endpointing` and `diarize`; every combination is valid. ## Text Formatting Both `partial_utterance` and `utterance` payloads contain text that has already been auto-capitalized and auto-punctuated by the model, in the language spoken. Clients can render the `text` field directly without additional casing or punctuation normalization. Note that in `partial_utterance` messages, both capitalization and punctuation in the final words may be revised in subsequent partials as the model sees more audio context. ## Error Handling If an error occurs during transcription, the server sends an error message and closes the connection. Connection rejections during handshake use WebSocket close codes (see below). ## WebSocket Close Codes - `1000`: Normal closure after the `done` message, or after the connection completes. - `1003`: Invalid query parameters (missing `audio_format`, invalid `sample_rate`, invalid `endpointing`, invalid `diarize`, invalid `time_stamps`, unsupported value, etc.). Also returned when `diarize=true` is requested but speaker labelling is not enabled on the serving instance, or when `time_stamps=true` is requested but word timestamps are not enabled on the serving instance. - `1011`: Internal server error. An `error` message is sent before the close. - `1013`: The service is temporarily at capacity and could not accept the connection. An `error` message is sent before the close with the text "The service is temporarily unavailable. Please try again later." This is not an account or billing condition - retry the connection. Note that connections requesting `diarize=true` have a lower concurrency ceiling than plain transcription, so a `diarize=true` connection may be refused while a plain one succeeds. - `4001`: The `api_key` query parameter is missing or invalid. - `4002`: Audio data didn't match the declared raw format (sent container bytes but declared a raw type), or the audio could not be decoded mid-stream. The `error` message is "Audio data does not match the declared audio_format." for a declared raw format mismatch, or "The audio could not be processed. It may be corrupted or in an unsupported format." for a mid-stream decode failure. - `4003`: The request is not permitted. - `4004`: The API key does not have access to this model. - `4029`: The request could not be completed due to insufficient credits. - `4030`: Concurrent request limit reached. - `4031`: Monthly usage limit reached. servers: - id: production protocol: wss host: platform.modulate.ai bindings: [] variables: [] address: /api/velma-2-stt-streaming-multilingual-vfast parameters: [] bindings: - protocol: ws version: latest value: query: type: object required: - api_key - audio_format properties: api_key: type: string description: | Your API key for authentication. Must be provided as a query parameter when establishing the WebSocket connection. audio_format: type: string description: | Container or raw format of the audio data the client will send. Required. For container formats, the audio header determines the sample rate and channel count, and `sample_rate` / `num_channels` query parameters are ignored. The MP4-family values (`mp4`, `m4a`, `m4b`, `m4r`, `m4v`, `3gp`, `3gpp`, `3ga`, `3g2`, `f4a`, `f4b`) must be sent in a streamable layout; otherwise the connection ends with an audio-processing error. For raw formats, `sample_rate` and `num_channels` are required. The raw `g722` and `vox` formats are additionally mono-only (`num_channels` must be `1`). For lowest end-to-end latency, send `audio_format=s16le&sample_rate=16000&num_channels=1`. enum: - s8 - s16le - s16be - s24le - s24be - s32le - s32be - u8 - u16le - u16be - u24le - u24be - u32le - u32be - f32le - f32be - f64le - f64be - mulaw - alaw - g722 - vox - 3g2 - 3ga - 3gp - 3gpp - 8svx - aa3 - aac - ac3 - act - adts - aif - aifc - aiff - amb - amr - asf - at3 - au - avr - awb - bwf - c2 - caf - dss - dts - dtshd - eac3 - ec3 - f4a - f4b - flac - gsm - iff - m2a - m2ts - m4a - m4b - m4r - m4v - mka - mkv - mlp - mp+ - mp1 - mp2 - mp3 - mp4 - mpa - mpc - mpga - mpp - mts - oga - ogg - ogx - oma - omg - opus - paf - pvf - qcp - ra - rf64 - rm - rmvb - snd - svx - thd - ts - tta - voc - vqf - w64 - wav - wave - weba - webm - wma - wmv sample_rate: type: integer description: | Source sample rate of the audio data in Hz. Required when `audio_format` is a raw type; ignored for container formats. enum: - 8000 - 11025 - 16000 - 22050 - 32000 - 44100 - 48000 - 96000 num_channels: type: integer description: | Number of audio channels in the source data (1-8). Required when `audio_format` is a raw type; ignored for container formats. minimum: 1 maximum: 8 endpointing: type: boolean default: false description: > Whether to segment speech into utterances at pauses. Optional; defaults to `false`. When `false` (the default), the connection produces a single final `utterance` at end-of-stream covering the whole stream (`start_ms=0`), and each `partial_utterance` reflects the whole stream so far. When `true`, trailing silence after speech closes the current utterance and starts a new one: the server emits one final `utterance` per speech segment (each with its own `start_ms` / `duration_ms`), and each `partial_utterance` reflects the current segment. The full transcript is the concatenation of the final `utterance` texts. In both modes every connection ends with at least one final `utterance`; a stream containing no speech yields a single final with empty `text`. Independent of `diarize`. When both are `true`, a segment ends at a pause OR at a speaker change, whichever comes first. Example: `audio_format=s16le&sample_rate=16000&num_channels=1&endpointing=true`. diarize: type: boolean default: false description: > Whether to label which speaker produced each utterance. Optional; defaults to `false`. When `false` (the default), no speaker fields appear in any message and the payloads are byte-identical to a non-diarized connection. When `true`, each final `utterance` additionally carries `speaker`, `n_speakers`, `speakers_capped`, and `speaker_purity`. A maximum of four concurrent speakers is supported; see "Speaker Labels" above for what happens beyond that and for the limits of the `speaker` index. Independent of `endpointing`; all four combinations are valid. Speaker labels are only meaningful per utterance, however, so `diarize=true&endpointing=false` yields one whole-stream utterance where `n_speakers` is the useful field rather than `speaker`. Requesting `diarize=true` where speaker labelling is not enabled on the serving instance is rejected during the handshake with close code `1003`. Diarized connections have a lower concurrency ceiling than plain transcription and may be refused with close code `1013` when that ceiling is reached. Example: `audio_format=s16le&sample_rate=16000&num_channels=1&endpointing=true&diarize=true`. time_stamps: type: boolean default: false description: > Whether to include word-level timestamps on final utterances. Optional; defaults to `false`. When `false` (the default), no `words` field appears in any message and the payloads are byte-identical to a connection that did not pass the parameter. When `true`, each final `utterance` additionally carries `words`: one entry per word of `text`, in order, each with `word`, `start_ms`, and `end_ms` in milliseconds from the beginning of the audio stream. `partial_utterance` messages are unaffected. Independent of `endpointing` and `diarize`; all combinations are valid. Requesting `time_stamps=true` where word timestamps are not enabled on the serving instance is rejected during the handshake with close code `1003`. Example: `audio_format=s16le&sample_rate=16000&num_channels=1&endpointing=true&time_stamps=true`. schemaProperties: - name: query type: object required: false properties: - name: api_key type: string description: | Your API key for authentication. Must be provided as a query parameter when establishing the WebSocket connection. required: true - name: audio_format type: string description: | Container or raw format of the audio data the client will send. Required. For container formats, the audio header determines the sample rate and channel count, and `sample_rate` / `num_channels` query parameters are ignored. The MP4-family values (`mp4`, `m4a`, `m4b`, `m4r`, `m4v`, `3gp`, `3gpp`, `3ga`, `3g2`, `f4a`, `f4b`) must be sent in a streamable layout; otherwise the connection ends with an audio-processing error. For raw formats, `sample_rate` and `num_channels` are required. The raw `g722` and `vox` formats are additionally mono-only (`num_channels` must be `1`). For lowest end-to-end latency, send `audio_format=s16le&sample_rate=16000&num_channels=1`. enumValues: - s8 - s16le - s16be - s24le - s24be - s32le - s32be - u8 - u16le - u16be - u24le - u24be - u32le - u32be - f32le - f32be - f64le - f64be - mulaw - alaw - g722 - vox - 3g2 - 3ga - 3gp - 3gpp - 8svx - aa3 - aac - ac3 - act - adts - aif - aifc - aiff - amb - amr - asf - at3 - au - avr - awb - bwf - c2 - caf - dss - dts - dtshd - eac3 - ec3 - f4a - f4b - flac - gsm - iff - m2a - m2ts - m4a - m4b - m4r - m4v - mka - mkv - mlp - mp+ - mp1 - mp2 - mp3 - mp4 - mpa - mpc - mpga - mpp - mts - oga - ogg - ogx - oma - omg - opus - paf - pvf - qcp - ra - rf64 - rm - rmvb - snd - svx - thd - ts - tta - voc - vqf - w64 - wav - wave - weba - webm - wma - wmv required: true - name: sample_rate type: integer description: | Source sample rate of the audio data in Hz. Required when `audio_format` is a raw type; ignored for container formats. enumValues: - 8000 - 11025 - 16000 - 22050 - 32000 - 44100 - 48000 - 96000 required: false - name: num_channels type: integer description: | Number of audio channels in the source data (1-8). Required when `audio_format` is a raw type; ignored for container formats. required: false - name: endpointing type: boolean description: > Whether to segment speech into utterances at pauses. Optional; defaults to `false`. When `false` (the default), the connection produces a single final `utterance` at end-of-stream covering the whole stream (`start_ms=0`), and each `partial_utterance` reflects the whole stream so far. When `true`, trailing silence after speech closes the current utterance and starts a new one: the server emits one final `utterance` per speech segment (each with its own `start_ms` / `duration_ms`), and each `partial_utterance` reflects the current segment. The full transcript is the concatenation of the final `utterance` texts. In both modes every connection ends with at least one final `utterance`; a stream containing no speech yields a single final with empty `text`. Independent of `diarize`. When both are `true`, a segment ends at a pause OR at a speaker change, whichever comes first. Example: `audio_format=s16le&sample_rate=16000&num_channels=1&endpointing=true`. required: false - name: diarize type: boolean description: > Whether to label which speaker produced each utterance. Optional; defaults to `false`. When `false` (the default), no speaker fields appear in any message and the payloads are byte-identical to a non-diarized connection. When `true`, each final `utterance` additionally carries `speaker`, `n_speakers`, `speakers_capped`, and `speaker_purity`. A maximum of four concurrent speakers is supported; see "Speaker Labels" above for what happens beyond that and for the limits of the `speaker` index. Independent of `endpointing`; all four combinations are valid. Speaker labels are only meaningful per utterance, however, so `diarize=true&endpointing=false` yields one whole-stream utterance where `n_speakers` is the useful field rather than `speaker`. Requesting `diarize=true` where speaker labelling is not enabled on the serving instance is rejected during the handshake with close code `1003`. Diarized connections have a lower concurrency ceiling than plain transcription and may be refused with close code `1013` when that ceiling is reached. Example: `audio_format=s16le&sample_rate=16000&num_channels=1&endpointing=true&diarize=true`. required: false - name: time_stamps type: boolean description: > Whether to include word-level timestamps on final utterances. Optional; defaults to `false`. When `false` (the default), no `words` field appears in any message and the payloads are byte-identical to a connection that did not pass the parameter. When `true`, each final `utterance` additionally carries `words`: one entry per word of `text`, in order, each with `word`, `start_ms`, and `end_ms` in milliseconds from the beginning of the audio stream. `partial_utterance` messages are unaffected. Independent of `endpointing` and `diarize`; all combinations are valid. Requesting `time_stamps=true` where word timestamps are not enabled on the serving instance is rejected during the handshake with close code `1003`. Example: `audio_format=s16le&sample_rate=16000&num_channels=1&endpointing=true&time_stamps=true`. required: false operations: - &ref_2 id: streamAudio title: Stream audio description: Stream audio for low-latency multilingual transcription type: send messages: - &ref_7 id: audioChunk contentType: application/octet-stream payload: - type: string format: binary x-parser-schema-id: name: Audio Data Chunk description: | Raw audio data chunk for transcription. Sent zero or more times by the client during the stream lifetime. Each chunk's bytes are decoded by the server according to the connection's `audio_format` query parameter. headers: [] jsonPayloadSchema: type: string format: binary x-parser-schema-id: title: Audio Data Chunk description: | Raw audio data chunk for transcription. Sent zero or more times by the client during the stream lifetime. Each chunk's bytes are decoded by the server according to the connection's `audio_format` query parameter. example: '{}' bindings: [] extensions: - id: x-parser-unique-object-id value: audioChunk - &ref_8 id: endOfStream contentType: text/plain payload: - type: string const: '' x-parser-schema-id: name: End of Audio Stream description: | Empty text frame signaling end of audio. Sent exactly once by the client after the last audio chunk. The server responds with a final `utterance` message followed by a `done` message, then closes the connection. headers: [] jsonPayloadSchema: type: string const: '' x-parser-schema-id: title: End of Audio Stream description: | Empty text frame signaling end of audio. Sent exactly once by the client after the last audio chunk. The server responds with a final `utterance` message followed by a `done` message, then closes the connection. example: '""' bindings: [] extensions: - id: x-parser-unique-object-id value: endOfStream bindings: [] extensions: &ref_0 - id: x-parser-unique-object-id value: velma2SttStreamingMultilingualVfast - &ref_1 id: receiveTranscription title: Receive transcription description: Receive partial and final transcription results from the server type: receive messages: - &ref_3 id: partialUtteranceResult contentType: application/json payload: - name: Partial Utterance Result description: > In-progress transcript snapshot for the CURRENT speech segment. Sent zero or more times per segment, roughly every 1.5 seconds. The `text` field contains the current segment's transcript so far - not a delta, and not the whole stream - with auto-capitalization and auto-punctuation already applied. Clients should REPLACE the current segment's previously displayed partial text, not append. Partials never carry speaker labels, even with `diarize=true`, and never carry word timestamps, even with `time_stamps=true`. type: object properties: - name: type type: string description: partial_utterance required: true - name: partial_utterance type: object required: true properties: - name: text type: string description: > The complete transcript built so far for the current scope - the whole connection by default, or the current speech segment when `endpointing=true` - with auto-capitalization (sentence case, proper nouns, etc.) and auto-punctuation (commas, periods, question marks, and the language's own conventions such as Spanish `¿`) already applied by the model, in the language spoken. This is NOT a delta from the previous partial - clients should display this as a replacement for any previously displayed partial text. The last few words, including their capitalization and punctuation, may be revised in subsequent partials as the model sees more audio context. required: true - name: is_final type: boolean description: | Always `false` for `partial_utterance` messages. Use the message `type` field to distinguish partial from final. required: true headers: [] jsonPayloadSchema: type: object required: - type - partial_utterance properties: type: type: string const: partial_utterance x-parser-schema-id: partial_utterance: type: object required: - text - is_final properties: text: type: string description: > The complete transcript built so far for the current scope - the whole connection by default, or the current speech segment when `endpointing=true` - with auto-capitalization (sentence case, proper nouns, etc.) and auto-punctuation (commas, periods, question marks, and the language's own conventions such as Spanish `¿`) already applied by the model, in the language spoken. This is NOT a delta from the previous partial - clients should display this as a replacement for any previously displayed partial text. The last few words, including their capitalization and punctuation, may be revised in subsequent partials as the model sees more audio context. example: Hola, ¿cómo estás x-parser-schema-id: is_final: type: boolean const: false description: | Always `false` for `partial_utterance` messages. Use the message `type` field to distinguish partial from final. x-parser-schema-id: x-parser-schema-id: StreamingPartialUtterance x-parser-schema-id: title: Partial Utterance Result description: | In-progress transcript snapshot for the CURRENT speech segment. Sent zero or more times per segment, roughly every 1.5 seconds. The `text` field contains the current segment's transcript so far - not a delta, and not the whole stream - with auto-capitalization and auto-punctuation already applied. Clients should REPLACE the current segment's previously displayed partial text, not append. Partials never carry speaker labels, even with `diarize=true`, and never carry word timestamps, even with `time_stamps=true`. example: |- { "type": "partial_utterance", "partial_utterance": { "text": "Hola, ¿cómo estás", "is_final": false } } bindings: [] extensions: - id: x-parser-unique-object-id value: partialUtteranceResult - &ref_4 id: utteranceResult contentType: application/json payload: - name: Final Utterance Result description: > The final transcript for one segment of speech, with auto-capitalization and auto-punctuation applied, plus the segment's start time and duration. With `diarize=true` it also carries the speaker who produced it, and with `time_stamps=true` the start and end time of every word. Sent once per segment (when the segment ends, and once more for the last segment after end-of-stream), and supersedes the `partial_utterance` messages for that segment. The full transcript is the concatenation of these texts in order. type: object properties: - name: type type: string description: utterance required: true - name: utterance type: object required: true properties: - name: text type: string description: > The final transcribed text for one segment of speech, with auto-capitalization and auto-punctuation already applied by the model, in the language spoken. This is the authoritative transcript for the segment and will not be revised. The full transcript is the concatenation of the `text` of every `utterance` in order. required: true - name: is_final type: boolean description: > Always `true` for `utterance` messages. Use the message `type` field to distinguish partial from final. required: true - name: start_ms type: integer description: > Start time of this segment, in milliseconds from the beginning of the audio stream. Approximate: speech onset is detected at the server's internal processing granularity, so the start may differ from the first spoken word by a fraction of a second. required: true - name: duration_ms type: integer description: > Duration of this segment's speech, in milliseconds, excluding the trailing pause that ended the segment. Approximate, with the same boundary granularity as `start_ms`. With endpointing off, the single final utterance instead covers the whole stream, including any leading and trailing silence. required: true - name: speaker type: integer description: > Present only when `diarize=true`. Zero-based index of the speaker who produced this utterance, stable for the lifetime of the connection but meaningless outside it - the same person on a new connection may receive a different index. `null` means diarization was requested but no speaker could be determined for this utterance, which happens for very short or silent segments. That is distinct from the field being absent, which means diarization was not requested at all. With `endpointing=false` there is a single whole-stream utterance that may contain several speakers, so this field is indicative only in that mode - read `n_speakers` instead. required: false - name: n_speakers type: integer description: > Present only when `diarize=true`. Count of distinct speakers detected on the connection so far. This is the meaningful speaker signal when `endpointing=false`, where one `speaker` index cannot describe a whole multi-speaker stream. required: false - name: speakers_capped type: boolean description: > Present only when `diarize=true`. `true` once all four speaker slots are in use, meaning the connection may contain more speakers than can be represented. Beyond four, additional speakers are merged into existing labels rather than reported separately - their speech is still transcribed, but `speaker` will be wrong for some utterances and there is no other signal that this happened. Clients that act on speaker identity should check this field. required: false - name: speaker_purity type: number description: > Present only when `diarize=true`. The proportion of this utterance's speech attributed to the speaker named in `speaker`. `1.0` means the utterance was cleanly single-speaker. A lower value means the utterance spans a speaker change that the segmentation did not split, so its single `speaker` label describes only the majority of it. required: false - name: words type: array description: > Present only when `time_stamps=true`. One entry per word of `text`, in the same order. Absent - not an empty array - when word timestamps were not requested, so a client can tell "timestamps off" from "no words". required: false properties: - name: word type: string description: > The word as it appears in the utterance `text`, including any attached punctuation and casing. required: true - name: start_ms type: integer description: > Start time of the word, in milliseconds from the beginning of the audio stream - the same clock as the utterance's `start_ms`, not relative to the utterance. required: true - name: end_ms type: integer description: > End time of the word, in milliseconds from the beginning of the audio stream. Always greater than or equal to `start_ms`. required: true headers: [] jsonPayloadSchema: type: object required: - type - utterance properties: type: type: string const: utterance x-parser-schema-id: utterance: type: object required: - text - is_final - start_ms - duration_ms properties: text: type: string description: > The final transcribed text for one segment of speech, with auto-capitalization and auto-punctuation already applied by the model, in the language spoken. This is the authoritative transcript for the segment and will not be revised. The full transcript is the concatenation of the `text` of every `utterance` in order. example: Hola, ¿cómo estás hoy? x-parser-schema-id: is_final: type: boolean const: true description: > Always `true` for `utterance` messages. Use the message `type` field to distinguish partial from final. x-parser-schema-id: start_ms: type: integer minimum: 0 description: > Start time of this segment, in milliseconds from the beginning of the audio stream. Approximate: speech onset is detected at the server's internal processing granularity, so the start may differ from the first spoken word by a fraction of a second. example: 0 x-parser-schema-id: duration_ms: type: integer minimum: 0 description: > Duration of this segment's speech, in milliseconds, excluding the trailing pause that ended the segment. Approximate, with the same boundary granularity as `start_ms`. With endpointing off, the single final utterance instead covers the whole stream, including any leading and trailing silence. example: 2360 x-parser-schema-id: speaker: type: integer nullable: true minimum: 0 maximum: 3 description: > Present only when `diarize=true`. Zero-based index of the speaker who produced this utterance, stable for the lifetime of the connection but meaningless outside it - the same person on a new connection may receive a different index. `null` means diarization was requested but no speaker could be determined for this utterance, which happens for very short or silent segments. That is distinct from the field being absent, which means diarization was not requested at all. With `endpointing=false` there is a single whole-stream utterance that may contain several speakers, so this field is indicative only in that mode - read `n_speakers` instead. example: 0 x-parser-schema-id: n_speakers: type: integer minimum: 0 description: | Present only when `diarize=true`. Count of distinct speakers detected on the connection so far. This is the meaningful speaker signal when `endpointing=false`, where one `speaker` index cannot describe a whole multi-speaker stream. example: 2 x-parser-schema-id: speakers_capped: type: boolean description: > Present only when `diarize=true`. `true` once all four speaker slots are in use, meaning the connection may contain more speakers than can be represented. Beyond four, additional speakers are merged into existing labels rather than reported separately - their speech is still transcribed, but `speaker` will be wrong for some utterances and there is no other signal that this happened. Clients that act on speaker identity should check this field. example: false x-parser-schema-id: speaker_purity: type: number minimum: 0 maximum: 1 description: > Present only when `diarize=true`. The proportion of this utterance's speech attributed to the speaker named in `speaker`. `1.0` means the utterance was cleanly single-speaker. A lower value means the utterance spans a speaker change that the segmentation did not split, so its single `speaker` label describes only the majority of it. example: 1 x-parser-schema-id: words: type: array items: type: object required: - word - start_ms - end_ms properties: word: type: string description: > The word as it appears in the utterance `text`, including any attached punctuation and casing. example: Hola, x-parser-schema-id: start_ms: type: integer minimum: 0 description: > Start time of the word, in milliseconds from the beginning of the audio stream - the same clock as the utterance's `start_ms`, not relative to the utterance. example: 1240 x-parser-schema-id: end_ms: type: integer minimum: 0 description: > End time of the word, in milliseconds from the beginning of the audio stream. Always greater than or equal to `start_ms`. example: 1560 x-parser-schema-id: x-parser-schema-id: WordTimestamp description: > Present only when `time_stamps=true`. One entry per word of `text`, in the same order. Absent - not an empty array - when word timestamps were not requested, so a client can tell "timestamps off" from "no words". x-parser-schema-id: x-parser-schema-id: StreamingUtterance x-parser-schema-id: title: Final Utterance Result description: | The final transcript for one segment of speech, with auto-capitalization and auto-punctuation applied, plus the segment's start time and duration. With `diarize=true` it also carries the speaker who produced it, and with `time_stamps=true` the start and end time of every word. Sent once per segment (when the segment ends, and once more for the last segment after end-of-stream), and supersedes the `partial_utterance` messages for that segment. The full transcript is the concatenation of these texts in order. example: |- { "type": "utterance", "utterance": { "text": "Hola, ¿cómo estás hoy?", "is_final": true, "start_ms": 0, "duration_ms": 2360 } } bindings: [] extensions: - id: x-parser-unique-object-id value: utteranceResult - &ref_5 id: errorResult contentType: application/json payload: - name: Error description: | An error occurred during transcription. Sent at most once per connection; the WebSocket closes shortly after. No further messages follow an `error`. type: object properties: - name: type type: string description: error required: true - name: error type: string description: Human-readable error message required: true headers: [] jsonPayloadSchema: type: object required: - type - error properties: type: type: string const: error x-parser-schema-id: error: type: string description: Human-readable error message x-parser-schema-id: x-parser-schema-id: title: Error description: | An error occurred during transcription. Sent at most once per connection; the WebSocket closes shortly after. No further messages follow an `error`. example: |- { "type": "error", "error": "Internal server error" } bindings: [] extensions: - id: x-parser-unique-object-id value: errorResult - &ref_6 id: doneResult contentType: application/json payload: - name: Transcription Complete description: > Transcription completed successfully. Sent exactly once, immediately after the final `utterance` message. After this message the server closes the WebSocket. type: object properties: - name: type type: string description: done required: true - name: duration_ms type: integer description: Total duration of audio processed in milliseconds required: true headers: [] jsonPayloadSchema: type: object required: - type - duration_ms properties: type: type: string const: done x-parser-schema-id: duration_ms: type: integer minimum: 0 description: Total duration of audio processed in milliseconds x-parser-schema-id: x-parser-schema-id: title: Transcription Complete description: | Transcription completed successfully. Sent exactly once, immediately after the final `utterance` message. After this message the server closes the WebSocket. example: |- { "type": "done", "duration_ms": 14253 } bindings: [] extensions: - id: x-parser-unique-object-id value: doneResult bindings: [] extensions: *ref_0 sendOperations: - *ref_1 receiveOperations: - *ref_2 sendMessages: - *ref_3 - *ref_4 - *ref_5 - *ref_6 receiveMessages: - *ref_7 - *ref_8 extensions: - id: x-parser-unique-object-id value: velma2SttStreamingMultilingualVfast securitySchemes: [] ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.