> ## Documentation Index
> Fetch the complete documentation index at: https://docs.modulate.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Speech-to-Text Transcription Streaming Multilingual
> Real-time speech-to-text over WebSocket, with optional speaker diarization, emotion detection, accent detection, and PII/PHI tagging.
Real-time speech-to-text over WebSocket. Streams audio to the server and receives transcribed utterances as they are processed, with optional speaker diarization, emotion detection, accent detection, and PII/PHI tagging.
## Endpoint
```text theme={null}
wss://platform.modulate.ai/api/velma-2-stt-streaming
```
## Authentication
Pass your API key as a query parameter when opening the connection.
```text theme={null}
wss://platform.modulate.ai/api/velma-2-stt-streaming?api_key=YOUR_API_KEY
```
Unlike the batch endpoints, the streaming API does not use an `X-API-Key` header. The key must be in the query string at connection time.
See [Authentication and rate limits](/guides/authentication) for how to obtain and manage API keys.
## Supported audio formats
Self-describing container formats are auto-detected from headers (no `audio_format` query parameter needed). Raw / headerless formats require `audio_format`, `sample_rate`, and `num_channels`. For the authoritative list of accepted values, see the spec's `audio_format` enum or [Transcription](/get-started/stt).
Opus is recommended when you control the encoder — high quality at low bandwidth.
## Query parameters
| Parameter | Type | Default | Description |
| - | - | - | - |
| `api_key` | string | — | **Required.** Your API key |
| `speaker_diarization` | boolean | `true` | Identify and label distinct speakers |
| `emotion_signal` | boolean | `false` | Detect emotional tone per utterance |
| `accent_signal` | boolean | `false` | Detect speaker accent per utterance |
| `deepfake_signal` | boolean | `false` | Per-utterance synthetic-voice (deepfake) score |
| `pii_phi_tagging` | boolean | `false` | Wrap PII/PHI in tags within utterance text |
| `partial_results` | boolean | `false` | Stream interim `partial_utterance` messages with in-progress text — and the latest interim `emotion`, `accent`, and `deepfake_score` values when those signals are enabled — before each utterance is finalized |
| `language` | string | (auto-detect) | Optional language hint as a case-insensitive ISO 639-1 code (e.g. `en`, `fr`). BCP 47 region/script subtags (e.g. `en-US`) are accepted; only the primary language subtag is used. When omitted, the language is detected automatically for each utterance |
Every parameter except `api_key` can also be set — and overridden — in an optional [configuration frame](#configuration-frame) sent as the first WebSocket text frame.
For a full explanation of what each feature does and when to enable it, see [Transcription](/get-started/stt#signals).
## Connection flow
1. Connect to the WebSocket endpoint with `api_key` and any optional feature parameters.
2. Optionally send a JSON [configuration frame](#configuration-frame) as the first text frame, before any audio. If the first frame is binary, it is treated as the first audio chunk and the query-parameter defaults apply.
3. Stream raw audio as **binary** WebSocket frames. Frames can be any size.
4. Receive `utterance` JSON messages as speech is transcribed. If `partial_results=true`, also receive `partial_utterance` previews for the currently active utterance.
5. Send an **empty text frame** (`""`) to signal end of audio.
6. Receive a `done` message containing total audio duration.
7. The connection closes automatically.
## Configuration frame
You can optionally send a JSON configuration as the **first** WebSocket text frame, before any audio. When the first frame is text, it is parsed as this configuration; when the first frame is binary, it is treated as the first audio chunk and the query-parameter defaults apply — existing binary-first clients need no changes. Every field is optional, and a field present in the configuration overrides the matching query parameter.
| Field | Type | Description |
| - | - | - |
| `speaker_diarization` | boolean | Overrides the `speaker_diarization` query parameter |
| `emotion_signal` | boolean | Overrides the `emotion_signal` query parameter |
| `accent_signal` | boolean | Overrides the `accent_signal` query parameter |
| `deepfake_signal` | boolean | Overrides the `deepfake_signal` query parameter |
| `pii_phi_tagging` | boolean | Overrides the `pii_phi_tagging` query parameter |
| `partial_results` | boolean | Overrides the `partial_results` query parameter |
| `language` | string \| null | Overrides the `language` query parameter |
| `custom_terms` | array \| null | Custom vocabulary to bias transcription toward domain terms and names — see [Custom vocabulary](/get-started/stt#custom-vocabulary) |
```json theme={null}
{
"emotion_signal": true,
"language": "en",
"custom_terms": ["Modulate", { "term": "Velma", "pronunciations": ["VEL-muh"] }]
}
```
An invalid configuration frame (malformed JSON or a value outside the configuration schema) triggers an `error` message and closes the connection with code `1003`.
## Server messages
### `utterance`
Sent each time a speech segment is transcribed.
```json theme={null}
{
"type": "utterance",
"utterance": {
"utterance_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"text": "Hello, how are you today?",
"start_ms": 0,
"duration_ms": 2500,
"speaker": 1,
"language": "en",
"emotion": "Neutral",
"accent": "American",
"deepfake_score": null
}
}
```
#### Utterance fields
| Field | Type | Description |
| - | - | - |
| `utterance_uuid` | string (UUID) | Unique identifier for this utterance |
| `text` | string | Transcribed text |
| `start_ms` | integer | Start time in milliseconds from the beginning of the stream |
| `duration_ms` | integer | Duration of the utterance in milliseconds |
| `speaker` | integer | Speaker number, 1-indexed |
| `language` | string | Detected language code (e.g. `"en"`, `"fr"`) |
| `emotion` | string \| null | Detected emotion. `null` when `emotion_signal` is disabled |
| `accent` | string \| null | Detected accent. `null` when `accent_signal` is disabled |
| `deepfake_score` | float \| null | Synthetic-voice score from `0.0` (likely natural) to `1.0` (likely synthetic). `null` when `deepfake_signal` is disabled or when the utterance is shorter than 0.5 seconds |
For all valid `emotion` and `accent` values, see [Transcription](/get-started/stt#signals).
### `partial_utterance`
Sent only when `partial_results=true`. Delivers in-progress text for the currently active utterance as a low-latency preview. Each partial also carries the latest interim `emotion`, `accent`, and `deepfake_score` values for the utterance when those signals are enabled; each is `null` until a value is available. Each `partial_utterance` supersedes the previous one for the same utterance; the finalized `utterance` message supersedes all preceding partials.
```json theme={null}
{
"type": "partial_utterance",
"partial_utterance": {
"text": "Hello, how are",
"start_ms": 0,
"speaker": 1,
"emotion": null,
"accent": null,
"deepfake_score": null
}
}
```
#### Partial utterance fields
| Field | Type | Description |
| - | - | - |
| `text` | string | In-progress transcribed text. May be empty |
| `start_ms` | integer \| null | Start time in milliseconds from the beginning of the stream. `null` if timing data is not yet available |
| `speaker` | integer \| null | Speaker number, 1-indexed. `null` if the speaker has not yet been identified |
| `emotion` | string \| null | Latest interim emotion label for the in-progress utterance. `null` when `emotion_signal` is disabled or no value is available yet |
| `accent` | string \| null | Latest interim accent label for the in-progress utterance. `null` when `accent_signal` is disabled or no value is available yet |
| `deepfake_score` | float \| null | Latest interim deepfake score for the in-progress utterance. `null` when `deepfake_signal` is disabled or no value is available yet |
### `done`
Sent after all audio has been processed, in response to the end-of-stream signal.
```json theme={null}
{
"type": "done",
"duration_ms": 45000
}
```
### `error`
Sent if transcription fails. The connection closes after this message.
```json theme={null}
{
"type": "error",
"error": "Internal server error"
}
```
## WebSocket close codes
| Code | Meaning |
| - | - |
| `1000` | Normal closure after a successful `done` message |
| `1003` | Invalid connection parameters (unsupported `audio_format`, invalid `sample_rate`, `num_channels`, or `language`; raw format missing `sample_rate`/`num_channels`), or an invalid first-frame configuration (malformed JSON or a value outside the configuration schema) |
| `1011` | Internal server error, or processing ended before completion |
| `4001` | The `api_key` query parameter is missing or invalid |
| `4003` | The request is not permitted |
| `4004` | The API key does not have access to this model |
| `4029` | Insufficient credits |
| `4030` | Concurrent request limit reached |
| `4031` | Monthly usage limit reached |
An `error` JSON message is sent before the connection closes (except on `1000`).
## Rate limits
* Concurrent connection limits apply per model.
* Connections that exceed limits are rejected during the WebSocket handshake with close code `4030`.
See [Authentication and rate limits](/guides/authentication) for retry guidance.
## Examples
```python theme={null}
import asyncio
import json
import aiohttp
API_KEY = "YOUR_API_KEY"
AUDIO_FILE = "recording.opus"
CHUNK_SIZE = 8192
async def transcribe_streaming():
url = (
f"wss://platform.modulate.ai/api/velma-2-stt-streaming"
f"?api_key={API_KEY}"
f"&speaker_diarization=true"
f"&emotion_signal=true"
f"&accent_signal=true"
)
utterances = []
async with aiohttp.ClientSession() as session:
async with session.ws_connect(url) as ws:
async def send_audio():
with open(AUDIO_FILE, "rb") as f:
while chunk := f.read(CHUNK_SIZE):
await ws.send_bytes(chunk)
await asyncio.sleep(CHUNK_SIZE / 4000)
await ws.send_str("")
send_task = asyncio.create_task(send_audio())
try:
async for msg in ws:
if msg.type == aiohttp.WSMsgType.TEXT:
data = json.loads(msg.data)
if data["type"] == "utterance":
u = data["utterance"]
utterances.append(u)
print(f"[Speaker {u['speaker']}] {u['text']}")
elif data["type"] == "done":
print(f"Done. Duration: {data['duration_ms']}ms")
break
elif data["type"] == "error":
print(f"Error: {data['error']}")
break
elif msg.type in (
aiohttp.WSMsgType.ERROR,
aiohttp.WSMsgType.CLOSE,
aiohttp.WSMsgType.CLOSED,
):
break
finally:
if not send_task.done():
send_task.cancel()
full_text = " ".join(u["text"] for u in utterances)
print(f"\nFull transcript:\n{full_text}")
asyncio.run(transcribe_streaming())
```
```javascript theme={null}
const WebSocket = require("ws");
const fs = require("fs");
const API_KEY = "YOUR_API_KEY";
const AUDIO_FILE = "recording.opus";
const CHUNK_SIZE = 8192;
const url = new URL("wss://platform.modulate.ai/api/velma-2-stt-streaming");
url.searchParams.set("api_key", API_KEY);
url.searchParams.set("speaker_diarization", "true");
url.searchParams.set("emotion_signal", "true");
const ws = new WebSocket(url.toString());
const utterances = [];
ws.on("open", () => {
const stream = fs.createReadStream(AUDIO_FILE, { highWaterMark: CHUNK_SIZE });
stream.on("data", (chunk) => ws.send(chunk));
stream.on("end", () => ws.send(""));
});
ws.on("message", (data) => {
const msg = JSON.parse(data.toString());
if (msg.type === "utterance") {
utterances.push(msg.utterance);
console.log(`[Speaker ${msg.utterance.speaker}] ${msg.utterance.text}`);
} else if (msg.type === "done") {
console.log(`Done. Duration: ${msg.duration_ms}ms`);
console.log("Transcript:", utterances.map((u) => u.text).join(" "));
ws.close();
} else if (msg.type === "error") {
console.error("Error:", msg.error);
ws.close();
}
});
ws.on("error", (err) => console.error("WebSocket error:", err.message));
```
WebSocket APIs cannot be tested with cURL. For command-line testing, use [`websocat`](https://github.com/vi/websocat).
## Related
* [Which API should I use?](/guides/which-api) — when streaming is the right choice vs batch
* [Transcription](/get-started/stt#signals) — full reference for diarization, emotion, accent, and PII/PHI tagging
* [Authentication and rate limits](/guides/authentication)
## AsyncAPI
````yaml api/velma_2_stt_streaming.yaml velma2SttStreaming
id: velma2SttStreaming
title: Velma2 stt streaming
description: >
Establish a WebSocket connection for real-time transcription with automatic
language
detection. The service automatically detects the language for each utterance
in
real-time. Different utterances within the same stream may be identified as
different
languages if code-switching occurs.
## Connection Flow
1. Connect with `api_key` and optional query parameters
2. Connection will be accepted if authentication succeeds and limits are not
exceeded
3. Optionally send a JSON configuration as the first text frame, before any
audio.
If the first frame is binary it is treated as the first audio chunk and the
query-parameter defaults apply.
4. Stream audio data as binary WebSocket frames
5. Receive utterance results as JSON messages
6. Send empty text frame ("") when audio stream ends
7. Receive final "done" message with total audio duration
8. Connection closes automatically after completion
## Error Handling
If an error occurs during transcription, the server sends one terminal error
message
and closes the connection. No further results or `done` follow that error.
Connection rejections during handshake use WebSocket close codes.
## WebSocket Close Codes
- `1000`: Normal closure after the `done` message, or after the
connection completes.
- `1003`: Invalid connection parameters (e.g. unsupported `audio_format`,
invalid `sample_rate` or `num_channels`, an invalid `language`, or a raw
format missing `sample_rate`/`num_channels`), or an invalid first-frame
configuration (malformed JSON or a value outside the documented
configuration schema), or rejected input audio. An `error` message is
sent before the close.
- `1011`: Internal server error, or processing ended before completion.
An `error` message is sent before the close; when processing ends
without a `done` message, the `error` message is
"Unable to complete the request. Please try again."
- `4001`: The `api_key` query parameter is missing or invalid. An
`error` message is sent before the close.
- `4003`: The request is not permitted. An `error` message is sent
before the close.
- `4004`: The API key does not have access to this model. An `error`
message is sent before the close.
- `4029`: The request could not be completed due to insufficient credits.
An `error` message is sent before the close.
- `4030`: Concurrent request limit reached. An `error` message is sent
before the close.
- `4031`: Monthly usage limit reached. An `error` message is sent before
the close.
servers:
- id: production
protocol: wss
host: platform.modulate.ai
bindings: []
variables: []
address: /api/velma-2-stt-streaming
parameters: []
bindings:
- protocol: ws
version: latest
value:
query:
type: object
required:
- api_key
properties:
api_key:
type: string
description: >
Your API key for authentication. Required. Must be provided as a
query
parameter when establishing the WebSocket connection; the
connection is
rejected during the handshake if it is missing.
speaker_diarization:
type: boolean
default: true
description: >
Enable speaker diarization to identify different speakers in the
audio stream.
When enabled, each utterance includes a speaker identifier
(1-indexed integer).
emotion_signal:
type: boolean
default: false
description: >
Enable emotion detection for each utterance. When enabled, the
`emotion` field
in each utterance will contain the detected emotional tone of the
speaker.
accent_signal:
type: boolean
default: false
description: >
Enable accent detection for each utterance. When enabled, the
`accent` field
in each utterance will contain the detected accent of the speaker.
deepfake_signal:
type: boolean
default: false
description: >
Enable synthetic voice (deepfake) detection for each utterance.
When enabled,
the `deepfake_score` field in each utterance will contain a score
from 0.0
(likely natural) to 1.0 (likely synthetic).
pii_phi_tagging:
type: boolean
default: false
description: >
Enable PII/PHI tagging. When enabled, personally identifiable
information and
personal health information in the transcribed text will be
wrapped with
appropriate tags.
audio_format:
type: string
description: >
Audio encoding format. Determines how the service interprets
incoming audio data.
**Self-describing formats** (`wav`, `mp3`, `ogg`, `flac`, `webm`,
`aac`, `aiff`):
These include container formats (AIFF, OGG, WAV, WebM) and codecs
whose streams
carry per-frame headers (AAC/ADTS, FLAC, MP3). When omitted
(default), the service
auto-detects the format from these headers. You may specify the
format explicitly
to override auto-detection. Do not specify `sample_rate` or
`num_channels` for
self-describing formats - the headers already carry this metadata.
**Raw/headerless formats** (`s8`..`f64be`, `mulaw`, `alaw`):
These formats are raw sample data with no headers. When using a
raw format, you
must also specify `sample_rate` and `num_channels` so the service
knows how to
interpret the byte stream. Omitting either will result in an
error.
enum:
- s8
- s16le
- s16be
- s24le
- s24be
- s32le
- s32be
- u8
- u16le
- u16be
- u24le
- u24be
- u32le
- u32be
- f32le
- f32be
- f64le
- f64be
- mulaw
- alaw
- wav
- mp3
- ogg
- flac
- webm
- aac
- aiff
sample_rate:
type: integer
description: >
Audio sample rate in Hz. Required when `audio_format` is a
raw/headerless format
(PCM, mu-law, A-law). Do not specify for self-describing formats
(WAV, MP3, OGG,
etc.) - their headers already include this metadata. Cannot be
specified without
`audio_format`.
enum:
- 8000
- 11025
- 16000
- 22050
- 32000
- 44100
- 48000
- 96000
num_channels:
type: integer
minimum: 1
maximum: 8
description: >
Number of audio channels (1-8). Required when `audio_format` is a
raw/headerless
format (PCM, mu-law, A-law). Do not specify for self-describing
formats (WAV, MP3,
OGG, etc.) - their headers already include this metadata. Use 1
for mono, 2 for
stereo. Cannot be specified without `audio_format`.
partial_results:
type: boolean
default: false
description: >
Enable partial (interim) transcription results. When enabled, the
server sends
`partial_utterance` messages containing in-progress text as speech
is being
recognized, in addition to the finalized `utterance` messages.
Each partial also
carries the latest interim `emotion`, `accent`, and
`deepfake_score` for the
utterance when those signals are enabled; each is null until a
value is
available. Each partial replaces the previous partial for the same
utterance,
and the finalized `utterance` supersedes the partials that
preceded it for that
utterance. A partial for a later utterance may arrive before an
earlier
finalized `utterance`.
language:
type: string
pattern: ^[A-Za-z]{2,3}([-_][A-Za-z0-9]+)*$
description: >
Optional language hint as a case-insensitive ISO 639-1 code (e.g.
"en",
"fr"). BCP 47 region/script subtags (e.g. "en-US") are accepted;
only the
primary language subtag is used. When omitted, the language is
detected
automatically for each utterance.
schemaProperties:
- name: query
type: object
required: false
properties:
- name: api_key
type: string
description: >
Your API key for authentication. Required. Must be provided as a
query
parameter when establishing the WebSocket connection; the
connection is
rejected during the handshake if it is missing.
required: true
- name: speaker_diarization
type: boolean
description: >
Enable speaker diarization to identify different speakers in the
audio stream.
When enabled, each utterance includes a speaker identifier
(1-indexed integer).
required: false
- name: emotion_signal
type: boolean
description: >
Enable emotion detection for each utterance. When enabled, the
`emotion` field
in each utterance will contain the detected emotional tone of the
speaker.
required: false
- name: accent_signal
type: boolean
description: >
Enable accent detection for each utterance. When enabled, the
`accent` field
in each utterance will contain the detected accent of the speaker.
required: false
- name: deepfake_signal
type: boolean
description: >
Enable synthetic voice (deepfake) detection for each utterance.
When enabled,
the `deepfake_score` field in each utterance will contain a score
from 0.0
(likely natural) to 1.0 (likely synthetic).
required: false
- name: pii_phi_tagging
type: boolean
description: >
Enable PII/PHI tagging. When enabled, personally identifiable
information and
personal health information in the transcribed text will be
wrapped with
appropriate tags.
required: false
- name: audio_format
type: string
description: >
Audio encoding format. Determines how the service interprets
incoming audio data.
**Self-describing formats** (`wav`, `mp3`, `ogg`, `flac`, `webm`,
`aac`, `aiff`):
These include container formats (AIFF, OGG, WAV, WebM) and codecs
whose streams
carry per-frame headers (AAC/ADTS, FLAC, MP3). When omitted
(default), the service
auto-detects the format from these headers. You may specify the
format explicitly
to override auto-detection. Do not specify `sample_rate` or
`num_channels` for
self-describing formats - the headers already carry this metadata.
**Raw/headerless formats** (`s8`..`f64be`, `mulaw`, `alaw`):
These formats are raw sample data with no headers. When using a
raw format, you
must also specify `sample_rate` and `num_channels` so the service
knows how to
interpret the byte stream. Omitting either will result in an
error.
enumValues:
- s8
- s16le
- s16be
- s24le
- s24be
- s32le
- s32be
- u8
- u16le
- u16be
- u24le
- u24be
- u32le
- u32be
- f32le
- f32be
- f64le
- f64be
- mulaw
- alaw
- wav
- mp3
- ogg
- flac
- webm
- aac
- aiff
required: false
- name: sample_rate
type: integer
description: >
Audio sample rate in Hz. Required when `audio_format` is a
raw/headerless format
(PCM, mu-law, A-law). Do not specify for self-describing formats
(WAV, MP3, OGG,
etc.) - their headers already include this metadata. Cannot be
specified without
`audio_format`.
enumValues:
- 8000
- 11025
- 16000
- 22050
- 32000
- 44100
- 48000
- 96000
required: false
- name: num_channels
type: integer
description: >
Number of audio channels (1-8). Required when `audio_format` is a
raw/headerless
format (PCM, mu-law, A-law). Do not specify for self-describing
formats (WAV, MP3,
OGG, etc.) - their headers already include this metadata. Use 1
for mono, 2 for
stereo. Cannot be specified without `audio_format`.
required: false
- name: partial_results
type: boolean
description: >
Enable partial (interim) transcription results. When enabled, the
server sends
`partial_utterance` messages containing in-progress text as speech
is being
recognized, in addition to the finalized `utterance` messages.
Each partial also
carries the latest interim `emotion`, `accent`, and
`deepfake_score` for the
utterance when those signals are enabled; each is null until a
value is
available. Each partial replaces the previous partial for the same
utterance,
and the finalized `utterance` supersedes the partials that
preceded it for that
utterance. A partial for a later utterance may arrive before an
earlier
finalized `utterance`.
required: false
- name: language
type: string
description: >
Optional language hint as a case-insensitive ISO 639-1 code (e.g.
"en",
"fr"). BCP 47 region/script subtags (e.g. "en-US") are accepted;
only the
primary language subtag is used. When omitted, the language is
detected
automatically for each utterance.
required: false
operations:
- &ref_2
id: streamAudio
title: Stream audio
description: Stream audio for real-time transcription with automatic language detection
type: send
messages:
- &ref_7
id: streamConfig
contentType: application/json
payload:
- name: Stream Configuration
description: >
Optional JSON configuration sent as the FIRST WebSocket text
frame,
before any audio. When the first frame is text it is parsed as
this
configuration; when the first frame is binary it is treated as the
first
audio chunk and the query-parameter defaults apply. A field that
has a
matching query parameter overrides it. Supply custom_terms in this
frame.
type: object
properties:
- name: speaker_diarization
type: boolean
required: false
- name: emotion_signal
type: boolean
required: false
- name: accent_signal
type: boolean
required: false
- name: deepfake_signal
type: boolean
required: false
- name: pii_phi_tagging
type: boolean
required: false
- name: partial_results
type: boolean
required: false
- name: language
type: string
required: false
- name: custom_terms
type: array
description: >-
Optional custom vocabulary to bias transcription toward domain
terms and names. Each entry is either a plain string (the
term) or a CustomTerm object. Blank terms and blank
pronunciations are dropped. At most 1000 entries, and the term
strings serialized together as JSON must total under 8000
characters. Null or omitted means no biasing.
required: false
headers: []
jsonPayloadSchema:
type: object
description: >-
Optional first-frame stream configuration. Every field is optional;
a field that has a matching query parameter overrides it.
properties:
speaker_diarization:
type: boolean
x-parser-schema-id:
emotion_signal:
type: boolean
x-parser-schema-id:
accent_signal:
type: boolean
x-parser-schema-id:
deepfake_signal:
type: boolean
x-parser-schema-id:
pii_phi_tagging:
type: boolean
x-parser-schema-id:
partial_results:
type: boolean
x-parser-schema-id:
language:
oneOf:
- type: string
pattern: ^[A-Za-z]{2,3}([-_][A-Za-z0-9]+)*$
x-parser-schema-id:
- type: 'null'
x-parser-schema-id:
x-parser-schema-id:
custom_terms:
description: >-
Optional custom vocabulary to bias transcription toward domain
terms and names. Each entry is either a plain string (the term)
or a CustomTerm object. Blank terms and blank pronunciations are
dropped. At most 1000 entries, and the term strings serialized
together as JSON must total under 8000 characters. Null or
omitted means no biasing.
oneOf:
- type: 'null'
x-parser-schema-id:
- type: array
maxItems: 1000
items:
oneOf:
- type: string
x-parser-schema-id:
- type: object
required:
- term
properties:
term:
type: string
minLength: 1
description: The vocabulary term to bias transcription toward.
x-parser-schema-id:
definition:
type: string
maxLength: 256
description: Optional short definition of the term.
x-parser-schema-id:
pronunciations:
type: array
maxItems: 16
items:
type: string
maxLength: 128
x-parser-schema-id:
description: >-
Optional pronunciation hints, as X-SAMPA notation
or hyphen-separated ASCII respelling with the
stressed syllable in capitals (e.g. GOO-guhl).
Printable ASCII only.
x-parser-schema-id:
x-parser-schema-id: CustomTerm
x-parser-schema-id:
x-parser-schema-id:
x-parser-schema-id:
x-parser-schema-id: STTStreamConfig
title: Stream Configuration
description: >
Optional JSON configuration sent as the FIRST WebSocket text frame,
before any audio. When the first frame is text it is parsed as this
configuration; when the first frame is binary it is treated as the
first
audio chunk and the query-parameter defaults apply. A field that has a
matching query parameter overrides it. Supply custom_terms in this
frame.
example: |-
{
"custom_terms": [
"Modulate",
{
"term": "Velma",
"definition": "Modulate's conversation understanding model family",
"pronunciations": [
"VEL-muh"
]
}
]
}
bindings: []
extensions:
- id: x-parser-unique-object-id
value: streamConfig
- &ref_8
id: audioChunk
contentType: application/octet-stream
payload:
- type: string
format: binary
x-parser-schema-id:
name: Audio Data Chunk
description: Raw audio data chunk for real-time transcription
headers: []
jsonPayloadSchema:
type: string
format: binary
x-parser-schema-id:
title: Audio Data Chunk
description: Raw audio data chunk for real-time transcription
example: '{}'
bindings: []
extensions:
- id: x-parser-unique-object-id
value: audioChunk
- &ref_9
id: endOfStream
contentType: text/plain
payload:
- type: string
const: ''
x-parser-schema-id:
name: End of Audio Stream
description: Empty text frame signaling end of audio stream
headers: []
jsonPayloadSchema:
type: string
const: ''
x-parser-schema-id:
title: End of Audio Stream
description: Empty text frame signaling end of audio stream
example: '""'
bindings: []
extensions:
- id: x-parser-unique-object-id
value: endOfStream
bindings: []
extensions: &ref_0
- id: x-parser-unique-object-id
value: velma2SttStreaming
- &ref_1
id: receiveTranscription
title: Receive transcription
description: Receive transcription results from the server
type: receive
messages:
- &ref_3
id: utteranceResult
contentType: application/json
payload:
- name: Utterance Result
description: A completed utterance with detected language
type: object
properties:
- name: type
type: string
description: utterance
required: true
- name: utterance
type: object
required: true
properties:
- name: utterance_uuid
type: string
description: >
Unique identifier for this utterance. Reuses the
identifier from any
preceding partials for this utterance. A final without
preceding
partials has its own identifier.
required: true
- name: text
type: string
description: >
The transcribed text for this utterance. Represents a
single continuous segment
of speech detected in the audio stream. May contain
PII/PHI tags if tagging
is enabled. Always present; may be an empty string.
required: true
- name: start_ms
type: integer
description: >
The start time of this utterance in milliseconds, relative
to the beginning
of the audio stream. Allows precise synchronization with
the original audio timing.
required: true
- name: duration_ms
type: integer
description: The duration of this utterance in milliseconds.
required: true
- name: speaker
type: integer
description: >
Speaker identifier for this utterance (1-indexed). When
speaker diarization is
enabled, this identifies which speaker produced this
utterance. Speaker numbers
are consistent within a single connection but may vary
between different connections.
required: true
- name: language
type: string
description: >
The automatically detected language code for this
utterance (e.g., "en" for English,
"fr" for French).
required: true
- name: emotion
type: string
description: >
The detected emotional tone of the speaker for this
utterance. Only present when
`emotion_signal` is enabled. Possible values are: Angry,
Disgusted, Afraid, Happy,
Neutral, Other, Sad, Surprised, and Unknown.
required: true
- name: accent
type: string
description: >
The detected accent of the speaker for this utterance.
Only present when
`accent_signal` is enabled. Possible values include:
American, British, Australian,
Southern, Indian, Irish, Scottish, Eastern_European,
African, Asian,
Latin_American, Middle_Eastern, Unknown.
required: true
- name: deepfake_score
type: number
description: >
Synthetic voice (deepfake) detection score for this
utterance. Null if deepfake
detection is disabled or if the utterance is too short for
analysis (< 0.5 seconds).
Values range from 0.0 (likely natural human speech) to 1.0
(likely AI-generated
synthetic speech).
required: true
headers: []
jsonPayloadSchema:
type: object
required:
- type
- utterance
properties:
type:
type: string
const: utterance
x-parser-schema-id:
utterance:
type: object
required:
- utterance_uuid
- text
- start_ms
- duration_ms
- speaker
- language
- emotion
- accent
- deepfake_score
properties:
utterance_uuid:
type: string
format: uuid
description: >
Unique identifier for this utterance. Reuses the identifier
from any
preceding partials for this utterance. A final without
preceding
partials has its own identifier.
example: a1b2c3d4-e5f6-7890-abcd-ef1234567890
x-parser-schema-id:
text:
type: string
description: >
The transcribed text for this utterance. Represents a single
continuous segment
of speech detected in the audio stream. May contain PII/PHI
tags if tagging
is enabled. Always present; may be an empty string.
example: Hello, how are you today?
x-parser-schema-id:
start_ms:
type: integer
minimum: 0
description: >
The start time of this utterance in milliseconds, relative
to the beginning
of the audio stream. Allows precise synchronization with the
original audio timing.
example: 0
x-parser-schema-id:
duration_ms:
type: integer
minimum: 0
description: The duration of this utterance in milliseconds.
example: 2500
x-parser-schema-id:
speaker:
type: integer
description: >
Speaker identifier for this utterance (1-indexed). When
speaker diarization is
enabled, this identifies which speaker produced this
utterance. Speaker numbers
are consistent within a single connection but may vary
between different connections.
example: 1
x-parser-schema-id:
language:
type: string
description: >
The automatically detected language code for this utterance
(e.g., "en" for English,
"fr" for French).
example: en
x-parser-schema-id:
emotion:
oneOf:
- type: string
x-parser-schema-id:
- type: 'null'
x-parser-schema-id:
description: >
The detected emotional tone of the speaker for this
utterance. Only present when
`emotion_signal` is enabled. Possible values are: Angry,
Disgusted, Afraid, Happy,
Neutral, Other, Sad, Surprised, and Unknown.
example: Neutral
x-parser-schema-id:
accent:
oneOf:
- type: string
x-parser-schema-id:
- type: 'null'
x-parser-schema-id:
description: >
The detected accent of the speaker for this utterance. Only
present when
`accent_signal` is enabled. Possible values include:
American, British, Australian,
Southern, Indian, Irish, Scottish, Eastern_European,
African, Asian,
Latin_American, Middle_Eastern, Unknown.
example: American
x-parser-schema-id:
deepfake_score:
oneOf:
- type: number
format: double
minimum: 0
maximum: 1
x-parser-schema-id:
- type: 'null'
x-parser-schema-id:
description: >
Synthetic voice (deepfake) detection score for this
utterance. Null if deepfake
detection is disabled or if the utterance is too short for
analysis (< 0.5 seconds).
Values range from 0.0 (likely natural human speech) to 1.0
(likely AI-generated
synthetic speech).
example: 0.05
x-parser-schema-id:
x-parser-schema-id: StreamingUtterance
x-parser-schema-id:
title: Utterance Result
description: A completed utterance with detected language
example: |-
{
"type": "utterance",
"utterance": {
"utterance_uuid": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
"text": "Bonjour, comment allez-vous?",
"start_ms": 0,
"duration_ms": 2800,
"speaker": 1,
"language": "fr",
"emotion": null,
"accent": null,
"deepfake_score": null
}
}
bindings: []
extensions:
- id: x-parser-unique-object-id
value: utteranceResult
- &ref_4
id: partialUtteranceResult
contentType: application/json
payload:
- name: Partial Utterance Result
description: In-progress transcription text (interim, not yet finalized)
type: object
properties:
- name: type
type: string
description: partial_utterance
required: true
- name: partial_utterance
type: object
required: true
properties:
- name: utterance_uuid
type: string
description: >
Stable identifier shared by every partial and the final
result for
this utterance. Use this value to correlate interleaved
utterances,
including when start_ms is null.
required: true
- name: text
type: string
description: >
In-progress transcription text. This is a preview that may
change as more
audio is processed. Each new partial replaces the previous
one for the same
utterance, and the finalized utterance supersedes the
partials that preceded it
for that utterance. A partial for a later utterance may
arrive before an earlier
finalized utterance. Always present; may be an empty
string.
required: true
- name: start_ms
type: integer
description: >
Start time of the in-progress speech segment in
milliseconds, relative to the
beginning of the audio stream. May be null if timing data
is not yet available.
required: true
- name: speaker
type: integer
description: >
Speaker identifier (1-indexed). Matches the speaker
numbering used in
finalized utterances. May be null if speaker has not yet
been identified.
required: true
- name: emotion
type: string
description: >
Latest interim emotion label for the in-progress
utterance, or null when the
emotion signal is not enabled or no value is available
yet.
required: true
- name: accent
type: string
description: >
Latest interim accent label for the in-progress utterance,
or null when the
accent signal is not enabled or no value is available yet.
required: true
- name: deepfake_score
type: number
description: >
Latest interim deepfake score for the in-progress
utterance, or null when the
deepfake signal is not enabled or no value is available
yet.
required: true
headers: []
jsonPayloadSchema:
type: object
required:
- type
- partial_utterance
properties:
type:
type: string
const: partial_utterance
x-parser-schema-id:
partial_utterance:
type: object
required:
- utterance_uuid
- text
- start_ms
- speaker
- emotion
- accent
- deepfake_score
properties:
utterance_uuid:
type: string
format: uuid
description: >
Stable identifier shared by every partial and the final
result for
this utterance. Use this value to correlate interleaved
utterances,
including when start_ms is null.
example: b2c3d4e5-f6a7-8901-bcde-f12345678901
x-parser-schema-id:
text:
type: string
description: >
In-progress transcription text. This is a preview that may
change as more
audio is processed. Each new partial replaces the previous
one for the same
utterance, and the finalized utterance supersedes the
partials that preceded it
for that utterance. A partial for a later utterance may
arrive before an earlier
finalized utterance. Always present; may be an empty string.
example: Hello, how are you
x-parser-schema-id:
start_ms:
oneOf:
- type: integer
minimum: 0
x-parser-schema-id:
- type: 'null'
x-parser-schema-id:
description: >
Start time of the in-progress speech segment in
milliseconds, relative to the
beginning of the audio stream. May be null if timing data is
not yet available.
example: 0
x-parser-schema-id:
speaker:
oneOf:
- type: integer
x-parser-schema-id:
- type: 'null'
x-parser-schema-id:
description: >
Speaker identifier (1-indexed). Matches the speaker
numbering used in
finalized utterances. May be null if speaker has not yet
been identified.
example: 1
x-parser-schema-id:
emotion:
oneOf:
- type: string
x-parser-schema-id:
- type: 'null'
x-parser-schema-id:
description: >
Latest interim emotion label for the in-progress utterance,
or null when the
emotion signal is not enabled or no value is available yet.
x-parser-schema-id:
accent:
oneOf:
- type: string
x-parser-schema-id:
- type: 'null'
x-parser-schema-id:
description: >
Latest interim accent label for the in-progress utterance,
or null when the
accent signal is not enabled or no value is available yet.
x-parser-schema-id:
deepfake_score:
oneOf:
- type: number
format: double
minimum: 0
maximum: 1
x-parser-schema-id:
- type: 'null'
x-parser-schema-id:
description: >
Latest interim deepfake score for the in-progress utterance,
or null when the
deepfake signal is not enabled or no value is available yet.
x-parser-schema-id:
x-parser-schema-id: PartialUtterance
x-parser-schema-id:
title: Partial Utterance Result
description: In-progress transcription text (interim, not yet finalized)
example: |-
{
"type": "partial_utterance",
"partial_utterance": {
"utterance_uuid": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
"text": "Bonjour, comment",
"start_ms": 0,
"speaker": 1,
"emotion": null,
"accent": null,
"deepfake_score": null
}
}
bindings: []
extensions:
- id: x-parser-unique-object-id
value: partialUtteranceResult
- &ref_5
id: errorResult
contentType: application/json
payload:
- name: Error
description: An error occurred during transcription
type: object
properties:
- name: type
type: string
description: error
required: true
- name: error
type: string
description: Human-readable error message
required: true
headers: []
jsonPayloadSchema:
type: object
required:
- type
- error
properties:
type:
type: string
const: error
x-parser-schema-id:
error:
type: string
description: Human-readable error message
x-parser-schema-id:
x-parser-schema-id:
title: Error
description: An error occurred during transcription
example: |-
{
"type": "error",
"error": "Internal server error"
}
bindings: []
extensions:
- id: x-parser-unique-object-id
value: errorResult
- &ref_6
id: doneResult
contentType: application/json
payload:
- name: Transcription Complete
description: Transcription completed successfully
type: object
properties:
- name: type
type: string
description: done
required: true
- name: duration_ms
type: integer
description: Total duration of audio processed in milliseconds
required: true
headers: []
jsonPayloadSchema:
type: object
required:
- type
- duration_ms
properties:
type:
type: string
const: done
x-parser-schema-id:
duration_ms:
type: integer
minimum: 0
description: Total duration of audio processed in milliseconds
x-parser-schema-id:
x-parser-schema-id:
title: Transcription Complete
description: Transcription completed successfully
example: |-
{
"type": "done",
"duration_ms": 45000
}
bindings: []
extensions:
- id: x-parser-unique-object-id
value: doneResult
bindings: []
extensions: *ref_0
sendOperations:
- *ref_1
receiveOperations:
- *ref_2
sendMessages:
- *ref_3
- *ref_4
- *ref_5
- *ref_6
receiveMessages:
- *ref_7
- *ref_8
- *ref_9
extensions:
- id: x-parser-unique-object-id
value: velma2SttStreaming
securitySchemes: []
````
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.