> ## Documentation Index > Fetch the complete documentation index at: https://docs.modulate.ai/llms.txt > Use this file to discover all available pages before exploring further. # Velma Streaming > Real-time conversation analysis over WebSocket — stream audio and receive clips, conversation type, participant roles, behaviors, topics, sentiment, and a summary as they are produced. Real-time conversation analysis over WebSocket. Stream audio to the server and receive analysis events as they are produced: transcribed clips, the inferred conversation type, participant roles, behavior detections, topics, per-speaker topic sentiment, and a summary — terminating with a final `done` event. ## Endpoint ```text theme={null} wss://platform.modulate.ai/api/velma-2-streaming ``` ## Authentication Pass your API key as a query parameter when opening the connection. ```text theme={null} wss://platform.modulate.ai/api/velma-2-streaming?api_key=YOUR_API_KEY ``` Unlike the batch endpoint, the streaming API does not use an `X-API-Key` header. The key must be in the query string at connection time. See [Authentication and rate limits](/guides/authentication) for how to obtain and manage API keys. ## Connection parameters Connection parameters carry only the API key and audio-format hints. All analysis configuration is sent in the [config frame](#configuration), not the query string. | Parameter | Type | Required | Description | | - | - | - | - | | `api_key` | string | Yes | Your API key | | `audio_format` | string | Raw formats only | Audio encoding. Omit for self-describing formats; required for raw/headerless formats. May optionally be set to override auto-detection | | `sample_rate` | integer | Raw formats only | Sample rate in Hz. Required for raw formats; must not be set otherwise | | `num_channels` | integer | Raw formats only | Number of channels (1–8). Required for raw formats; must not be set otherwise | ## Supported audio formats **Self-describing formats** (auto-detected from file headers — no extra parameters needed): AAC, AIFF, FLAC, MP3, OGG, WAV, WebM **OGG / Opus:** OGG is a container that may carry Opus-encoded audio. Pass `audio_format=ogg`, not `audio_format=opus`. **Raw / headerless formats** (require `audio_format`, `sample_rate`, and `num_channels`): `s8`, `s16le`, `s16be`, `s24le`, `s24be`, `s32le`, `s32be`, `u8`, `u16le`, `u16be`, `u24le`, `u24be`, `u32le`, `u32be`, `f32le`, `f32be`, `f64le`, `f64be`, `mulaw`, `alaw` Valid sample rates: `8000`, `11025`, `16000`, `22050`, `32000`, `44100`, `48000`, `96000` ## Configuration After the connection opens, send exactly **one text frame before any audio**: * The literal string `default` to use the built-in default configuration, **or** * A JSON-encoded `BatchConfig` describing the conversation types, participant roles, behaviors, STT options, and which aggregate outputs (topics, sentiments, summary) to produce. The `BatchConfig` schema is identical to the batch endpoint — see the [Batch](/api-reference/velma/batch) reference for the full field list. Behaviors may be referenced from the preset catalog with the `preset:` syntax; list available presets with [List behavior presets](/api-reference/velma/presets). ```json theme={null} { "behaviors": ["preset:conflict_de_escalation", "preset:complaints"], "stt": { "speaker_diarization": true, "emotion_signal": true }, "produce_topics": true, "produce_topic_sentiments": true, "produce_summary": true } ``` You must send the config frame before any audio. Sending a binary audio frame before the config frame is a protocol error and closes the connection with code `1003`. ## Connection flow 1. Connect to the WebSocket endpoint with `api_key` (and `audio_format`, `sample_rate`, `num_channels` for raw formats). 2. Send one text frame: either the literal string `default` or a JSON-encoded `BatchConfig`. 3. Stream audio as **binary** WebSocket frames. Frames can be any size. 4. Receive analysis events as JSON text frames as results are produced. 5. Send an **empty text frame** (`""`) to signal end of audio. 6. Receive a final `done` event with the total audio duration. 7. The connection closes automatically. ## Server events The server emits JSON text frames as results are produced. Every event carries a `type` discriminator. The payload objects (clip, conversation-type pick, participant-role pick, behavior detection, topic sentiment) use the same schemas the batch endpoint returns — see the [Batch](/api-reference/velma/batch) reference for full field definitions. The `partial_clip` and `clip_update` payloads are streaming-only and documented below. | `type` | Payload key | Description | | - | - | - | | `clip` | `clip` | A transcribed clip with speaker label, timing, and optional emotion / accent / deepfake signals | | `partial_clip` | `partial_clip` | An in-progress clip streamed while an utterance is still being spoken, before it finalizes | | `clip_update` | `clip_update` | Refined values for a previously finalized clip | | `conversation_type` | `pick` | The inferred or default conversation-type classification for the session | | `participant_role` | `pick` | The inferred or default role for a speaker | | `behavior_detection` | `detection` | A per-behavior detection result | | `topics` | `topics` | The aggregated list of conversation topics; each event fully replaces the previous one | | `topic_sentiment` | `topic_sentiment` | Per-speaker sentiment for one aggregated topic; a later event supersedes an earlier one for the same topic and speaker | | `summary` | `text` | A free-form summary of the conversation; each event fully replaces the previous one | | `done` | `duration_ms` | Streaming completed; carries the total audio duration | | `error` | `error` | An error occurred; the connection closes after this event | ### `clip` A transcribed clip. Emitted progressively as speech is processed. ```json theme={null} { "type": "clip", "clip": { "clip_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", "text": "Thanks for calling support, how can I help?", "start_ms": 0, "duration_ms": 3200, "speaker_label": "speaker_1", "language": "en", "emotion": null, "accent": null, "deepfake_score": null } } ``` The `emotion`, `accent`, and `deepfake_score` fields are `null` unless the corresponding STT options are enabled in the config frame. ### `partial_clip` An in-progress clip streamed while an utterance is still being spoken, before it finalizes. Multiple partials may be emitted for the same `clip_uuid` as the utterance grows; the eventual `clip` event for that utterance reuses the same `clip_uuid`, so a run of partials can be correlated with its final clip. ```json theme={null} { "type": "partial_clip", "partial_clip": { "clip_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", "text": "Thanks for calling support,", "start_ms": 0, "end_ms": 1800, "speaker_label": "speaker_1", "emotion": null, "accent": null, "deepfake_score": null } } ``` `end_ms` is the current end of the in-progress utterance and grows as the utterance extends; it is `null` until available. The finalized clip reports `duration_ms` instead of `end_ms`. `speaker_label` is `null` until diarization resolves the speaker. `emotion`, `accent`, and `deepfake_score` carry the latest in-progress values, or `null` when no value is available yet. ### `clip_update` Refined values for a previously finalized clip; `clip_update.clip_uuid` matches the `clip_uuid` of an earlier `clip` event. A finalized clip may receive any number of `clip_update` events (including none), each emitted after that clip's `clip` event — possibly interleaved with events for other clips — and always before the `done` event on clean completion. For each field present, the latest received value supersedes the value on the `clip` event and on any earlier `clip_update` for that clip. ```json theme={null} { "type": "clip_update", "clip_update": { "clip_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", "emotion": "Calm", "accent": "American" } } ``` ### `conversation_type` The conversation-type classification for the session. ```json theme={null} { "type": "conversation_type", "pick": { "conversation_type_uuid": "8f1d2c3b-4a5e-6f70-8192-a3b4c5d6e7f8", "name": "Customer support call", "confidence": 0.92, "selection_source": "inferred", "detail": "Caller is seeking help resolving a billing issue.", "reasoning": "The agent greets the caller and offers assistance with an account problem." } } ``` `selection_source` is one of `inferred`, `auto_selected_single_option`, or `default`. ### `participant_role` A role assignment for one speaker. Emitted once per identified speaker. ```json theme={null} { "type": "participant_role", "pick": { "speaker_label": "speaker_1", "participant_role_uuid": "2b7c9d10-1e2f-3a4b-5c6d-7e8f90a1b2c3", "name": "Support agent", "confidence": 0.88, "selection_source": "inferred", "detail": "Speaker offers assistance and asks diagnostic questions.", "reasoning": "Greets on behalf of the company and drives the troubleshooting." } } ``` ### `behavior_detection` A per-behavior detection result. Emitted once per configured behavior. ```json theme={null} { "type": "behavior_detection", "detection": { "behavior_uuid": "c4d5e6f7-8091-a2b3-c4d5-e6f708192a3b", "behavior_name": "Empathy", "speaker_label": "speaker_1", "detected": true, "confidence": 0.81, "evidence_clip_uuids": ["a1b2c3d4-e5f6-7890-abcd-ef1234567890"], "definitive_clip_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", "reasoning": "Agent acknowledges the customer's frustration before resolving the issue." } } ``` ### `topics` The aggregated list of conversation topics. May be emitted more than once as the stream progresses; each event fully replaces the previous `topics` event, so always treat the latest as authoritative and never merge with earlier ones. ```json theme={null} { "type": "topics", "topics": ["billing", "refund policy", "account access"] } ``` ### `topic_sentiment` Per-speaker sentiment for one aggregated topic, keyed by topic and speaker. May be emitted more than once as the stream progresses; a later event supersedes an earlier one for the same topic and speaker. ```json theme={null} { "type": "topic_sentiment", "topic_sentiment": { "topic": "billing", "speaker_label": "speaker_2", "sentiment_score": -0.4, "sentiment_label": "negative" } } ``` `sentiment_score` ranges from `-1.0` (most negative) to `1.0` (most positive). ### `summary` A free-form summary of the conversation. May be emitted more than once as the stream progresses; each event fully replaces the previous `summary` event, so always treat the latest as authoritative. ```json theme={null} { "type": "summary", "text": "The customer called about a duplicate charge. The agent confirmed the error, issued a refund, and explained the billing cycle." } ``` ### `done` Sent after all audio has been processed, in response to the end-of-stream signal. ```json theme={null} { "type": "done", "duration_ms": 45000 } ``` ### `error` Sent if processing fails. The connection closes after this event. ```json theme={null} { "type": "error", "error": "Internal server error" } ``` ## WebSocket close codes | Code | Meaning | | - | - | | `1000` | Normal closure after a successful `done` message | | `1003` | Invalid or missing query parameters (unsupported audio format, sample rate, or channel count), or a protocol or configuration error — audio sent before the configuration frame, an unexpected text frame after configuration, invalid JSON, or an invalid configuration value (missing required field, unknown preset) | | `1011` | Internal server error, or processing ended before completion | | `4001` | The `api_key` query parameter is missing or invalid | | `4003` | The request is not permitted | | `4004` | The API key does not have access to this model | | `4029` | Insufficient credits | | `4030` | Concurrent request limit reached | | `4031` | Monthly usage limit reached | An `error` JSON message is sent before the connection closes (except on `1000`). ## Rate limits * Concurrent connection limits apply per model. * Connections that exceed limits are rejected during the WebSocket handshake with close code `4030`. See [Authentication and rate limits](/guides/authentication) for retry guidance. ## Examples The examples below send a JSON `BatchConfig` as the config frame. To use the built-in defaults instead, send the literal string `"default"` in place of the JSON. ```python theme={null} import asyncio import json import aiohttp API_KEY = "YOUR_API_KEY" AUDIO_FILE = "conversation.ogg" CHUNK_SIZE = 8192 # Either the string "default" or a JSON-encoded BatchConfig. CONFIG = json.dumps({ "behaviors": ["preset:conflict_de_escalation", "preset:complaints"], "stt": {"speaker_diarization": True, "emotion_signal": True}, "produce_topics": True, "produce_topic_sentiments": True, "produce_summary": True, }) async def analyze_streaming(): url = f"wss://platform.modulate.ai/api/velma-2-streaming?api_key={API_KEY}" async with aiohttp.ClientSession() as session: async with session.ws_connect(url) as ws: # 1. Send the config frame before any audio. await ws.send_str(CONFIG) async def send_audio(): with open(AUDIO_FILE, "rb") as f: while chunk := f.read(CHUNK_SIZE): await ws.send_bytes(chunk) await asyncio.sleep(CHUNK_SIZE / 4000) await ws.send_str("") # 2. Signal end of audio. send_task = asyncio.create_task(send_audio()) try: async for msg in ws: if msg.type == aiohttp.WSMsgType.TEXT: event = json.loads(msg.data) etype = event["type"] if etype == "clip": c = event["clip"] print(f"[{c['speaker_label']}] {c['text']}") elif etype == "partial_clip": pc = event["partial_clip"] print(f"[partial {pc['clip_uuid'][:8]}] {pc['text']}") elif etype == "clip_update": cu = event["clip_update"] print(f"[update {cu['clip_uuid'][:8]}] emotion={cu.get('emotion')} accent={cu.get('accent')}") elif etype == "conversation_type": print(f"Conversation type: {event['pick']['name']}") elif etype == "participant_role": p = event["pick"] print(f"Role for {p['speaker_label']}: {p['name']}") elif etype == "behavior_detection": d = event["detection"] print(f"Behavior {d['behavior_name']}: detected={d['detected']}") elif etype == "topics": print(f"Topics: {', '.join(event['topics'])}") elif etype == "topic_sentiment": ts = event["topic_sentiment"] print(f"Sentiment ({ts['topic']}, {ts['speaker_label']}): {ts['sentiment_label']}") elif etype == "summary": print(f"Summary: {event['text']}") elif etype == "done": print(f"Done. Duration: {event['duration_ms']}ms") break elif etype == "error": print(f"Error: {event['error']}") break elif msg.type in ( aiohttp.WSMsgType.ERROR, aiohttp.WSMsgType.CLOSE, aiohttp.WSMsgType.CLOSED, ): break finally: if not send_task.done(): send_task.cancel() asyncio.run(analyze_streaming()) ``` ```javascript theme={null} const WebSocket = require("ws"); const fs = require("fs"); const API_KEY = "YOUR_API_KEY"; const AUDIO_FILE = "conversation.ogg"; const CHUNK_SIZE = 8192; // Either the string "default" or a JSON-encoded BatchConfig. const CONFIG = JSON.stringify({ behaviors: ["preset:conflict_de_escalation", "preset:complaints"], stt: { speaker_diarization: true, emotion_signal: true }, produce_topics: true, produce_topic_sentiments: true, produce_summary: true, }); const url = new URL("wss://platform.modulate.ai/api/velma-2-streaming"); url.searchParams.set("api_key", API_KEY); const ws = new WebSocket(url.toString()); ws.on("open", () => { // 1. Send the config frame before any audio. ws.send(CONFIG); // 2. Stream audio as binary frames, then signal end of audio. const stream = fs.createReadStream(AUDIO_FILE, { highWaterMark: CHUNK_SIZE }); stream.on("data", (chunk) => ws.send(chunk)); stream.on("end", () => ws.send("")); }); ws.on("message", (data) => { const event = JSON.parse(data.toString()); switch (event.type) { case "clip": console.log(`[${event.clip.speaker_label}] ${event.clip.text}`); break; case "partial_clip": console.log(`[partial ${event.partial_clip.clip_uuid.slice(0, 8)}] ${event.partial_clip.text}`); break; case "clip_update": { const cu = event.clip_update; console.log(`[update ${cu.clip_uuid.slice(0, 8)}] emotion=${cu.emotion} accent=${cu.accent}`); break; } case "conversation_type": console.log(`Conversation type: ${event.pick.name}`); break; case "participant_role": console.log(`Role for ${event.pick.speaker_label}: ${event.pick.name}`); break; case "behavior_detection": console.log(`Behavior ${event.detection.behavior_name}: detected=${event.detection.detected}`); break; case "topics": console.log(`Topics: ${event.topics.join(", ")}`); break; case "topic_sentiment": { const ts = event.topic_sentiment; console.log(`Sentiment (${ts.topic}, ${ts.speaker_label}): ${ts.sentiment_label}`); break; } case "summary": console.log(`Summary: ${event.text}`); break; case "done": console.log(`Done. Duration: ${event.duration_ms}ms`); ws.close(); break; case "error": console.error(`Error: ${event.error}`); ws.close(); break; } }); ws.on("error", (err) => console.error("WebSocket error:", err.message)); ``` WebSocket APIs cannot be tested with cURL. For command-line testing, use [`websocat`](https://github.com/vi/websocat). ## Related * [Velma overview](/api-reference/velma/overview) — what Velma analyzes and when to use batch vs streaming * [Velma Batch](/api-reference/velma/batch) — the `BatchConfig` and event payload schemas in full * [List behavior presets](/api-reference/velma/presets) — discover behavior preset identifiers * [Authentication and rate limits](/guides/authentication) ## AsyncAPI ````yaml api/velma_2_streaming.yaml velma2Streaming id: velma2Streaming title: Velma2 streaming description: > WebSocket channel for streaming velma-2. ## Connection Flow 1. Connect with `api_key` and optional audio-format query parameters. 2. Connection is accepted if authentication succeeds and limits are not exceeded. 3. Send one text frame: either the literal string `default` or a JSON-encoded `BatchConfig`, within 60 seconds of acceptance. 4. Stream audio data as binary WebSocket frames. 5. Send an empty text frame ("") to signal end of audio. 6. Receive streaming events as JSON messages. 7. Receive a final `done` event with total audio duration. 8. Connection closes automatically after completion. ## Error Handling If an error occurs, the server sends an `error` event and closes the connection. Connection rejections during handshake use WebSocket close codes. ## WebSocket Close Codes - `1000`: Normal closure after the `done` message, or after the connection completes. - `1003`: Invalid or missing query parameters (unsupported audio format / sample rate / channel count), or a protocol or configuration error. An `error` message is sent before the close. Missing the first-config deadline or sending audio before the configuration text frame sends the `error` message "A configuration text frame must be sent before any audio data."; a further text frame after the configuration was already provided sends "Unexpected text frame after the configuration was already provided."; unparseable JSON or a non-object configuration sends "Invalid configuration."; an invalid configuration value sends a descriptive validation message (e.g. "behaviors.0.name: missing required field" or "behaviors.0: invalid behavior preset; use list-presets to discover available presets"). - `1011`: Internal server error, or processing ended before completion. An `error` message is sent before the close; when processing ends without a `done` message, the `error` message is "Unable to complete the request. Please try again." - `4001`: The `api_key` query parameter is missing or invalid. An `error` message is sent before the close. - `4003`: The request is not permitted. An `error` message is sent before the close. - `4004`: The API key does not have access to this model. An `error` message is sent before the close. - `4029`: The request could not be completed due to insufficient credits. An `error` message is sent before the close. - `4030`: Concurrent request limit reached. An `error` message is sent before the close. - `4031`: Monthly usage limit reached. An `error` message is sent before the close. servers: - id: production protocol: wss host: platform.modulate.ai bindings: [] variables: [] address: /api/velma-2-streaming parameters: [] bindings: - protocol: ws version: latest value: query: type: object required: - api_key properties: api_key: type: string description: | Your API key for authentication. Required. Must be provided as a query parameter when establishing the WebSocket connection; the connection is rejected during the handshake if it is missing. audio_format: type: string description: | Audio encoding format. Self-describing formats (AAC, AIFF, FLAC, MP3, OGG, WAV, WebM) are auto-detected when omitted. Raw/headerless formats (PCM, mu-law, A-law) require this parameter plus `sample_rate` and `num_channels`. enum: - s8 - s16le - s16be - s24le - s24be - s32le - s32be - u8 - u16le - u16be - u24le - u24be - u32le - u32be - f32le - f32be - f64le - f64be - mulaw - alaw - wav - mp3 - ogg - flac - webm - aac - aiff sample_rate: type: integer description: | Audio sample rate in Hz. Required when `audio_format` is a raw/headerless format. Cannot be specified without `audio_format`. enum: - 8000 - 11025 - 16000 - 22050 - 32000 - 44100 - 48000 - 96000 num_channels: type: integer minimum: 1 maximum: 8 description: | Number of audio channels (1-8). Required when `audio_format` is a raw/headerless format. Cannot be specified without `audio_format`. schemaProperties: - name: query type: object required: false properties: - name: api_key type: string description: | Your API key for authentication. Required. Must be provided as a query parameter when establishing the WebSocket connection; the connection is rejected during the handshake if it is missing. required: true - name: audio_format type: string description: | Audio encoding format. Self-describing formats (AAC, AIFF, FLAC, MP3, OGG, WAV, WebM) are auto-detected when omitted. Raw/headerless formats (PCM, mu-law, A-law) require this parameter plus `sample_rate` and `num_channels`. enumValues: - s8 - s16le - s16be - s24le - s24be - s32le - s32be - u8 - u16le - u16be - u24le - u24be - u32le - u32be - f32le - f32be - f64le - f64be - mulaw - alaw - wav - mp3 - ogg - flac - webm - aac - aiff required: false - name: sample_rate type: integer description: | Audio sample rate in Hz. Required when `audio_format` is a raw/headerless format. Cannot be specified without `audio_format`. enumValues: - 8000 - 11025 - 16000 - 22050 - 32000 - 44100 - 48000 - 96000 required: false - name: num_channels type: integer description: | Number of audio channels (1-8). Required when `audio_format` is a raw/headerless format. Cannot be specified without `audio_format`. required: false operations: - &ref_6 id: sendStreamConfig title: Send stream config description: First text frame - `default` or JSON-encoded BatchConfig. type: send messages: - &ref_20 id: streamConfig contentType: application/json payload: - oneOf: &ref_2 - type: object properties: conversation_types: description: > Candidate conversation types. If this key is omitted, the built-in default conversation types are used; pass an empty array to provide no candidates. type: array items: &ref_0 type: object required: - conversation_type_uuid - name - short_description - detailed_description properties: conversation_type_uuid: type: string format: uuid x-parser-schema-id: name: type: string minLength: 1 x-parser-schema-id: short_description: type: string x-parser-schema-id: detailed_description: type: string x-parser-schema-id: x-parser-schema-id: ConversationTypeDef x-parser-schema-id: participant_roles: description: | Candidate participant roles. If this key is omitted, the built-in default participant roles are used; pass an empty array to provide no candidates. type: array items: &ref_1 type: object required: - participant_role_uuid - name - short_description - detailed_description properties: participant_role_uuid: type: string format: uuid x-parser-schema-id: name: type: string minLength: 1 x-parser-schema-id: short_description: type: string x-parser-schema-id: detailed_description: type: string x-parser-schema-id: applies_to_conversation_type_uuids: description: >- Null/missing means applies to all conversation types. oneOf: - type: array items: type: string format: uuid x-parser-schema-id: x-parser-schema-id: - type: 'null' x-parser-schema-id: x-parser-schema-id: x-parser-schema-id: ParticipantRoleDef x-parser-schema-id: behaviors: description: | Each entry is either a full BehaviorDef object or a preset reference string matching `preset:` (e.g. `preset:complaints`). Preset references are expanded into full BehaviorDef objects before processing. Use the list-presets endpoint to discover available identifiers. If this key is omitted, no behaviors are evaluated. type: array items: oneOf: - type: object required: - behavior_uuid - name - short_description - detailed_description properties: behavior_uuid: type: string format: uuid x-parser-schema-id: name: type: string minLength: 1 x-parser-schema-id: short_description: type: string x-parser-schema-id: detailed_description: type: string x-parser-schema-id: applies_to_conversation_type_uuids: oneOf: - type: array items: type: string format: uuid x-parser-schema-id: x-parser-schema-id: - type: 'null' x-parser-schema-id: x-parser-schema-id: applies_to_participant_role_uuids: oneOf: - type: array items: type: string format: uuid x-parser-schema-id: x-parser-schema-id: - type: 'null' x-parser-schema-id: x-parser-schema-id: x-parser-schema-id: BehaviorDef - type: string pattern: ^preset:.+$ x-parser-schema-id: x-parser-schema-id: x-parser-schema-id: stt: type: object description: >- Speech-to-text options for the transcription stage: the per-request STT flags, the language hint, and custom_terms. properties: speaker_diarization: type: boolean default: true x-parser-schema-id: emotion_signal: type: boolean default: false description: >- Accepted for compatibility. Emotion detection is always enabled regardless of this flag. x-parser-schema-id: accent_signal: type: boolean default: false description: >- Accepted for compatibility. Accent detection is always enabled regardless of this flag. x-parser-schema-id: deepfake_signal: type: boolean default: false description: >- Accepted for compatibility. Deepfake detection is always enabled regardless of this flag. x-parser-schema-id: pii_phi_tagging: type: boolean default: false description: >- Tag finalized clips and summaries for PII/PHI. Partial clips remain untagged. Text whose annotations cannot be verified is returned unchanged; tagging may delay finalized clips and subsequent events. x-parser-schema-id: language: description: > Optional language hint as a case-insensitive ISO 639-1 code (e.g. "en", "fr"). BCP 47 region/script subtags (e.g. "en-US") are accepted; only the primary language subtag is used. Whitespace is trimmed. When omitted, null, or blank, the language is detected automatically for each clip. oneOf: - type: string pattern: ^\s*([A-Za-z]{2,3}([-_][A-Za-z0-9]+)*)?\s*$ x-parser-schema-id: - type: 'null' x-parser-schema-id: x-parser-schema-id: custom_terms: description: >- Optional custom vocabulary to bias transcription toward domain terms and names. Each entry is either a plain string (the term) or a CustomTerm object. Blank terms and blank pronunciations are dropped. At most 1000 entries, and the term strings serialized together as JSON must total under 8000 characters. Null or omitted means no biasing. oneOf: - type: 'null' x-parser-schema-id: - type: array maxItems: 1000 items: oneOf: - type: string x-parser-schema-id: - type: object required: - term properties: term: type: string minLength: 1 description: >- The vocabulary term to bias transcription toward. x-parser-schema-id: definition: type: string maxLength: 256 description: Optional short definition of the term. x-parser-schema-id: pronunciations: type: array maxItems: 16 items: type: string maxLength: 128 x-parser-schema-id: description: >- Optional pronunciation hints, as X-SAMPA notation or hyphen-separated ASCII respelling with the stressed syllable in capitals (e.g. GOO-guhl). Printable ASCII only. x-parser-schema-id: x-parser-schema-id: CustomTerm x-parser-schema-id: x-parser-schema-id: x-parser-schema-id: x-parser-schema-id: STTOptions produce_topics: type: boolean default: true x-parser-schema-id: produce_topic_sentiments: type: boolean default: true x-parser-schema-id: produce_summary: type: boolean default: true x-parser-schema-id: default_conversation_type: description: > Fallback conversation type used when no candidate conversation type applies. If this key is omitted, a built-in fallback is supplied automatically; set it to null to disable the fallback. oneOf: - *ref_0 - type: 'null' x-parser-schema-id: x-parser-schema-id: default_participant_role: description: > Fallback participant role used when no eligible candidates are available or role inference fails. Its conversation-type restrictions apply whenever a conversation type is known. If this key is omitted, an unrestricted built-in fallback is supplied automatically; set it to null to disable the fallback. oneOf: - *ref_1 - type: 'null' x-parser-schema-id: x-parser-schema-id: x-parser-schema-id: BatchConfig - type: string const: default x-parser-schema-id: x-parser-schema-id: name: Stream Configuration description: | First text frame: either the literal string `default` (to use the built-in default configuration) or a JSON-encoded BatchConfig object. To set STT options such as `custom_terms`, include an `stt` key in the JSON-encoded BatchConfig. headers: [] jsonPayloadSchema: oneOf: *ref_2 x-parser-schema-id: title: Stream Configuration description: | First text frame: either the literal string `default` (to use the built-in default configuration) or a JSON-encoded BatchConfig object. To set STT options such as `custom_terms`, include an `stt` key in the JSON-encoded BatchConfig. example: '{}' bindings: [] extensions: - id: x-parser-unique-object-id value: streamConfig bindings: [] extensions: &ref_3 - id: x-parser-unique-object-id value: velma2Streaming - &ref_7 id: sendAudioChunk title: Send audio chunk description: Binary audio frame. type: send messages: - &ref_21 id: audioChunk contentType: application/octet-stream payload: - type: string format: binary x-parser-schema-id: name: Audio Data Chunk description: Raw audio data chunk. headers: [] jsonPayloadSchema: type: string format: binary x-parser-schema-id: title: Audio Data Chunk description: Raw audio data chunk. example: '{}' bindings: [] extensions: - id: x-parser-unique-object-id value: audioChunk bindings: [] extensions: *ref_3 - &ref_8 id: sendEndOfStream title: Send end of stream description: Empty text frame signaling end of audio. type: send messages: - &ref_22 id: endOfStream contentType: text/plain payload: - type: string const: '' x-parser-schema-id: name: End of Audio Stream description: Empty text frame signalling end of audio stream. headers: [] jsonPayloadSchema: type: string const: '' x-parser-schema-id: title: End of Audio Stream description: Empty text frame signalling end of audio stream. example: '{}' bindings: [] extensions: - id: x-parser-unique-object-id value: endOfStream bindings: [] extensions: *ref_3 - &ref_5 id: receiveStreamingEvents title: Receive streaming events description: Server-emitted events. Discriminated by the `type` field. type: receive messages: - &ref_9 id: clipEvent contentType: application/json payload: - name: Clip description: A transcribed clip with metadata. type: object properties: - name: type type: string description: clip required: true - name: clip type: object required: true properties: - name: clip_uuid type: string required: true - name: text type: string required: true - name: start_ms type: integer required: true - name: duration_ms type: integer required: true - name: speaker_label type: string required: true - name: language type: string required: true - name: emotion type: string description: >- Predicted emotion label for the clip, or null when no emotion label is available. required: false - name: accent type: string description: >- Predicted accent label for the clip, or null when no accent label is available. required: false - name: deepfake_score type: number description: >- Synthetic voice detection score for the clip, or null when no reading is available. required: false headers: [] jsonPayloadSchema: type: object required: - type - clip properties: type: type: string const: clip x-parser-schema-id: clip: type: object required: - clip_uuid - text - start_ms - duration_ms - speaker_label - language properties: clip_uuid: type: string format: uuid x-parser-schema-id: text: type: string x-parser-schema-id: start_ms: type: integer minimum: 0 x-parser-schema-id: duration_ms: type: integer minimum: 0 x-parser-schema-id: speaker_label: type: string x-parser-schema-id: language: type: string x-parser-schema-id: emotion: description: >- Predicted emotion label for the clip, or null when no emotion label is available. oneOf: - type: string x-parser-schema-id: - type: 'null' x-parser-schema-id: x-parser-schema-id: accent: description: >- Predicted accent label for the clip, or null when no accent label is available. oneOf: - type: string x-parser-schema-id: - type: 'null' x-parser-schema-id: x-parser-schema-id: deepfake_score: description: >- Synthetic voice detection score for the clip, or null when no reading is available. oneOf: - type: number format: double minimum: 0 maximum: 1 x-parser-schema-id: - type: 'null' x-parser-schema-id: x-parser-schema-id: x-parser-schema-id: Clip x-parser-schema-id: title: Clip description: A transcribed clip with metadata. example: |- { "type": "", "clip": { "clip_uuid": "", "text": "", "start_ms": 123, "duration_ms": 123, "speaker_label": "", "language": "", "emotion": "", "accent": "", "deepfake_score": 123 } } bindings: [] extensions: - id: x-parser-unique-object-id value: clipEvent - &ref_10 id: partialClipEvent contentType: application/json payload: - name: Partial Clip description: > An in-progress clip streamed while an utterance is still being spoken, before it finalizes. Multiple partials may be emitted for the same `clip_uuid` as the utterance grows; the eventual `clip` event for that utterance reuses the same `clip_uuid`, so a run of partials can be correlated with its final clip. type: object properties: - name: type type: string description: partial_clip required: true - name: partial_clip type: object required: true properties: - name: clip_uuid type: string description: >- Stable identifier shared across a clip's lifetime. The eventual finalized `clip` event for the same utterance reuses this value. required: true - name: text type: string required: true - name: start_ms type: integer required: true - name: end_ms type: integer description: >- Current end of the in-progress utterance; grows as the utterance extends. Null until available. The finalized clip reports `duration_ms` instead of `end_ms`. required: false - name: speaker_label type: string description: Null until diarization resolves the speaker. required: false - name: emotion type: string description: >- Latest in-progress emotion label, or null when no value is available yet. required: false - name: accent type: string description: >- Latest in-progress accent label, or null when no value is available yet. required: false - name: deepfake_score type: number description: >- Latest in-progress deepfake score, or null when no value is available yet. required: false headers: [] jsonPayloadSchema: type: object required: - type - partial_clip properties: type: type: string const: partial_clip x-parser-schema-id: partial_clip: type: object required: - clip_uuid - text - start_ms properties: clip_uuid: type: string format: uuid description: >- Stable identifier shared across a clip's lifetime. The eventual finalized `clip` event for the same utterance reuses this value. x-parser-schema-id: text: type: string x-parser-schema-id: start_ms: type: integer minimum: 0 x-parser-schema-id: end_ms: description: >- Current end of the in-progress utterance; grows as the utterance extends. Null until available. The finalized clip reports `duration_ms` instead of `end_ms`. oneOf: - type: integer minimum: 0 x-parser-schema-id: - type: 'null' x-parser-schema-id: x-parser-schema-id: speaker_label: description: Null until diarization resolves the speaker. oneOf: - type: string x-parser-schema-id: - type: 'null' x-parser-schema-id: x-parser-schema-id: emotion: description: >- Latest in-progress emotion label, or null when no value is available yet. oneOf: - type: string x-parser-schema-id: - type: 'null' x-parser-schema-id: x-parser-schema-id: accent: description: >- Latest in-progress accent label, or null when no value is available yet. oneOf: - type: string x-parser-schema-id: - type: 'null' x-parser-schema-id: x-parser-schema-id: deepfake_score: description: >- Latest in-progress deepfake score, or null when no value is available yet. oneOf: - type: number format: double minimum: 0 maximum: 1 x-parser-schema-id: - type: 'null' x-parser-schema-id: x-parser-schema-id: x-parser-schema-id: PartialClip x-parser-schema-id: title: Partial Clip description: | An in-progress clip streamed while an utterance is still being spoken, before it finalizes. Multiple partials may be emitted for the same `clip_uuid` as the utterance grows; the eventual `clip` event for that utterance reuses the same `clip_uuid`, so a run of partials can be correlated with its final clip. example: |- { "type": "", "partial_clip": { "clip_uuid": "", "text": "", "start_ms": 123, "end_ms": 123, "speaker_label": "", "emotion": "", "accent": "", "deepfake_score": 123 } } bindings: [] extensions: - id: x-parser-unique-object-id value: partialClipEvent - &ref_11 id: clipUpdateEvent contentType: application/json payload: - name: Clip Update description: > Refined values for a previously finalized clip; `clip_update.clip_uuid` matches the `clip_uuid` of an earlier `clip` event. A finalized clip may receive any number of `clip_update` events (including none), each emitted after that clip's `clip` event (possibly interleaved with events for other clips) and always before the `done` event on clean completion. For each field present, the latest received value supersedes the value on the `clip` event and on any earlier `clip_update` for that clip. If refinement is unavailable for a clip, its previously available values are retained. type: object properties: - name: type type: string description: clip_update required: true - name: clip_update type: object required: true properties: - name: clip_uuid type: string description: >- Matches the `clip_uuid` of a previously emitted `clip` event. required: true - name: emotion type: string description: >- Refined emotion label for the clip, or null when no emotion label is available. required: false - name: accent type: string description: >- Refined accent label for the clip, or null when no accent label is available. required: false headers: [] jsonPayloadSchema: type: object required: - type - clip_update properties: type: type: string const: clip_update x-parser-schema-id: clip_update: type: object required: - clip_uuid properties: clip_uuid: type: string format: uuid description: >- Matches the `clip_uuid` of a previously emitted `clip` event. x-parser-schema-id: emotion: description: >- Refined emotion label for the clip, or null when no emotion label is available. oneOf: - type: string x-parser-schema-id: - type: 'null' x-parser-schema-id: x-parser-schema-id: accent: description: >- Refined accent label for the clip, or null when no accent label is available. oneOf: - type: string x-parser-schema-id: - type: 'null' x-parser-schema-id: x-parser-schema-id: x-parser-schema-id: ClipUpdate x-parser-schema-id: title: Clip Update description: | Refined values for a previously finalized clip; `clip_update.clip_uuid` matches the `clip_uuid` of an earlier `clip` event. A finalized clip may receive any number of `clip_update` events (including none), each emitted after that clip's `clip` event (possibly interleaved with events for other clips) and always before the `done` event on clean completion. For each field present, the latest received value supersedes the value on the `clip` event and on any earlier `clip_update` for that clip. If refinement is unavailable for a clip, its previously available values are retained. example: |- { "type": "", "clip_update": { "clip_uuid": "", "emotion": "", "accent": "" } } bindings: [] extensions: - id: x-parser-unique-object-id value: clipUpdateEvent - &ref_12 id: conversationTypeEvent contentType: application/json payload: - name: Conversation Type Pick description: > Inferred or default conversation-type pick for the session. May be updated as the conversation progresses; the latest pick replaces the previous one. When `conversation_type_uuid` changes, or the first type arrives after role events, discard all previous participant-role picks. Subsequent `participant_role` events establish the current assignments; a speaker may remain unassigned if no applicable role is available. Ending the audio does not force a final type or role reassessment; already pending updates may still arrive before `done`. type: object properties: - name: type type: string description: conversation_type required: true - name: pick type: object required: true properties: - name: conversation_type_uuid type: string required: true - name: name type: string required: true - name: confidence type: number required: true - name: selection_source type: string enumValues: - inferred - auto_selected_single_option - default required: true - name: detail type: string required: true - name: reasoning type: string required: false headers: [] jsonPayloadSchema: type: object required: - type - pick properties: type: type: string const: conversation_type x-parser-schema-id: pick: type: object required: - conversation_type_uuid - name - confidence - selection_source - detail properties: conversation_type_uuid: type: string format: uuid x-parser-schema-id: name: type: string x-parser-schema-id: confidence: type: number format: double minimum: 0 maximum: 1 x-parser-schema-id: selection_source: &ref_4 type: string enum: - inferred - auto_selected_single_option - default x-parser-schema-id: SelectionSource detail: type: string x-parser-schema-id: reasoning: oneOf: - type: string x-parser-schema-id: - type: 'null' x-parser-schema-id: x-parser-schema-id: x-parser-schema-id: ConversationTypePick x-parser-schema-id: title: Conversation Type Pick description: > Inferred or default conversation-type pick for the session. May be updated as the conversation progresses; the latest pick replaces the previous one. When `conversation_type_uuid` changes, or the first type arrives after role events, discard all previous participant-role picks. Subsequent `participant_role` events establish the current assignments; a speaker may remain unassigned if no applicable role is available. Ending the audio does not force a final type or role reassessment; already pending updates may still arrive before `done`. example: |- { "type": "", "pick": { "conversation_type_uuid": "", "name": "", "confidence": 123, "selection_source": "", "detail": "", "reasoning": "" } } bindings: [] extensions: - id: x-parser-unique-object-id value: conversationTypeEvent - &ref_13 id: participantRoleEvent contentType: application/json payload: - name: Participant Role Pick description: > Inferred or default participant-role pick for a speaker. A later event replaces the earlier pick for the same `speaker_label`. Role inference is reconsidered only after additional finalized speech from that speaker. Following a conversation-type change, a compatible earlier role may be carried forward in a new event without reassessment. type: object properties: - name: type type: string description: participant_role required: true - name: pick type: object required: true properties: - name: speaker_label type: string required: true - name: participant_role_uuid type: string required: true - name: name type: string required: true - name: confidence type: number required: true - name: selection_source type: string enumValues: - inferred - auto_selected_single_option - default required: true - name: detail type: string required: true - name: reasoning type: string required: false headers: [] jsonPayloadSchema: type: object required: - type - pick properties: type: type: string const: participant_role x-parser-schema-id: pick: type: object required: - speaker_label - participant_role_uuid - name - confidence - selection_source - detail properties: speaker_label: type: string x-parser-schema-id: participant_role_uuid: type: string format: uuid x-parser-schema-id: name: type: string x-parser-schema-id: confidence: type: number format: double minimum: 0 maximum: 1 x-parser-schema-id: selection_source: *ref_4 detail: type: string x-parser-schema-id: reasoning: oneOf: - type: string x-parser-schema-id: - type: 'null' x-parser-schema-id: x-parser-schema-id: x-parser-schema-id: ParticipantRolePick x-parser-schema-id: title: Participant Role Pick description: | Inferred or default participant-role pick for a speaker. A later event replaces the earlier pick for the same `speaker_label`. Role inference is reconsidered only after additional finalized speech from that speaker. Following a conversation-type change, a compatible earlier role may be carried forward in a new event without reassessment. example: |- { "type": "", "pick": { "speaker_label": "", "participant_role_uuid": "", "name": "", "confidence": 123, "selection_source": "", "detail": "", "reasoning": "" } } bindings: [] extensions: - id: x-parser-unique-object-id value: participantRoleEvent - &ref_14 id: behaviorDetectionEvent contentType: application/json payload: - name: Behavior Detection description: > Positive detection for a behavior and speaker. A behavior is emitted at most once per speaker. Detections may arrive after end of audio; continue reading until `done` to receive completed analysis. Emitted detections remain historical when conversation types or roles change. Updated classifications govern subsequent checks; previous detections are not withdrawn or emitted again. Detections may arrive while an utterance is still in progress, after its `partial_clip` event, and before type or role picks are available. type: object properties: - name: type type: string description: behavior_detection required: true - name: detection type: object required: true properties: - name: behavior_uuid type: string required: true - name: behavior_name type: string required: true - name: speaker_label type: string required: true - name: detected type: boolean required: true - name: confidence type: number required: false - name: evidence_clip_uuids type: array description: >- Clip UUIDs supporting the detection, drawn from this speaker's clips. A reference may identify an in-progress clip previously received in a `partial_clip` event; its eventual final clip uses the same UUID. required: false properties: - name: item type: string required: false - name: definitive_clip_uuid type: string description: >- UUID of a representative clip for this speaker, or null when unavailable. required: false - name: reasoning type: string required: false headers: [] jsonPayloadSchema: type: object required: - type - detection properties: type: type: string const: behavior_detection x-parser-schema-id: detection: type: object required: - behavior_uuid - behavior_name - speaker_label - detected properties: behavior_uuid: type: string format: uuid x-parser-schema-id: behavior_name: type: string x-parser-schema-id: speaker_label: oneOf: - type: string x-parser-schema-id: - type: 'null' x-parser-schema-id: x-parser-schema-id: detected: type: boolean x-parser-schema-id: confidence: oneOf: - type: number format: double minimum: 0 maximum: 1 x-parser-schema-id: - type: 'null' x-parser-schema-id: x-parser-schema-id: evidence_clip_uuids: description: >- Clip UUIDs supporting the detection, drawn from this speaker's clips. A reference may identify an in-progress clip previously received in a `partial_clip` event; its eventual final clip uses the same UUID. type: array items: type: string format: uuid x-parser-schema-id: x-parser-schema-id: definitive_clip_uuid: description: >- UUID of a representative clip for this speaker, or null when unavailable. oneOf: - type: string format: uuid x-parser-schema-id: - type: 'null' x-parser-schema-id: x-parser-schema-id: reasoning: oneOf: - type: string x-parser-schema-id: - type: 'null' x-parser-schema-id: x-parser-schema-id: x-parser-schema-id: BehaviorDetection x-parser-schema-id: title: Behavior Detection description: | Positive detection for a behavior and speaker. A behavior is emitted at most once per speaker. Detections may arrive after end of audio; continue reading until `done` to receive completed analysis. Emitted detections remain historical when conversation types or roles change. Updated classifications govern subsequent checks; previous detections are not withdrawn or emitted again. Detections may arrive while an utterance is still in progress, after its `partial_clip` event, and before type or role picks are available. example: No examples found bindings: [] extensions: - id: x-parser-unique-object-id value: behaviorDetectionEvent - &ref_15 id: topicsEvent contentType: application/json payload: - name: Topics description: > Aggregated list of conversation topics. May be emitted more than once as the stream progresses; each event fully replaces the previous `topics` event, so always treat the latest as authoritative and never merge with earlier ones. type: object properties: - name: type type: string description: topics required: true - name: topics type: array required: true properties: - name: item type: string required: false headers: [] jsonPayloadSchema: type: object required: - type - topics properties: type: type: string const: topics x-parser-schema-id: topics: type: array items: type: string x-parser-schema-id: x-parser-schema-id: x-parser-schema-id: title: Topics description: > Aggregated list of conversation topics. May be emitted more than once as the stream progresses; each event fully replaces the previous `topics` event, so always treat the latest as authoritative and never merge with earlier ones. example: |- { "type": "", "topics": { "item": "" } } bindings: [] extensions: - id: x-parser-unique-object-id value: topicsEvent - &ref_16 id: topicSentimentEvent contentType: application/json payload: - name: Topic Sentiment description: > Per-speaker sentiment for an aggregated topic, keyed by topic and speaker. May be emitted more than once as the stream progresses; a later event supersedes an earlier one for the same topic and speaker. type: object properties: - name: type type: string description: topic_sentiment required: true - name: topic_sentiment type: object required: true properties: - name: topic type: string required: true - name: speaker_label type: string required: true - name: sentiment_score type: number required: true - name: sentiment_label type: string required: true headers: [] jsonPayloadSchema: type: object required: - type - topic_sentiment properties: type: type: string const: topic_sentiment x-parser-schema-id: topic_sentiment: type: object required: - topic - speaker_label - sentiment_score - sentiment_label properties: topic: type: string x-parser-schema-id: speaker_label: type: string x-parser-schema-id: sentiment_score: type: number format: double minimum: -1 maximum: 1 x-parser-schema-id: sentiment_label: type: string x-parser-schema-id: x-parser-schema-id: TopicSentiment x-parser-schema-id: title: Topic Sentiment description: > Per-speaker sentiment for an aggregated topic, keyed by topic and speaker. May be emitted more than once as the stream progresses; a later event supersedes an earlier one for the same topic and speaker. example: |- { "type": "", "topic_sentiment": { "topic": "", "speaker_label": "", "sentiment_score": 123, "sentiment_label": "" } } bindings: [] extensions: - id: x-parser-unique-object-id value: topicSentimentEvent - &ref_17 id: summaryEvent contentType: application/json payload: - name: Summary description: > Free-form summary of the conversation. May be emitted more than once as the stream progresses; each event fully replaces the previous `summary` event, so always treat the latest as authoritative. type: object properties: - name: type type: string description: summary required: true - name: text type: string required: true headers: [] jsonPayloadSchema: type: object required: - type - text properties: type: type: string const: summary x-parser-schema-id: text: type: string x-parser-schema-id: x-parser-schema-id: title: Summary description: > Free-form summary of the conversation. May be emitted more than once as the stream progresses; each event fully replaces the previous `summary` event, so always treat the latest as authoritative. example: |- { "type": "", "text": "" } bindings: [] extensions: - id: x-parser-unique-object-id value: summaryEvent - &ref_18 id: errorEvent contentType: application/json payload: - name: Error description: An error occurred during processing. type: object properties: - name: type type: string description: error required: true - name: error type: string description: Human-readable error message required: true headers: [] jsonPayloadSchema: type: object required: - type - error properties: type: type: string const: error x-parser-schema-id: error: type: string description: Human-readable error message x-parser-schema-id: x-parser-schema-id: title: Error description: An error occurred during processing. example: |- { "type": "error", "error": "Internal server error" } bindings: [] extensions: - id: x-parser-unique-object-id value: errorEvent - &ref_19 id: doneEvent contentType: application/json payload: - name: Done description: Streaming completed. type: object properties: - name: type type: string description: done required: true - name: duration_ms type: integer required: true headers: [] jsonPayloadSchema: type: object required: - type - duration_ms properties: type: type: string const: done x-parser-schema-id: duration_ms: type: integer minimum: 0 x-parser-schema-id: x-parser-schema-id: title: Done description: Streaming completed. example: |- { "type": "done", "duration_ms": 45000 } bindings: [] extensions: - id: x-parser-unique-object-id value: doneEvent bindings: [] extensions: *ref_3 sendOperations: - *ref_5 receiveOperations: - *ref_6 - *ref_7 - *ref_8 sendMessages: - *ref_9 - *ref_10 - *ref_11 - *ref_12 - *ref_13 - *ref_14 - *ref_15 - *ref_16 - *ref_17 - *ref_18 - *ref_19 receiveMessages: - *ref_20 - *ref_21 - *ref_22 extensions: - id: x-parser-unique-object-id value: velma2Streaming securitySchemes: [] ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.