# Accent Detection Batch
Source: https://docs.modulate.ai/api-reference/accent/batch

api/velma_2_accent_batch.yaml POST /api/velma-2-accent-batch
Classify the speaker accent of an audio file. Returns a whole-file accent label plus a time series of fixed-length windows, each with its own label, in a single synchronous response.



# Accent Detection
Source: https://docs.modulate.ai/api-reference/accent/overview

Accent detection API — classify the speaker accent of an audio file, with a whole-file label and a per-window time series, in a single synchronous HTTP POST.

Accent Detection classifies the accent of the speech in an audio file from the voice signal. It returns a single whole-file `accent` label plus a time series of consecutive fixed-length windows, each with its own label. It is a pure classification endpoint — no transcript, diarization, or enrichment data is produced.

|                   | Batch                                                         |
| ----------------- | ------------------------------------------------------------- |
| **Use case**      | Classify the speaker accent of an audio file                  |
| **Protocol**      | HTTP POST                                                     |
| **Max file size** | 100 MB                                                        |
| **Output**        | Whole-file `accent` label + per-window time series            |
| **Options**       | `use_ensemble` for a more thorough analysis at higher latency |

Each `time_series` entry covers one fixed-length window, delimited by its `start_ms` and `duration_ms`; a trailing remainder shorter than one full window is omitted. A file shorter than one window returns an empty `time_series` — the whole-file `accent` label is always present.

<Note>
  If you need accent labels alongside a transcript, use the `accent_signal` enrichment on Multilingual Transcription instead — see [Accent Detection](/get-started/accent).
</Note>

For a side-by-side comparison with the other Modulate capabilities, see [Which API should I use?](/guides/which-api).

## Authentication

Uses the `X-API-Key` header. See [Authentication and rate limits](/guides/authentication).


# AI Music Detection Batch
Source: https://docs.modulate.ai/api-reference/ai-music-detection/batch

api/velma_2_ai_music_detection_batch.yaml POST /api/velma-2-ai-music-detection-batch
Detect AI-generated music in an audio file. Returns a clip-level verdict plus a per-window breakdown of vocal and instrumental AI content.



# AI Music Detection
Source: https://docs.modulate.ai/api-reference/ai-music-detection/overview

AI music detection APIs - detect AI-generated music with clip-level verdicts and per-window vocal/instrumental breakdowns, batch and real-time streaming.

AI music detection determines whether a clip contains AI-generated music. Each window is independently scored for AI-generated vocals and AI-generated instrumental content, whenever it has enough of the corresponding content type to score - a window can carry both scores, one, or neither. Window results are aggregated into a clip-level `primary_verdict` of `ai-vocal-music`, `ai-instrumental`, or `not-ai-music`.

<Note>
  This is distinct from [music detection](/api-reference/music-detection/overview), which classifies audio as music, speech, or neither. AI music detection answers a different question: *is this music AI-generated?*
</Note>

|                               | Batch                                        | Streaming                                                                                                            |
| ----------------------------- | -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| **Use case**                  | Classify a complete audio file               | Real-time per-window classification                                                                                  |
| **Protocol**                  | HTTP POST                                    | WebSocket                                                                                                            |
| **Output**                    | Clip-level verdict plus per-window breakdown | Per-window vocal and instrumental AI results emitted progressively, final clip-level summary on completion           |
| **Instrumental AI detection** | Included per window and clip-level           | Included per window (live) and clip-level; the clip-level `done` result uses more audio context and is more reliable |

For a side-by-side comparison with the other Modulate capabilities, see [Which API should I use?](/guides/which-api).

## Authentication

Batch uses the `X-API-Key` header. Streaming uses an `api_key` query parameter at connection time. See [Authentication and rate limits](/guides/authentication).

## Performance notes

* Per-window results can be less accurate than the clip-level verdict and its confidence. Rely on the clip-level result when judging a whole song or segment.
* A window's vocal-AI and instrumental-AI scores are independent - a window can carry both, one, or neither, depending on how much of each content type it contains.
* Heavily processed or high-production tracks are sometimes mislabeled as AI-generated. This is a known gap targeted by future model updates.


# AI Music Detection Streaming
Source: https://docs.modulate.ai/api-reference/ai-music-detection/streaming

Real-time AI music detection over WebSocket. Per-window vocal-AI and instrumental-AI results are emitted progressively as audio arrives, followed by a final clip-level summary.

Real-time AI music detection over WebSocket. The client streams audio and receives per-window vocal-AI and instrumental-AI results as they become available, followed by a final clip-level summary on completion.

## Endpoint

```text theme={null}
wss://platform.modulate.ai/api/velma-2-ai-music-detection-streaming
```

## Authentication

Pass your API key as a query parameter on the connection URL:

```text theme={null}
wss://.../velma-2-ai-music-detection-streaming?api_key=YOUR_API_KEY&audio_format=mp3
```

## Connection parameters

| Parameter      | Required     | Description                                |
| -------------- | ------------ | ------------------------------------------ |
| `api_key`      | Yes          | Your API key                               |
| `audio_format` | Yes          | Audio format - see supported formats below |
| `sample_rate`  | Raw PCM only | Sample rate in Hz                          |
| `num_channels` | Raw PCM only | Number of channels (1-8)                   |

## Supported audio formats

**Container formats** - `sample_rate` and `num_channels` are ignored if supplied (the headers already carry this metadata):

`mp3`, `wav`, `flac`, `m4a`, `mp4`, `ogg`, `opus`, `webm`, `aac`, `aiff`, `wma`, `amr`, and `au`, plus 67 others

<Accordion title="All 80 accepted container values">
  `3g2`, `3ga`, `3gp`, `3gpp`, `8svx`, `aa3`, `aac`, `ac3`, `act`, `adts`, `aif`, `aifc`, `aiff`,
  `amb`, `amr`, `asf`, `at3`, `au`, `avr`, `awb`, `bwf`, `c2`, `caf`, `dss`, `dts`, `dtshd`,
  `eac3`, `ec3`, `f4a`, `f4b`, `flac`, `gsm`, `iff`, `m2a`, `m2ts`, `m4a`, `m4b`, `m4r`, `m4v`,
  `mka`, `mkv`, `mlp`, `mp+`, `mp1`, `mp2`, `mp3`, `mp4`, `mpa`, `mpc`, `mpga`, `mpp`, `mts`,
  `oga`, `ogg`, `ogx`, `oma`, `omg`, `opus`, `paf`, `pvf`, `qcp`, `ra`, `rf64`, `rm`, `rmvb`,
  `snd`, `spx`, `svx`, `thd`, `ts`, `tta`, `voc`, `vqf`, `w64`, `wav`, `wave`, `weba`, `webm`,
  `wma`, `wmv`
</Accordion>

The MP4-family values (`mp4`, `m4a`, `m4b`, `m4r`, `m4v`, `3gp`, `3gpp`, `3ga`, `3g2`, `f4a`, `f4b`) must be sent in a streamable layout; otherwise the connection ends with an audio-processing error.

**Raw PCM formats** - `sample_rate` and `num_channels` are required:

`s8`, `s16le`, `s16be`, `s24le`, `s24be`, `s32le`, `s32be`, `u8`, `u16le`, `u16be`, `u24le`, `u24be`, `u32le`, `u32be`, `f32le`, `f32be`, `f64le`, `f64be`, `mulaw`, `alaw`, `g722`, `vox`

`g722` and `vox` are mono-only: `num_channels` must be `1`.

**Valid sample rates:** 8000, 11025, 16000, 22050, 32000, 44100, 48000, 96000

## Protocol

### Client -> server

| Message               | Description                                            |
| --------------------- | ------------------------------------------------------ |
| Binary frame          | Chunk of audio bytes in the declared format (any size) |
| Empty text frame `""` | Signals end of stream                                  |

### Server -> client

| Message                               | Description                                                              |
| ------------------------------------- | ------------------------------------------------------------------------ |
| `{"type": "window", "window": {...}}` | Per-window result - emitted for each completed 4-second window, in order |
| `{"type": "done", ...}`               | Stream complete - clip-level verdict plus instrumental AI detection      |
| `{"type": "error", "error": "..."}`   | An error occurred - connection will close                                |

<Note>
  Vocal AI and instrumental AI are both scored independently on each 4-second window as audio arrives and reported in `window` messages - a window can carry both scores, one, or neither, depending on how much of each content type it contains. The window-level instrumental score is computed from a single short window, so it can be less accurate than the clip-level score in the final `done` message, which is computed from the full accumulated audio.
</Note>

### Window object

| Field                         | Type            | Description                                                                                                                                                                                                                                                                                       |
| ----------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `start_time_ms`               | integer         | Window start time in milliseconds                                                                                                                                                                                                                                                                 |
| `end_time_ms`                 | integer         | Window end time in milliseconds                                                                                                                                                                                                                                                                   |
| `vocal_percentage`            | float           | Percentage of the window containing vocal content (0-100)                                                                                                                                                                                                                                         |
| `vocal_ai_probability`        | float or `null` | Probability the window contains AI-generated vocals (0-1). `null` if the window doesn't have enough vocal content to score, was too short to analyse, or no content was found to classify                                                                                                         |
| `vocal_ai_confidence`         | float or `null` | Confidence in `vocal_ai_probability` (0-1); `null` under the same conditions as that field                                                                                                                                                                                                        |
| `instrumental_percentage`     | float           | Percentage of the window containing instrumental music content (0-100)                                                                                                                                                                                                                            |
| `instrumental_ai_probability` | float or `null` | Probability the window contains AI-generated instrumental content (0-1), from a single-window analysis (lower accuracy than the `done` message's clip-level score). `null` if the window is mostly silence, instrumental-AI detection is unavailable, or the window's audio could not be analysed |
| `instrumental_ai_confidence`  | float or `null` | Confidence in `instrumental_ai_probability` (0-1); `null` under the same conditions as that field                                                                                                                                                                                                 |
| `silence_percentage`          | float           | Percentage of the window containing neither vocal nor instrumental content (0-100)                                                                                                                                                                                                                |

`vocal_ai_probability`/`vocal_ai_confidence` and `instrumental_ai_probability`/`instrumental_ai_confidence` are gated and scored independently, so a window can carry both pairs, one, or neither.

### Done object

| Field                        | Type    | Description                                                                                                                                                                                                  |
| ---------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `duration_ms`                | integer | Total duration of the streamed audio in milliseconds                                                                                                                                                         |
| `window_count`               | integer | Total number of windows analysed during the session                                                                                                                                                          |
| `primary_verdict`            | string  | Clip-level classification: `"ai-vocal-music"`, `"ai-instrumental"`, or `"not-ai-music"`                                                                                                                      |
| `vocal_percentage`           | float   | Average percentage of audio with vocal content, across all windows (0-100)                                                                                                                                   |
| `vocal_ai_percentage`        | float   | Percentage of the full clip duration classified as AI-generated vocals (0-100), recomputed at end-of-stream. May differ from what the live per-window messages suggested                                     |
| `vocal_ai_confidence`        | float   | Average confidence across frames classified as AI-generated vocals, from the end-of-stream recompute (0-1); `0` if none were found                                                                           |
| `instrumental_percentage`    | float   | Average percentage of audio with instrumental content, across all windows (0-100)                                                                                                                            |
| `instrumental_ai_percentage` | float   | AI detection score for the clip's instrumental content (0-100), computed once at end-of-stream from the full accumulated audio - more reliable than the live per-window `instrumental_ai_probability` values |
| `instrumental_ai_confidence` | float   | Confidence in the instrumental AI assessment for the full clip (0-1)                                                                                                                                         |
| `silence_percentage`         | float   | Average percentage of audio with neither vocal nor instrumental content (0-100)                                                                                                                              |

## WebSocket close codes

| Code   | Meaning                                                                                       |
| ------ | --------------------------------------------------------------------------------------------- |
| `1000` | Normal closure after the `done` message, or after the connection completes                    |
| `1003` | Invalid or missing query parameters (unknown format, bad sample rate, missing `audio_format`) |
| `1011` | Internal server error, or processing ended before completion                                  |
| `4001` | The `api_key` query parameter is missing or invalid                                           |
| `4002` | Audio could not be decoded, or does not match the declared raw `audio_format`                 |
| `4003` | The request is not permitted                                                                  |
| `4004` | The API key does not have access to this model                                                |
| `4029` | The request could not be completed due to insufficient credits                                |
| `4030` | Concurrent request limit reached                                                              |
| `4031` | Monthly usage limit reached                                                                   |

An `error` message is sent before the connection closes for every case above except `1000`.

## Rate limits

* Concurrent connection limits apply per model


# Audio Event Detection Batch
Source: https://docs.modulate.ai/api-reference/audio-event-detection/batch

api/velma_2_audio_event_classifier.yaml POST /api/velma-2-audio-event-classifier
Detect non-speech sound events in an audio file. Returns a probability for every supported event, plus the audio duration, in a single synchronous response.



# Audio Event Detection
Source: https://docs.modulate.ai/api-reference/audio-event-detection/overview

Audio event detection API — score an audio file against a fixed set of non-speech sound events in a single synchronous HTTP POST.

Audio Event Detection scores an audio file against a fixed set of non-speech sound events: instruments, human vocalizations such as laughter and coughing, and environmental noises such as a knock or a gunshot. It returns a probability for every supported event on every call, plus the duration of the processed audio. It is a pure classification endpoint, producing no transcript, diarization, or PII/PHI tagging.

|                   | Batch                                                              |
| ----------------- | ------------------------------------------------------------------ |
| **Use case**      | Score an audio file against a fixed set of non-speech sound events |
| **Protocol**      | HTTP POST                                                          |
| **Max file size** | 100 MB                                                             |
| **Output**        | A probability per event, plus `duration_ms`                        |
| **Options**       | None. Only `upload_file` is accepted                               |

`probs` carries 42 keys on every response. `cry` is an independent probability. The other 41 are drawn from a single shared distribution that sums to 1 across those keys, so they rank the most prominent event rather than reporting independent detections. See [Audio Event Detection](/get-started/audio-event-detection) for how to read the two kinds of value.

For a side-by-side comparison with the other Modulate capabilities, see [Which API should I use?](/guides/which-api).

## Authentication

Uses the `X-API-Key` header. See [Authentication and rate limits](/guides/authentication).


# Emotion Detection Batch
Source: https://docs.modulate.ai/api-reference/emotion/batch

api/velma_2_emotion_batch.yaml POST /api/velma-2-emotion-batch
Classify the emotional tone of an audio file. Returns a whole-file emotion label plus a time series of fixed-length windows, each with its own label, in a single synchronous response.



# Emotion Detection
Source: https://docs.modulate.ai/api-reference/emotion/overview

Emotion detection API — classify the emotional tone of an audio file, with a whole-file label and a per-window time series, in a single synchronous HTTP POST.

Emotion Detection classifies the emotional tone of an audio file from the voice signal. It returns a single whole-file `emotion` label plus a time series of consecutive fixed-length windows, each with its own label. It is a pure classification endpoint — no transcript, diarization, or enrichment data is produced.

|                   | Batch                                                         |
| ----------------- | ------------------------------------------------------------- |
| **Use case**      | Classify the emotional tone of an audio file                  |
| **Protocol**      | HTTP POST                                                     |
| **Max file size** | 100 MB                                                        |
| **Output**        | Whole-file `emotion` label + per-window time series           |
| **Options**       | `use_ensemble` for a more thorough analysis at higher latency |

Each `time_series` entry covers one fixed-length window, delimited by its `start_ms` and `duration_ms`; a trailing remainder shorter than one full window is omitted. A file shorter than one window returns an empty `time_series` — the whole-file `emotion` label is always present.

<Note>
  If you need emotion labels alongside a transcript, use the `emotion_signal` enrichment on Multilingual Transcription instead — see [Emotion Detection](/get-started/emotion).
</Note>

For a side-by-side comparison with the other Modulate capabilities, see [Which API should I use?](/guides/which-api).

## Authentication

Uses the `X-API-Key` header. See [Authentication and rate limits](/guides/authentication).


# Language Detection Batch
Source: https://docs.modulate.ai/api-reference/language-detection/batch

api/velma_2_language_detection_batch.yaml POST /api/velma-2-language-detection-batch
Identify the spoken language of an audio file. Returns an ISO 639-1 language code, human-readable display name, and confidence score in a single synchronous response.



# Language Detection
Source: https://docs.modulate.ai/api-reference/language-detection/overview

Language detection API — identify the spoken language of an audio file in a single synchronous HTTP POST.

Language detection identifies the spoken language of an audio file and returns a confidence-scored result. It is a pure classification endpoint — no transcription, diarization, or enrichment data is produced.

|                    | Batch                                          |
| ------------------ | ---------------------------------------------- |
| **Use case**       | Identify the spoken language of an audio file  |
| **Protocol**       | HTTP POST                                      |
| **Languages**      | 100 spoken languages                           |
| **Max file size**  | 100 MB                                         |
| **Audio analyzed** | First 30 seconds only                          |
| **Output**         | ISO 639-1 code, display name, confidence score |

For a side-by-side comparison with the other Modulate capabilities, see [Which API should I use?](/guides/which-api).

## Authentication

Uses the `X-API-Key` header. See [Authentication and rate limits](/guides/authentication).


# Music Detection Batch
Source: https://docs.modulate.ai/api-reference/music-detection/batch

api/velma_2_music_detection_batch.yaml POST /api/velma-2-music-detection-batch
Classify music and speech in an audio file. Returns frame-level probabilities, a primary label, and percentage breakdowns of content type.



# Music Detection
Source: https://docs.modulate.ai/api-reference/music-detection/overview

Music detection APIs — batch classification and real-time streaming over WebSocket.

Music detection classifies audio as music, speech, or neither, returning frame-level probabilities across the clip.

|              | Batch                          | Streaming                                     |
| ------------ | ------------------------------ | --------------------------------------------- |
| **Use case** | Classify a complete audio file | Real-time frame-by-frame classification       |
| **Protocol** | HTTP POST                      | WebSocket                                     |
| **Output**   | Full response after processing | Frames emitted progressively as audio arrives |
| **Latency**  | Proportional to file length    | \~192ms per frame                             |

For a side-by-side comparison with the other Modulate capabilities, see [Which API should I use?](/guides/which-api).

## Authentication

Batch uses the `X-API-Key` header. Streaming uses an `api_key` query parameter at connection time. See [Authentication and rate limits](/guides/authentication).


# Music Detection Streaming
Source: https://docs.modulate.ai/api-reference/music-detection/streaming

Real-time frame-level music and speech classification over WebSocket. Frames are emitted progressively as audio is streamed.

Real-time frame-level music and speech classification over WebSocket. Frames are returned progressively as audio is streamed — no need to wait for the full file to upload before results begin arriving.

## Endpoint

```text theme={null}
wss://platform.modulate.ai/api/velma-2-music-detection-streaming
```

## Authentication

Pass your API key as a query parameter on the connection URL:

```text theme={null}
wss://.../velma-2-music-detection-streaming?api_key=YOUR_API_KEY&audio_format=s16le&...
```

## Features

* **Real-time output** — frames emitted progressively after each 192ms chunk of audio
* **Music detection** — identifies frames containing music content
* **Speech detection** — identifies frames containing speech content
* **Non-exclusive labels** — music and speech are independent; both can be high simultaneously (e.g. music with vocals)
* **Any chunk size** — send audio in whatever chunk size suits your pipeline
* **Container and raw PCM support** — stream compressed files or raw PCM directly from a microphone

## Connection parameters

| Parameter      | Required     | Description                                |
| -------------- | ------------ | ------------------------------------------ |
| `api_key`      | Yes          | Your API key                               |
| `audio_format` | Yes          | Audio format — see supported formats below |
| `sample_rate`  | Raw PCM only | Sample rate in Hz                          |
| `num_channels` | Raw PCM only | Number of channels (1–8)                   |

## Supported audio formats

**Container formats** — `sample_rate` and `num_channels` are ignored if supplied (the headers already carry this metadata):

`mp3`, `wav`, `flac`, `m4a`, `mp4`, `ogg`, `opus`, `webm`, `aac`, `aiff`, `wma`, `amr`, and `au`, plus 67 others

<Accordion title="All 80 accepted container values">
  `3g2`, `3ga`, `3gp`, `3gpp`, `8svx`, `aa3`, `aac`, `ac3`, `act`, `adts`, `aif`, `aifc`, `aiff`,
  `amb`, `amr`, `asf`, `at3`, `au`, `avr`, `awb`, `bwf`, `c2`, `caf`, `dss`, `dts`, `dtshd`,
  `eac3`, `ec3`, `f4a`, `f4b`, `flac`, `gsm`, `iff`, `m2a`, `m2ts`, `m4a`, `m4b`, `m4r`, `m4v`,
  `mka`, `mkv`, `mlp`, `mp+`, `mp1`, `mp2`, `mp3`, `mp4`, `mpa`, `mpc`, `mpga`, `mpp`, `mts`,
  `oga`, `ogg`, `ogx`, `oma`, `omg`, `opus`, `paf`, `pvf`, `qcp`, `ra`, `rf64`, `rm`, `rmvb`,
  `snd`, `spx`, `svx`, `thd`, `ts`, `tta`, `voc`, `vqf`, `w64`, `wav`, `wave`, `weba`, `webm`,
  `wma`, `wmv`
</Accordion>

The MP4-family values (`mp4`, `m4a`, `m4b`, `m4r`, `m4v`, `3gp`, `3gpp`, `3ga`, `3g2`, `f4a`, `f4b`) must be sent in a streamable layout; otherwise the connection ends with an audio-processing error.

**Raw PCM formats** — `sample_rate` and `num_channels` are required:

`s8`, `s16le`, `s16be`, `s24le`, `s24be`, `s32le`, `s32be`, `u8`, `u16le`, `u16be`, `u24le`, `u24be`, `u32le`, `u32be`, `f32le`, `f32be`, `f64le`, `f64be`, `mulaw`, `alaw`, `g722`, `vox`

`g722` and `vox` are mono-only: `num_channels` must be `1`.

**Valid sample rates:** 8000, 11025, 16000, 22050, 32000, 44100, 48000, 96000

## Protocol

### Client → server

| Message               | Description                                            |
| --------------------- | ------------------------------------------------------ |
| Binary frame          | Chunk of audio bytes in the declared format (any size) |
| Empty text frame `""` | Signals end of stream                                  |

### Server → client

| Message                                     | Description                                                  |
| ------------------------------------------- | ------------------------------------------------------------ |
| `{"type": "frame", "frame": {...}}`         | Frame result — emitted after each 192ms chunk                |
| `{"type": "done", "duration_ms": ..., ...}` | Stream complete — includes overall summary across all frames |
| `{"type": "error", "error": "..."}`         | An error occurred — connection will close                    |

### Frame object

| Field           | Type    | Description                      |
| --------------- | ------- | -------------------------------- |
| `start_time_ms` | integer | Frame start time in milliseconds |
| `end_time_ms`   | integer | Frame end time in milliseconds   |
| `music_prob`    | float   | Music probability (0.0–1.0)      |
| `speech_prob`   | float   | Speech probability (0.0–1.0)     |

### Done object

| Field           | Type    | Description                                                                                                              |
| --------------- | ------- | ------------------------------------------------------------------------------------------------------------------------ |
| `duration_ms`   | integer | Total audio duration processed in milliseconds                                                                           |
| `frame_count`   | integer | Total number of frames returned                                                                                          |
| `music_pct`     | float   | Percentage of the analysed duration classified as music, rounded to one decimal place. `0.0` when no audio was analysed  |
| `speech_pct`    | float   | Percentage of the analysed duration classified as speech, rounded to one decimal place. `0.0` when no audio was analysed |
| `primary_label` | string  | Dominant classification: `"music"`, `"speech"`, `"neither"`, or `"unknown"` (no frames produced from the audio)          |

## WebSocket close codes

| Code   | Meaning                                                                            |
| ------ | ---------------------------------------------------------------------------------- |
| `1000` | Normal closure after a successful `done` message                                   |
| `1003` | Invalid query parameters (unknown format, bad sample rate, missing `audio_format`) |
| `4002` | Audio could not be decoded or does not match the declared format                   |
| `4003` | Access denied or server-side usage check failed                                    |
| `4029` | Insufficient credits                                                               |

## Chunking behaviour

Audio is buffered in 192ms chunks (one output frame each). Frames are emitted as soon as each chunk is ready, so results begin arriving within 192ms of the first audio being received. At end-of-stream, any remaining audio ≥ 192ms is processed and its frames are emitted before the `done` message.

## Examples

<CodeGroup>
  ```python Python (raw PCM) theme={null}
  import asyncio
  import websockets
  import json

  WS_URL = "wss://platform.modulate.ai/api/velma-2-music-detection-streaming"
  API_KEY = "YOUR_API_KEY"

  async def stream_audio(file_path: str) -> None:
      url = (
          f"{WS_URL}?api_key={API_KEY}"
          f"&audio_format=s16le&sample_rate=16000&num_channels=1"
      )
      async with websockets.connect(url) as ws:
          # Send audio in chunks
          with open(file_path, "rb") as f:
              while chunk := f.read(16000):  # 0.5s of s16le/16kHz mono
                  await ws.send(chunk)

          # Signal end of stream
          await ws.send("")

          # Receive results
          async for message in ws:
              msg = json.loads(message)
              if msg["type"] == "frame":
                  frame = msg["frame"]
                  print(
                      f"{frame['start_time_ms']}ms – {frame['end_time_ms']}ms  "
                      f"music={frame['music_prob']:.4f}  speech={frame['speech_prob']:.4f}"
                  )
              elif msg["type"] == "done":
                  print(f"\nDone — {msg['duration_ms']}ms, {msg['frame_count']} frames")
                  print(f"music={msg['music_pct']}%  speech={msg['speech_pct']}%  label={msg['primary_label']}")
                  break
              elif msg["type"] == "error":
                  raise RuntimeError(f"Server error: {msg['error']}")

  asyncio.run(stream_audio("/path/to/audio.raw"))
  ```

  ```python Python (container format) theme={null}
  import asyncio
  import websockets
  import json

  WS_URL = "wss://platform.modulate.ai/api/velma-2-music-detection-streaming"
  API_KEY = "YOUR_API_KEY"

  async def stream_audio_file(file_path: str, audio_format: str) -> None:
      url = f"{WS_URL}?api_key={API_KEY}&audio_format={audio_format}"
      async with websockets.connect(url) as ws:
          with open(file_path, "rb") as f:
              while chunk := f.read(65536):
                  await ws.send(chunk)
          await ws.send("")

          async for message in ws:
              msg = json.loads(message)
              if msg["type"] == "frame":
                  frame = msg["frame"]
                  print(
                      f"{frame['start_time_ms']}ms – {frame['end_time_ms']}ms  "
                      f"music={frame['music_prob']:.4f}  speech={frame['speech_prob']:.4f}"
                  )
              elif msg["type"] == "done":
                  print(f"\nDone — {msg['duration_ms']}ms, {msg['frame_count']} frames")
                  print(f"music={msg['music_pct']}%  speech={msg['speech_pct']}%  label={msg['primary_label']}")
                  break
              elif msg["type"] == "error":
                  raise RuntimeError(f"Server error: {msg['error']}")

  asyncio.run(stream_audio_file("/path/to/audio.mp3", "mp3"))
  ```

  ```javascript JavaScript (Node.js) theme={null}
  import { WebSocket } from "ws";
  import { createReadStream } from "fs";

  const WS_URL = "wss://platform.modulate.ai/api/velma-2-music-detection-streaming";
  const API_KEY = "YOUR_API_KEY";

  async function streamAudio(filePath, audioFormat) {
    const url = `${WS_URL}?api_key=${API_KEY}&audio_format=${audioFormat}`;
    const ws = new WebSocket(url);

    await new Promise((resolve, reject) => {
      ws.on("open", () => {
        const stream = createReadStream(filePath, { highWaterMark: 65536 });
        stream.on("data", (chunk) => ws.send(chunk));
        stream.on("end", () => ws.send(""));
        stream.on("error", reject);
      });

      ws.on("message", (data) => {
        const msg = JSON.parse(data);
        if (msg.type === "frame") {
          const { start_time_ms, end_time_ms, music_prob, speech_prob } = msg.frame;
          console.log(
            `${start_time_ms}ms – ${end_time_ms}ms  ` +
            `music=${music_prob.toFixed(4)}  speech=${speech_prob.toFixed(4)}`
          );
        } else if (msg.type === "done") {
          console.log(`\nDone — ${msg.duration_ms}ms, ${msg.frame_count} frames`);
          console.log(`music=${msg.music_pct}%  speech=${msg.speech_pct}%  label=${msg.primary_label}`);
          ws.close();
          resolve();
        } else if (msg.type === "error") {
          reject(new Error(`Server error: ${msg.error}`));
        }
      });

      ws.on("error", reject);
    });
  }

  await streamAudio("/path/to/audio.mp3", "mp3");
  ```
</CodeGroup>

## Rate limits

* Concurrent connection limits apply per model


# PII/PHI Redaction Batch
Source: https://docs.modulate.ai/api-reference/redaction/batch

api/velma_2_pii_phi_redaction_batch.yaml POST /api/velma-2-pii-phi-redaction-batch
Transcribe a pre-recorded audio file and redact PII/PHI from both the transcript text and the returned audio.



# PII/PHI Redaction
Source: https://docs.modulate.ai/api-reference/redaction/overview

PII/PHI Redaction transcribes audio, replaces sensitive spans with empty marker tags, and silences the matching audio ranges.

PII/PHI Redaction transcribes audio, replaces detected PII/PHI spans with empty marker tags (e.g. `<pii:name></pii:name>`, `<pii:ssn></pii:ssn>`, `<phi></phi>`) in the transcript, **and** silences the matching ranges in the returned audio.

|                             | Batch                              | Streaming                                    |
| --------------------------- | ---------------------------------- | -------------------------------------------- |
| **Use case**                | Transcription with audio redaction | Real-time transcription with audio redaction |
| **Protocol**                | HTTP POST                          | WebSocket                                    |
| **Languages**               | Multilingual                       | Multilingual                                 |
| **Speaker diarization**     | ✓                                  | ✓                                            |
| **PII/PHI audio redaction** | ✓                                  | ✓                                            |

<Note>
  Need PII/PHI flagged in the transcript but the **audio left intact**? Use the [STT APIs](/api-reference/stt/overview) with PII/PHI tagging instead.
</Note>

For a side-by-side comparison with the other Modulate capabilities, see [Which API should I use?](/guides/which-api).

## Authentication

Batch uses the `X-API-Key` header. Streaming uses an `api_key` query parameter at connection time. See [Authentication and rate limits](/guides/authentication).


# PII/PHI Redaction Streaming
Source: https://docs.modulate.ai/api-reference/redaction/streaming

Real-time PII/PHI redaction over WebSocket — receive a redacted transcript and a redacted MP3 clip per utterance.

Real-time PII/PHI redaction over WebSocket. Streams audio to the server and receives, per utterance, a redacted transcript and a redacted MP3 clip with the PII/PHI ranges silenced.

## Endpoint

```text theme={null}
wss://platform.modulate.ai/api/velma-2-pii-phi-redaction-streaming
```

## Authentication

Pass your API key as a query parameter when opening the connection.

```text theme={null}
wss://platform.modulate.ai/api/velma-2-pii-phi-redaction-streaming?api_key=YOUR_API_KEY
```

<Warning>
  Unlike the batch endpoints, the streaming API does not use an `X-API-Key` header. The key must be in the query string at connection time.
</Warning>

See [Authentication and rate limits](/guides/authentication) for how to obtain and manage API keys.

## Supported audio formats

**Self-describing formats** (auto-detected from file headers — no extra parameters needed):
AAC, AIFF, FLAC, MP3, OGG, WAV, WebM

<Note>
  **OGG / Opus:** OGG is a container that may carry Opus-encoded audio. Pass `audio_format=ogg`, not `audio_format=opus`.
</Note>

**Raw / headerless formats** (require `audio_format`, `sample_rate`, and `num_channels`):
`s8`, `s16le`, `s16be`, `s24le`, `s24be`, `s32le`, `s32be`, `u8`, `u16le`, `u16be`, `u24le`, `u24be`, `u32le`, `u32be`, `f32le`, `f32be`, `f64le`, `f64be`, `mulaw`, `alaw`

Valid sample rates: `8000`, `11025`, `16000`, `22050`, `32000`, `44100`, `48000`, `96000`

## Query parameters

| Parameter                    | Type    | Default       | Description                                                                                                                                                                                                                                                |
| ---------------------------- | ------- | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `api_key`                    | string  | —             | **Required.** Your API key                                                                                                                                                                                                                                 |
| `speaker_diarization`        | boolean | `true`        | Identify and label distinct speakers                                                                                                                                                                                                                       |
| `audio_format`               | string  | (auto-detect) | Audio encoding format. Omit for self-describing formats; required for raw formats                                                                                                                                                                          |
| `sample_rate`                | integer | —             | Sample rate in Hz. Required for raw formats only                                                                                                                                                                                                           |
| `num_channels`               | integer | —             | Number of channels (1–8). Required for raw formats only                                                                                                                                                                                                    |
| `start_redaction_padding_ms` | integer | `100`         | Extra silence (ms) prepended before each redacted audio range                                                                                                                                                                                              |
| `end_redaction_padding_ms`   | integer | `0`           | Extra silence (ms) appended after each redacted audio range                                                                                                                                                                                                |
| `language`                   | string  | (auto-detect) | Optional language hint as a case-insensitive ISO 639-1 code (e.g. `en`, `fr`). BCP 47 region/script subtags (e.g. `en-US`) are accepted; only the primary language subtag is used. When omitted, the language is detected automatically for each utterance |

## Connection flow

1. Connect to the WebSocket endpoint with `api_key` and any optional parameters.
2. Stream audio data as **binary** WebSocket frames. Frames can be any size.
3. Receive **frame pairs** per utterance: a JSON text frame, optionally followed by a binary MP3 frame.
4. Send an **empty text frame** (`""`) to signal end of audio.
5. Receive a `done` JSON frame, optionally followed by a final binary MP3 frame for any trailing audio.
6. The connection closes automatically.

## Server messages

The server sends frame pairs: a JSON text frame indicating the utterance, optionally followed by a binary MP3 frame. The `redacted_audio` field in the JSON tells you whether a binary frame follows.

### `utterance` (JSON + optional binary MP3)

Sent when a speech segment has been transcribed and redacted.

**JSON frame:**

```json theme={null}
{
  "type": "utterance",
  "utterance": {
    "utterance_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "text": "Hello, my name is <pii:name></pii:name>.",
    "start_ms": 0,
    "duration_ms": 3000,
    "speaker": 1,
    "language": "en"
  },
  "redacted_audio": {
    "start_ms": 0,
    "duration_ms": 3000
  }
}
```

When `redacted_audio` is not `null`, a binary MP3 frame follows immediately. It covers the window from the last emitted audio point to the end of this utterance, with PII/PHI ranges silenced.

When `redacted_audio` is `null`, no binary frame follows. This occurs for out-of-order utterances whose audio window was already emitted in a previous clip — the redacted text is still delivered.

#### Utterance fields

| Field            | Type          | Description                                                                                                                                                                              |
| ---------------- | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `utterance_uuid` | string (UUID) | Unique identifier for this utterance                                                                                                                                                     |
| `text`           | string        | Redacted text. Each detected PII/PHI span is replaced with an empty marker tag (e.g. `<pii:name></pii:name>`, `<pii:ssn></pii:ssn>`, `<phi></phi>`), with the surrounding text preserved |
| `start_ms`       | integer       | Start time in milliseconds from the beginning of the stream                                                                                                                              |
| `duration_ms`    | integer       | Duration of the utterance in milliseconds                                                                                                                                                |
| `speaker`        | integer       | Speaker number, 1-indexed. Consistent within a connection                                                                                                                                |
| `language`       | string        | Detected language code (e.g. `"en"`, `"fr"`)                                                                                                                                             |

#### Redacted audio info fields

| Field         | Type    | Description                                                                 |
| ------------- | ------- | --------------------------------------------------------------------------- |
| `start_ms`    | integer | Start time of the MP3 clip in milliseconds from the beginning of the stream |
| `duration_ms` | integer | Duration of the MP3 clip in milliseconds                                    |

### `done` (JSON + optional binary MP3)

Sent after all audio has been processed, in response to the end-of-stream signal.

```json theme={null}
{
  "type": "done",
  "duration_ms": 45000,
  "trailing_redacted_audio": {
    "start_ms": 43000,
    "duration_ms": 2000
  }
}
```

When `trailing_redacted_audio` is not `null`, a binary MP3 frame follows containing any remaining audio after the last utterance, with any applicable PII/PHI ranges silenced.

When `trailing_redacted_audio` is `null`, no binary frame follows.

### `error`

Sent if redaction fails during processing. The connection closes after this message. No binary frame follows.

```json theme={null}
{
  "type": "error",
  "error": "Internal server error"
}
```

## WebSocket close codes

| Code   | Meaning                                                                                                                                                           |
| ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `1000` | Normal closure after a successful `done` message                                                                                                                  |
| `1003` | Invalid connection parameters (unsupported `audio_format`, invalid `sample_rate`, `num_channels`, or `language`; raw format missing `sample_rate`/`num_channels`) |
| `4003` | Request could not be validated, or is not permitted (auth failure, missing model access)                                                                          |
| `4029` | Insufficient credits, or concurrent-connection limit exceeded                                                                                                     |

An `error` JSON message is sent before the connection closes (except on `1000`).

## Rate limits

* Concurrent connection limits apply per model.
* Connections that exceed limits are rejected during the WebSocket handshake with close code `4029`.

See [Authentication and rate limits](/guides/authentication) for retry guidance.

## Redaction tags

Each detected PII/PHI span is replaced with an empty marker tag in the transcript text: `<phi></phi>` for health information and `<pii:CATEGORY></pii:CATEGORY>` for personal information, where CATEGORY identifies the detected entity type. The surrounding text is preserved. For more detail, see the **PII/PHI Redaction (Batch)** API reference.

Currently, all entity types the model can detect are redacted. Per-entity configurability is planned for a future release.

## Examples

<Tabs>
  <Tab title="Python (aiohttp)">
    ```python theme={null}
    import asyncio
    import json
    import aiohttp

    API_KEY = "YOUR_API_KEY"
    AUDIO_FILE = "recording.ogg"
    CHUNK_SIZE = 8192

    async def redact_streaming():
        url = (
            "wss://platform.modulate.ai/api/velma-2-pii-phi-redaction-streaming"
            f"?api_key={API_KEY}"
            "&speaker_diarization=true"
            "&start_redaction_padding_ms=100"
            "&end_redaction_padding_ms=0"
        )

        utterances = []
        audio_clips = []

        async with aiohttp.ClientSession() as session:
            async with session.ws_connect(url) as ws:

                async def send_audio():
                    with open(AUDIO_FILE, "rb") as f:
                        while chunk := f.read(CHUNK_SIZE):
                            await ws.send_bytes(chunk)
                            await asyncio.sleep(CHUNK_SIZE / 4000)
                    await ws.send_str("")

                send_task = asyncio.create_task(send_audio())

                try:
                    done = False
                    async for msg in ws:
                        if msg.type == aiohttp.WSMsgType.TEXT:
                            data = json.loads(msg.data)
                            if data["type"] == "utterance":
                                u = data["utterance"]
                                utterances.append(u)
                                print(f"[Speaker {u['speaker']}] ({u['language']}) {u['start_ms']}ms: {u['text']}")
                            elif data["type"] == "done":
                                print(f"\nDone. Duration: {data['duration_ms']}ms")
                                done = True
                                if not data.get("trailing_redacted_audio"):
                                    break
                            elif data["type"] == "error":
                                print(f"Error: {data['error']}")
                                break
                        elif msg.type == aiohttp.WSMsgType.BINARY:
                            audio_clips.append(msg.data)
                            if done:
                                break
                        elif msg.type in (
                            aiohttp.WSMsgType.ERROR,
                            aiohttp.WSMsgType.CLOSE,
                            aiohttp.WSMsgType.CLOSED,
                        ):
                            break
                finally:
                    if not send_task.done():
                        send_task.cancel()

        if audio_clips:
            with open("redacted.mp3", "wb") as f:
                for clip in audio_clips:
                    f.write(clip)

    asyncio.run(redact_streaming())
    ```
  </Tab>

  <Tab title="JavaScript (Node.js)">
    ```javascript theme={null}
    const WebSocket = require("ws");
    const fs = require("fs");

    const API_KEY = "YOUR_API_KEY";
    const AUDIO_FILE = "recording.ogg";
    const CHUNK_SIZE = 8192;

    const url = new URL(
      "wss://platform.modulate.ai/api/velma-2-pii-phi-redaction-streaming"
    );
    url.searchParams.set("api_key", API_KEY);
    url.searchParams.set("speaker_diarization", "true");
    url.searchParams.set("start_redaction_padding_ms", "100");
    url.searchParams.set("end_redaction_padding_ms", "0");

    const ws = new WebSocket(url.toString());
    const utterances = [];
    const audioClips = [];
    let isDone = false;

    ws.on("open", () => {
      const stream = fs.createReadStream(AUDIO_FILE, { highWaterMark: CHUNK_SIZE });
      stream.on("data", (chunk) => ws.send(chunk));
      stream.on("end", () => ws.send(""));
    });

    ws.on("message", (data, isBinary) => {
      if (isBinary) {
        audioClips.push(data);
        if (isDone) finalize();
        return;
      }

      const msg = JSON.parse(data.toString());
      if (msg.type === "utterance") {
        utterances.push(msg.utterance);
        console.log(
          `[Speaker ${msg.utterance.speaker}] (${msg.utterance.language}) ` +
          `${msg.utterance.start_ms}ms: ${msg.utterance.text}`
        );
      } else if (msg.type === "done") {
        console.log(`\nDone. Duration: ${msg.duration_ms}ms`);
        isDone = true;
        if (!msg.trailing_redacted_audio) finalize();
      } else if (msg.type === "error") {
        console.error("Error:", msg.error);
        ws.close();
      }
    });

    function finalize() {
      if (audioClips.length > 0) {
        const combined = Buffer.concat(audioClips);
        fs.writeFileSync("redacted.mp3", combined);
      }
      ws.close();
    }

    ws.on("error", (err) => console.error("WebSocket error:", err.message));
    ```
  </Tab>
</Tabs>

<Note>
  WebSocket APIs cannot be tested with cURL. For command-line testing, use [`websocat`](https://github.com/vi/websocat).
</Note>

## Related

* [Which API should I use?](/guides/which-api) — PII/PHI redaction vs PII/PHI tagging, batch vs streaming
* [Transcription](/get-started/stt#signals) — PII/PHI tagging option in the STT transcription APIs
* [Authentication and rate limits](/guides/authentication)


# Speech-to-Text Transcription Batch Multilingual
Source: https://docs.modulate.ai/api-reference/stt/batch

api/velma_2_stt_batch.yaml POST /api/velma-2-stt-batch
Multilingual batch transcription with automatic language detection, speaker diarization, emotion and accent detection, and PII/PHI tagging.



# Speech-to-Text Transcription Batch English VFast
Source: https://docs.modulate.ai/api-reference/stt/batch-english-vfast

api/velma_2_stt_batch_english_vfast.yaml POST /api/velma-2-stt-batch-english-vfast
Fast English-only batch transcription with optional word-level timings and speaker diarization. Trades enrichment features for the lowest possible turnaround.



# Multilingual Fast Transcription Batch
Source: https://docs.modulate.ai/api-reference/stt/batch-multilingual-vfast

api/velma_2_stt_batch_multilingual_vfast.yaml POST /api/velma-2-stt-batch-multilingual-vfast
Fast multilingual batch transcription. Optionally declare the spoken language for the fastest path; otherwise the language is detected automatically.



# Speech-to-text Transcription
Source: https://docs.modulate.ai/api-reference/stt/overview

Speech-to-text APIs — multilingual batch transcription, fast English-only batch, fast multilingual batch, real-time streaming, and low-latency English streaming.

Modulate offers five speech-to-text endpoints. Pick the one that matches your latency, language, and feature needs.

|                                          | Multilingual (batch)             | Multilingual (streaming)     | English Fast (batch)             | English Fast (streaming)                    | Multilingual Fast (batch)                     |
| ---------------------------------------- | -------------------------------- | ---------------------------- | -------------------------------- | ------------------------------------------- | --------------------------------------------- |
| **Use case**                             | Transcription with rich metadata | Real-time transcription      | Fast English-only transcription  | Low-latency English real-time transcription | Fast multilingual transcription               |
| **Protocol**                             | HTTP POST                        | WebSocket                    | HTTP POST                        | WebSocket                                   | HTTP POST                                     |
| **Languages**                            | Multilingual                     | Multilingual                 | English only                     | English only                                | Multilingual                                  |
| **Language declaration**                 | ✓ (optional `language` hint)     | ✓ (optional `language` hint) | —                                | —                                           | ✓ (optional `language`, skips auto-detection) |
| **Speaker diarization**                  | ✓                                | ✓                            | ✓ (opt-in `speaker_diarization`) | —                                           | —                                             |
| **Word-level timings**                   | —                                | —                            | ✓ (opt-in `time_stamps`)         | —                                           | —                                             |
| **Word-level timings**                   | —                                | —                            | ✓ (opt-in `time_stamps`)         | —                                           | —                                             |
| **Emotion / accent detection**           | ✓                                | ✓                            | —                                | —                                           | —                                             |
| **PII/PHI tagging**                      | ✓                                | ✓                            | —                                | —                                           | —                                             |
| **Partial transcripts during streaming** | —                                | ✓ (opt-in `partial_results`) | —                                | ✓ (every \~1.5 s)                           | —                                             |

For a side-by-side comparison with the other Modulate capabilities, see [Which API should I use?](/guides/which-api).

## Authentication

Batch endpoints use the `X-API-Key` header. Streaming endpoints use an `api_key` query parameter at connection time. See [Authentication and rate limits](/guides/authentication).


# Speech-to-Text Transcription Streaming Multilingual
Source: https://docs.modulate.ai/api-reference/stt/streaming

Real-time speech-to-text over WebSocket, with optional speaker diarization, emotion detection, accent detection, and PII/PHI tagging.

Real-time speech-to-text over WebSocket. Streams audio to the server and receives transcribed utterances as they are processed, with optional speaker diarization, emotion detection, accent detection, and PII/PHI tagging.

## Endpoint

```text theme={null}
wss://platform.modulate.ai/api/velma-2-stt-streaming
```

## Authentication

Pass your API key as a query parameter when opening the connection.

```text theme={null}
wss://platform.modulate.ai/api/velma-2-stt-streaming?api_key=YOUR_API_KEY
```

<Warning>
  Unlike the batch endpoints, the streaming API does not use an `X-API-Key` header. The key must be in the query string at connection time.
</Warning>

See [Authentication and rate limits](/guides/authentication) for how to obtain and manage API keys.

## Supported audio formats

Self-describing container formats are auto-detected from headers (no `audio_format` query parameter needed). Raw / headerless formats require `audio_format`, `sample_rate`, and `num_channels`. For the authoritative list of accepted values, see the spec's `audio_format` enum or [Transcription](/get-started/stt).

Opus is recommended when you control the encoder — high quality at low bandwidth.

## Query parameters

| Parameter             | Type    | Default       | Description                                                                                                                                                                                                                                                |
| --------------------- | ------- | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `api_key`             | string  | —             | **Required.** Your API key                                                                                                                                                                                                                                 |
| `speaker_diarization` | boolean | `true`        | Identify and label distinct speakers                                                                                                                                                                                                                       |
| `emotion_signal`      | boolean | `false`       | Detect emotional tone per utterance                                                                                                                                                                                                                        |
| `accent_signal`       | boolean | `false`       | Detect speaker accent per utterance                                                                                                                                                                                                                        |
| `deepfake_signal`     | boolean | `false`       | Per-utterance synthetic-voice (deepfake) score                                                                                                                                                                                                             |
| `pii_phi_tagging`     | boolean | `false`       | Wrap PII/PHI in tags within utterance text                                                                                                                                                                                                                 |
| `partial_results`     | boolean | `false`       | Stream interim `partial_utterance` messages with in-progress text — and the latest interim `emotion`, `accent`, and `deepfake_score` values when those signals are enabled — before each utterance is finalized                                            |
| `language`            | string  | (auto-detect) | Optional language hint as a case-insensitive ISO 639-1 code (e.g. `en`, `fr`). BCP 47 region/script subtags (e.g. `en-US`) are accepted; only the primary language subtag is used. When omitted, the language is detected automatically for each utterance |

Every parameter except `api_key` can also be set — and overridden — in an optional [configuration frame](#configuration-frame) sent as the first WebSocket text frame.

For a full explanation of what each feature does and when to enable it, see [Transcription](/get-started/stt#signals).

## Connection flow

1. Connect to the WebSocket endpoint with `api_key` and any optional feature parameters.
2. Optionally send a JSON [configuration frame](#configuration-frame) as the first text frame, before any audio. If the first frame is binary, it is treated as the first audio chunk and the query-parameter defaults apply.
3. Stream raw audio as **binary** WebSocket frames. Frames can be any size.
4. Receive `utterance` JSON messages as speech is transcribed. If `partial_results=true`, also receive `partial_utterance` previews for the currently active utterance.
5. Send an **empty text frame** (`""`) to signal end of audio.
6. Receive a `done` message containing total audio duration.
7. The connection closes automatically.

## Configuration frame

You can optionally send a JSON configuration as the **first** WebSocket text frame, before any audio. When the first frame is text, it is parsed as this configuration; when the first frame is binary, it is treated as the first audio chunk and the query-parameter defaults apply — existing binary-first clients need no changes. Every field is optional, and a field present in the configuration overrides the matching query parameter.

| Field                 | Type           | Description                                                                                                                         |
| --------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `speaker_diarization` | boolean        | Overrides the `speaker_diarization` query parameter                                                                                 |
| `emotion_signal`      | boolean        | Overrides the `emotion_signal` query parameter                                                                                      |
| `accent_signal`       | boolean        | Overrides the `accent_signal` query parameter                                                                                       |
| `deepfake_signal`     | boolean        | Overrides the `deepfake_signal` query parameter                                                                                     |
| `pii_phi_tagging`     | boolean        | Overrides the `pii_phi_tagging` query parameter                                                                                     |
| `partial_results`     | boolean        | Overrides the `partial_results` query parameter                                                                                     |
| `language`            | string \| null | Overrides the `language` query parameter                                                                                            |
| `custom_terms`        | array \| null  | Custom vocabulary to bias transcription toward domain terms and names — see [Custom vocabulary](/get-started/stt#custom-vocabulary) |

```json theme={null}
{
  "emotion_signal": true,
  "language": "en",
  "custom_terms": ["Modulate", { "term": "Velma", "pronunciations": ["VEL-muh"] }]
}
```

An invalid configuration frame (malformed JSON or a value outside the configuration schema) triggers an `error` message and closes the connection with code `1003`.

## Server messages

### `utterance`

Sent each time a speech segment is transcribed.

```json theme={null}
{
  "type": "utterance",
  "utterance": {
    "utterance_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "text": "Hello, how are you today?",
    "start_ms": 0,
    "duration_ms": 2500,
    "speaker": 1,
    "language": "en",
    "emotion": "Neutral",
    "accent": "American",
    "deepfake_score": null
  }
}
```

#### Utterance fields

| Field            | Type           | Description                                                                                                                                                                |
| ---------------- | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `utterance_uuid` | string (UUID)  | Unique identifier for this utterance                                                                                                                                       |
| `text`           | string         | Transcribed text                                                                                                                                                           |
| `start_ms`       | integer        | Start time in milliseconds from the beginning of the stream                                                                                                                |
| `duration_ms`    | integer        | Duration of the utterance in milliseconds                                                                                                                                  |
| `speaker`        | integer        | Speaker number, 1-indexed                                                                                                                                                  |
| `language`       | string         | Detected language code (e.g. `"en"`, `"fr"`)                                                                                                                               |
| `emotion`        | string \| null | Detected emotion. `null` when `emotion_signal` is disabled                                                                                                                 |
| `accent`         | string \| null | Detected accent. `null` when `accent_signal` is disabled                                                                                                                   |
| `deepfake_score` | float \| null  | Synthetic-voice score from `0.0` (likely natural) to `1.0` (likely synthetic). `null` when `deepfake_signal` is disabled or when the utterance is shorter than 0.5 seconds |

For all valid `emotion` and `accent` values, see [Transcription](/get-started/stt#signals).

### `partial_utterance`

Sent only when `partial_results=true`. Delivers in-progress text for the currently active utterance as a low-latency preview. Each partial also carries the latest interim `emotion`, `accent`, and `deepfake_score` values for the utterance when those signals are enabled; each is `null` until a value is available. Each `partial_utterance` supersedes the previous one for the same utterance; the finalized `utterance` message supersedes all preceding partials.

```json theme={null}
{
  "type": "partial_utterance",
  "partial_utterance": {
    "text": "Hello, how are",
    "start_ms": 0,
    "speaker": 1,
    "emotion": null,
    "accent": null,
    "deepfake_score": null
  }
}
```

#### Partial utterance fields

| Field            | Type            | Description                                                                                                                         |
| ---------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `text`           | string          | In-progress transcribed text. May be empty                                                                                          |
| `start_ms`       | integer \| null | Start time in milliseconds from the beginning of the stream. `null` if timing data is not yet available                             |
| `speaker`        | integer \| null | Speaker number, 1-indexed. `null` if the speaker has not yet been identified                                                        |
| `emotion`        | string \| null  | Latest interim emotion label for the in-progress utterance. `null` when `emotion_signal` is disabled or no value is available yet   |
| `accent`         | string \| null  | Latest interim accent label for the in-progress utterance. `null` when `accent_signal` is disabled or no value is available yet     |
| `deepfake_score` | float \| null   | Latest interim deepfake score for the in-progress utterance. `null` when `deepfake_signal` is disabled or no value is available yet |

### `done`

Sent after all audio has been processed, in response to the end-of-stream signal.

```json theme={null}
{
  "type": "done",
  "duration_ms": 45000
}
```

### `error`

Sent if transcription fails. The connection closes after this message.

```json theme={null}
{
  "type": "error",
  "error": "Internal server error"
}
```

## WebSocket close codes

| Code   | Meaning                                                                                                                                                                                                                                                                 |
| ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `1000` | Normal closure after a successful `done` message                                                                                                                                                                                                                        |
| `1003` | Invalid connection parameters (unsupported `audio_format`, invalid `sample_rate`, `num_channels`, or `language`; raw format missing `sample_rate`/`num_channels`), or an invalid first-frame configuration (malformed JSON or a value outside the configuration schema) |
| `4003` | Request could not be validated, or is not permitted (auth failure, missing model access)                                                                                                                                                                                |
| `4029` | Insufficient credits, or concurrent-connection limit exceeded                                                                                                                                                                                                           |

An `error` JSON message is sent before the connection closes (except on `1000`).

## Rate limits

* Concurrent connection limits apply per model.
* Connections that exceed limits are rejected during the WebSocket handshake with close code `4029`.

See [Authentication and rate limits](/guides/authentication) for retry guidance.

## Examples

<Tabs>
  <Tab title="Python (aiohttp)">
    ```python theme={null}
    import asyncio
    import json
    import aiohttp

    API_KEY = "YOUR_API_KEY"
    AUDIO_FILE = "recording.opus"
    CHUNK_SIZE = 8192

    async def transcribe_streaming():
        url = (
            f"wss://platform.modulate.ai/api/velma-2-stt-streaming"
            f"?api_key={API_KEY}"
            f"&speaker_diarization=true"
            f"&emotion_signal=true"
            f"&accent_signal=true"
        )

        utterances = []

        async with aiohttp.ClientSession() as session:
            async with session.ws_connect(url) as ws:

                async def send_audio():
                    with open(AUDIO_FILE, "rb") as f:
                        while chunk := f.read(CHUNK_SIZE):
                            await ws.send_bytes(chunk)
                            await asyncio.sleep(CHUNK_SIZE / 4000)
                    await ws.send_str("")

                send_task = asyncio.create_task(send_audio())

                try:
                    async for msg in ws:
                        if msg.type == aiohttp.WSMsgType.TEXT:
                            data = json.loads(msg.data)
                            if data["type"] == "utterance":
                                u = data["utterance"]
                                utterances.append(u)
                                print(f"[Speaker {u['speaker']}] {u['text']}")
                            elif data["type"] == "done":
                                print(f"Done. Duration: {data['duration_ms']}ms")
                                break
                            elif data["type"] == "error":
                                print(f"Error: {data['error']}")
                                break
                        elif msg.type in (
                            aiohttp.WSMsgType.ERROR,
                            aiohttp.WSMsgType.CLOSE,
                            aiohttp.WSMsgType.CLOSED,
                        ):
                            break
                finally:
                    if not send_task.done():
                        send_task.cancel()

        full_text = " ".join(u["text"] for u in utterances)
        print(f"\nFull transcript:\n{full_text}")

    asyncio.run(transcribe_streaming())
    ```
  </Tab>

  <Tab title="JavaScript (Node.js)">
    ```javascript theme={null}
    const WebSocket = require("ws");
    const fs = require("fs");

    const API_KEY = "YOUR_API_KEY";
    const AUDIO_FILE = "recording.opus";
    const CHUNK_SIZE = 8192;

    const url = new URL("wss://platform.modulate.ai/api/velma-2-stt-streaming");
    url.searchParams.set("api_key", API_KEY);
    url.searchParams.set("speaker_diarization", "true");
    url.searchParams.set("emotion_signal", "true");

    const ws = new WebSocket(url.toString());
    const utterances = [];

    ws.on("open", () => {
      const stream = fs.createReadStream(AUDIO_FILE, { highWaterMark: CHUNK_SIZE });
      stream.on("data", (chunk) => ws.send(chunk));
      stream.on("end", () => ws.send(""));
    });

    ws.on("message", (data) => {
      const msg = JSON.parse(data.toString());
      if (msg.type === "utterance") {
        utterances.push(msg.utterance);
        console.log(`[Speaker ${msg.utterance.speaker}] ${msg.utterance.text}`);
      } else if (msg.type === "done") {
        console.log(`Done. Duration: ${msg.duration_ms}ms`);
        console.log("Transcript:", utterances.map((u) => u.text).join(" "));
        ws.close();
      } else if (msg.type === "error") {
        console.error("Error:", msg.error);
        ws.close();
      }
    });

    ws.on("error", (err) => console.error("WebSocket error:", err.message));
    ```
  </Tab>
</Tabs>

<Note>
  WebSocket APIs cannot be tested with cURL. For command-line testing, use [`websocat`](https://github.com/vi/websocat).
</Note>

## Related

* [Which API should I use?](/guides/which-api) — when streaming is the right choice vs batch
* [Transcription](/get-started/stt#signals) — full reference for diarization, emotion, accent, and PII/PHI tagging
* [Authentication and rate limits](/guides/authentication)


# Speech-to-Text Streaming English
Source: https://docs.modulate.ai/api-reference/stt/streaming-vfast

Low-latency English speech-to-text over WebSocket. Emits rolling partial transcripts during streaming, with optional utterance segmentation at pauses (endpointing) or a single final transcript at end-of-stream. No enrichments.

Low-latency English speech-to-text over WebSocket. Emits a rolling partial transcript every \~1.5 seconds while audio streams in, then delivers one complete final transcript at end-of-stream — or, with [endpointing enabled](#utterance-segmentation-endpointing), a final transcript for each utterance as the speaker pauses. Pure transcription — no speaker diarization, emotion detection, accent detection, or PII/PHI tagging.

## Endpoint

```text theme={null}
wss://platform.modulate.ai/api/velma-2-stt-streaming-english-v2
```

## Authentication

Pass your API key as a query parameter when opening the connection.

```text theme={null}
wss://platform.modulate.ai/api/velma-2-stt-streaming-english-v2?api_key=YOUR_API_KEY&audio_format=ogg
```

<Warning>
  Unlike the batch endpoints, this API does not use an `X-API-Key` header. The key must be in the query string at connection time.
</Warning>

See [Authentication and rate limits](/guides/authentication) for how to obtain and manage API keys.

## Supported audio formats

**Container formats** — `sample_rate` and `num_channels` are ignored if supplied:

`mp3`, `wav`, `flac`, `m4a`, `mp4`, `ogg`, `opus`, `webm`, `aac`, `aiff`, `wma`, `amr`, and `au`, plus 67 others

<Accordion title="All 80 accepted container values">
  `3g2`, `3ga`, `3gp`, `3gpp`, `8svx`, `aa3`, `aac`, `ac3`, `act`, `adts`, `aif`, `aifc`, `aiff`,
  `amb`, `amr`, `asf`, `at3`, `au`, `avr`, `awb`, `bwf`, `c2`, `caf`, `dss`, `dts`, `dtshd`,
  `eac3`, `ec3`, `f4a`, `f4b`, `flac`, `gsm`, `iff`, `m2a`, `m2ts`, `m4a`, `m4b`, `m4r`, `m4v`,
  `mka`, `mkv`, `mlp`, `mp+`, `mp1`, `mp2`, `mp3`, `mp4`, `mpa`, `mpc`, `mpga`, `mpp`, `mts`,
  `oga`, `ogg`, `ogx`, `oma`, `omg`, `opus`, `paf`, `pvf`, `qcp`, `ra`, `rf64`, `rm`, `rmvb`,
  `snd`, `spx`, `svx`, `thd`, `ts`, `tta`, `voc`, `vqf`, `w64`, `wav`, `wave`, `weba`, `webm`,
  `wma`, `wmv`
</Accordion>

The MP4-family values (`mp4`, `m4a`, `m4b`, `m4r`, `m4v`, `3gp`, `3gpp`, `3ga`, `3g2`, `f4a`, `f4b`) must be sent in a streamable layout; otherwise the connection ends with an audio-processing error.

Opus audio is accepted either as `opus` or, for Opus-in-Ogg streams, as `ogg`.

**Raw PCM formats** — `sample_rate` and `num_channels` are required:

`s8`, `s16le`, `s16be`, `s24le`, `s24be`, `s32le`, `s32be`, `u8`, `u16le`, `u16be`, `u24le`, `u24be`, `u32le`, `u32be`, `f32le`, `f32be`, `f64le`, `f64be`, `mulaw`, `alaw`, `g722`, `vox`

`g722` and `vox` are mono-only: `num_channels` must be `1`.

**Valid sample rates:** 8000, 11025, 16000, 22050, 32000, 44100, 48000, 96000

For lowest end-to-end latency, send `audio_format=s16le&sample_rate=16000&num_channels=1`. This matches the model's native input format and bypasses the server's audio decoder entirely.

## Query parameters

| Parameter      | Required     | Description                                                                                                                                                                                                              |
| -------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `api_key`      | Yes          | Your API key                                                                                                                                                                                                             |
| `audio_format` | Yes          | Container or raw PCM format of the audio you will stream. See supported formats above.                                                                                                                                   |
| `sample_rate`  | Raw PCM only | Source sample rate in Hz. Required when `audio_format` is a raw PCM type; ignored for container formats.                                                                                                                 |
| `num_channels` | Raw PCM only | Number of audio channels (1–8). Required when `audio_format` is a raw PCM type; ignored for container formats. The server downmixes to mono internally.                                                                  |
| `endpointing`  | No           | `true` or `false` (default `false`). When `true`, speech is segmented into utterances at pauses — see [Utterance segmentation](#utterance-segmentation-endpointing). Any other value is rejected with close code `1003`. |

This endpoint accepts no enrichment toggles. Diarization, emotion, accent, and PII parameters are not recognized and have no effect. Use [STT Streaming](/api-reference/stt/streaming) if you need those features.

## Utterance segmentation (endpointing)

By default, the connection produces one final `utterance` at end-of-stream covering everything you sent, and each `partial_utterance` reflects the whole stream so far.

With `endpointing=true`, speech is segmented at pauses:

* Each `partial_utterance` contains the complete transcript of the **current segment** so far (still replace, never append).
* Each time a pause ends a segment, you immediately receive a final `utterance` for it — final text arrives shortly after the speaker stops, instead of at end-of-stream. Each final carries `start_ms` and `duration_ms` for its speech.
* The full transcript is the concatenation of the final `utterance` texts, in order.
* Every connection still ends with at least one final `utterance` (a stream containing no speech yields one with empty `text`), followed by `done`.

Use endpointing for live conversations — voice agents, assistants, meeting captions — where you want final text per utterance as it happens. Leave it off for whole-recording transcription where one final transcript is simpler to consume.

## Connection flow

1. Connect to the WebSocket endpoint with `api_key`, `audio_format`, (for raw PCM) `sample_rate` and `num_channels`, and optionally `endpointing`.
2. Stream audio as **binary** WebSocket frames. Frames can be any size; 4–64 KB is typical.
3. Receive `partial_utterance` JSON messages every \~1.5 seconds. Each contains the **complete transcript of its scope so far** — replace any previously displayed partial, do not append.
4. With `endpointing=true`, receive a final `utterance` message each time a pause ends a segment.
5. Send an **empty text frame** (`""`) to signal end of audio.
6. Receive the final `utterance` for any content not already finalized at a pause (the whole stream with endpointing off; skipped when the last segment was already finalized mid-stream).
7. Receive a `done` message with total audio duration.
8. The connection closes automatically.

## Server messages

### `partial_utterance`

Sent roughly every 1.5 seconds while audio is streaming. Each message contains the **complete transcript built so far for the current scope** — the whole connection by default, or the current speech segment when `endpointing=true`. It is not a delta from the previous message: replace your displayed partial text with each new value — never append.

```json theme={null}
{
  "type": "partial_utterance",
  "partial_utterance": {
    "text": "Hello, how are you",
    "is_final": false
  }
}
```

| Field                        | Type    | Description                                        |
| ---------------------------- | ------- | -------------------------------------------------- |
| `type`                       | string  | Always `"partial_utterance"`                       |
| `partial_utterance.text`     | string  | Complete transcript so far. Replace, don't append. |
| `partial_utterance.is_final` | boolean | Always `false`                                     |

### `utterance`

The final transcript for one utterance. Will not be revised. With endpointing off, sent exactly once at end-of-stream, covering the entire audio stream. With `endpointing=true`, sent each time a pause ends a segment; the full transcript is the concatenation of every `utterance` text in order.

```json theme={null}
{
  "type": "utterance",
  "utterance": {
    "text": "Hello, how are you doing today?",
    "is_final": true,
    "start_ms": 0,
    "duration_ms": 2360
  }
}
```

| Field                   | Type    | Description                                                                                                                                                                                    |
| ----------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `type`                  | string  | Always `"utterance"`                                                                                                                                                                           |
| `utterance.text`        | string  | Final transcript for this utterance. Will not be revised.                                                                                                                                      |
| `utterance.is_final`    | boolean | Always `true`                                                                                                                                                                                  |
| `utterance.start_ms`    | integer | Start of this utterance's speech, in milliseconds from the beginning of the stream. Approximate. `0` with endpointing off.                                                                     |
| `utterance.duration_ms` | integer | Duration of this utterance's speech in milliseconds, excluding the pause that ended it. Approximate. With endpointing off, covers the whole stream including any leading and trailing silence. |

### `done`

Sent immediately after the final `utterance`. Signals stream completion. The connection closes shortly after.

```json theme={null}
{
  "type": "done",
  "duration_ms": 14253
}
```

| Field         | Type    | Description                          |
| ------------- | ------- | ------------------------------------ |
| `type`        | string  | Always `"done"`                      |
| `duration_ms` | integer | Total audio duration in milliseconds |

### `error`

Sent if something goes wrong. The connection closes after this message. No further messages follow an `error`.

```json theme={null}
{
  "type": "error",
  "error": "Invalid audio_format='xyz'."
}
```

| Field   | Type   | Description                             |
| ------- | ------ | --------------------------------------- |
| `type`  | string | Always `"error"`                        |
| `error` | string | Human-readable description of the error |

## WebSocket close codes

| Code   | Meaning                                                                                                                  |
| ------ | ------------------------------------------------------------------------------------------------------------------------ |
| `1000` | Normal closure after the `done` message                                                                                  |
| `1003` | Invalid query parameters — missing `audio_format`, invalid `sample_rate`, invalid `endpointing`, unsupported value, etc. |
| `1011` | Internal server error. An `error` message is sent before the close.                                                      |
| `1013` | The service is temporarily at capacity. Not an account or billing condition — retry the connection.                      |
| `4001` | The `api_key` query parameter is missing or invalid                                                                      |
| `4002` | Audio bytes did not match the declared raw PCM format, or the audio could not be decoded mid-stream                      |
| `4003` | The request is not permitted                                                                                             |
| `4004` | The API key does not have access to this model                                                                           |
| `4029` | Insufficient credits                                                                                                     |
| `4030` | Concurrent request limit reached                                                                                         |
| `4031` | Monthly usage limit reached                                                                                              |

## Rate limits

* Concurrent connection limits apply per model; exceeding them rejects the handshake with close code `4030`.

See [Authentication and rate limits](/guides/authentication) for retry guidance.

## Examples

<Tabs>
  <Tab title="Python (websockets)">
    ```python theme={null}
    import asyncio
    import json
    import os
    import websockets

    API_KEY = os.environ["MODULATE_API_KEY"]
    AUDIO_FILE = "recording.ogg"
    CHUNK_SIZE = 8192

    async def transcribe():
        url = (
            f"wss://platform.modulate.ai/api/velma-2-stt-streaming-english-v2"
            f"?api_key={API_KEY}&audio_format=ogg"
        )

        async with websockets.connect(url, max_size=None) as ws:
            async def send_audio():
                with open(AUDIO_FILE, "rb") as f:
                    while chunk := f.read(CHUNK_SIZE):
                        await ws.send(chunk)
                await ws.send("")  # signal end-of-stream

            send_task = asyncio.create_task(send_audio())

            try:
                async for msg in ws:
                    data = json.loads(msg)
                    if data["type"] == "partial_utterance":
                        # Replace any previously displayed partial — not a delta
                        print(f"\r[partial] {data['partial_utterance']['text']}", end="", flush=True)
                    elif data["type"] == "utterance":
                        print(f"\n[final]   {data['utterance']['text']}")
                    elif data["type"] == "done":
                        print(f"\nDone. Duration: {data['duration_ms']}ms")
                        break
                    elif data["type"] == "error":
                        print(f"\nError: {data['error']}")
                        break
            finally:
                if not send_task.done():
                    send_task.cancel()

    asyncio.run(transcribe())
    ```
  </Tab>

  <Tab title="JavaScript (Node.js)">
    ```javascript theme={null}
    const WebSocket = require("ws");
    const fs = require("fs");

    const API_KEY = process.env.MODULATE_API_KEY;
    const AUDIO_FILE = "recording.ogg";
    const CHUNK_SIZE = 8192;

    const url = new URL("wss://platform.modulate.ai/api/velma-2-stt-streaming-english-v2");
    url.searchParams.set("api_key", API_KEY);
    url.searchParams.set("audio_format", "ogg");

    const ws = new WebSocket(url.toString());

    ws.on("open", () => {
      const stream = fs.createReadStream(AUDIO_FILE, { highWaterMark: CHUNK_SIZE });
      stream.on("data", (chunk) => ws.send(chunk));
      stream.on("end", () => ws.send(""));  // signal end-of-stream
    });

    ws.on("message", (data) => {
      const msg = JSON.parse(data.toString());
      if (msg.type === "partial_utterance") {
        // Each partial is the full transcript so far — replace, don't append
        process.stdout.write(`\r[partial] ${msg.partial_utterance.text}`);
      } else if (msg.type === "utterance") {
        console.log(`\n[final]   ${msg.utterance.text}`);
      } else if (msg.type === "done") {
        console.log(`\nDone. Duration: ${msg.duration_ms}ms`);
        ws.close();
      } else if (msg.type === "error") {
        console.error("\nError:", msg.error);
        ws.close();
      }
    });

    ws.on("error", (err) => console.error("WebSocket error:", err.message));
    ws.on("close", (code) => console.log(`Connection closed: ${code}`));
    ```
  </Tab>
</Tabs>

<Note>
  The examples above work unchanged with endpointing: append `&endpointing=true` to the URL and the same message loop prints one `[final]` line per pause instead of a single one at the end.
</Note>

<Note>
  WebSocket APIs cannot be tested with cURL. For command-line testing, use [`websocat`](https://github.com/vi/websocat).
</Note>

## Related

* [Which API should I use?](/guides/which-api) — when Streaming v2 is the right choice vs STT Streaming or batch
* [STT Streaming](/api-reference/stt/streaming) — multilingual streaming with speaker diarization and enrichments
* [Authentication and rate limits](/guides/authentication)


# Deepfake Detection Batch
Source: https://docs.modulate.ai/api-reference/svd/batch

api/velma_2_synthetic_voice_detection_batch.yaml POST /api/velma-2-synthetic-voice-detection-batch
Detect synthetic (AI-generated) voice in a pre-recorded audio file. Returns per-frame deepfake scores.



# Deepfake Detection
Source: https://docs.modulate.ai/api-reference/svd/overview

Synthetic voice detection — deepfake detection on pre-recorded files (batch) or live audio (streaming).

Synthetic voice detection (SVD) returns per-frame deepfake scores for an audio source. Choose batch for files you already have, or streaming for live audio over WebSocket.

|                             | Batch                                                                                   | Streaming                        |
| --------------------------- | --------------------------------------------------------------------------------------- | -------------------------------- |
| **Use case**                | Deepfake detection on a file                                                            | Real-time deepfake detection     |
| **Protocol**                | HTTP POST                                                                               | WebSocket                        |
| **Audio formats**           | 97 file extensions, including MP3, WAV, FLAC, M4A, MP4, OGG, Opus, WebM, AAC, AIFF, MOV | Raw PCM and 80 container formats |
| **Synthetic voice scoring** | Per-frame                                                                               | Per-frame                        |

For a side-by-side comparison with the other Modulate capabilities, see [Which API should I use?](/guides/which-api).

## Authentication

Batch uses the `X-API-Key` header. Streaming uses an `api_key` query parameter at connection time. See [Authentication and rate limits](/guides/authentication).


# Deepfake Detection Streaming
Source: https://docs.modulate.ai/api-reference/svd/streaming

Real-time deepfake detection over WebSocket, with per-frame verdicts and confidence scores delivered as analysis windows complete.

Real-time synthetic voice detection over WebSocket. Streams audio to the server and receives per-frame verdicts (`synthetic`, `non-synthetic`, or `no-content`) with confidence scores as analysis windows complete.

For a conceptual explanation of how detection works — including windowing, silence trimming, and the `no-content` verdict — see [Deepfake Detection](/get-started/deepfake).

## Endpoint

```text theme={null}
wss://platform.modulate.ai/api/velma-2-synthetic-voice-detection-streaming
```

## Authentication

Pass your API key as a query parameter when opening the connection.

```text theme={null}
wss://platform.modulate.ai/api/velma-2-synthetic-voice-detection-streaming?api_key=YOUR_API_KEY&audio_format=s16le&sample_rate=16000&num_channels=1
```

<Warning>
  Unlike the batch endpoint, the streaming API does not use an `X-API-Key` header. The key must be in the query string at connection time.
</Warning>

## Query parameters

| Parameter      | Type    | Required    | Description                                                                                                          |
| -------------- | ------- | ----------- | -------------------------------------------------------------------------------------------------------------------- |
| `api_key`      | string  | Yes         | Your API key                                                                                                         |
| `audio_format` | string  | Yes         | Audio encoding format — see [Deepfake Detection](/get-started/deepfake)                                              |
| `sample_rate`  | integer | Conditional | Required for raw (headerless) formats. One of: `8000`, `11025`, `16000`, `22050`, `32000`, `44100`, `48000`, `96000` |
| `num_channels` | integer | Conditional | Required for raw formats. 1–8                                                                                        |

For supported format values and format selection guidance, see [Deepfake Detection](/get-started/deepfake).

## Connection flow

1. Connect with `api_key`, `audio_format`, and (for raw formats) `sample_rate` and `num_channels`.
2. Stream audio as **binary** WebSocket frames. Frames can be any size.
3. Receive `frame` JSON messages as analysis windows complete.
4. Send an **empty text frame** (`""`) to signal end of audio.
5. Receive a `done` message with total duration and frame count.
6. The connection closes automatically.

## Server messages

### Frame result

Sent each time an analysis window is complete.

```json theme={null}
{
  "type": "frame",
  "frame": {
    "start_time_ms": 0,
    "end_time_ms": 4000,
    "verdict": "synthetic",
    "confidence": 0.9732
  }
}
```

| Field           | Type    | Description                                         |
| --------------- | ------- | --------------------------------------------------- |
| `start_time_ms` | integer | Frame start time in the audio stream (ms)           |
| `end_time_ms`   | integer | Frame end time in the audio stream (ms)             |
| `verdict`       | string  | `"synthetic"`, `"non-synthetic"`, or `"no-content"` |
| `confidence`    | float   | Confidence in the verdict, 0.0–1.0                  |

### Done

```json theme={null}
{
  "type": "done",
  "duration_ms": 12500,
  "frame_count": 10
}
```

| Field         | Type    | Description                                          |
| ------------- | ------- | ---------------------------------------------------- |
| `duration_ms` | integer | Total duration of the streamed audio in milliseconds |
| `frame_count` | integer | Total number of frames analyzed                      |

### Error

```json theme={null}
{
  "type": "error",
  "error": "Invalid audio_format='mid'. Valid values: ['3g2', '3ga', ...]"
}
```

## WebSocket close codes

| Code   | Meaning                                                                                  |
| ------ | ---------------------------------------------------------------------------------------- |
| `1000` | Normal closure after a successful `done` message                                         |
| `1003` | Invalid query parameters (bad format, sample rate, or channels)                          |
| `4002` | Audio could not be decoded or does not match the declared format                         |
| `4003` | Request could not be validated, or is not permitted (auth failure, missing model access) |
| `4029` | Insufficient credits, or concurrent-connection limit exceeded                            |

An `error` JSON message is sent before the connection closes (except on `1000`).

## Rate limits

* Concurrent connection limits apply per model.
* Connections that exceed limits are rejected during the WebSocket handshake with close code `4029`.

## Related

* [Deepfake Detection](/get-started/deepfake) — windowing, silence trimming, and scoring explained
* [Deepfake Detection](/get-started/deepfake) — format options for the streaming endpoint
* [Which API should I use?](/guides/which-api) — batch vs streaming tradeoffs
* [Authentication and rate limits](/guides/authentication)


# Velma Batch
Source: https://docs.modulate.ai/api-reference/velma/batch

api/velma_2_batch.yaml POST /api/velma-2-batch
Run full conversation analysis on an uploaded audio file — transcription, conversation type, participant roles, behaviors, topics, sentiment, and summary in one response.



# Velma
Source: https://docs.modulate.ai/api-reference/velma/overview

Velma is an audio-native voice intelligence model over REST or WebSocket — surface behaviors and risks in voice conversations using pre-built or custom detectors.

It analyzes audio signals alongside the words to surface behaviors and risks in voice conversations — fraud, customer churn, compliance violations, and more. Configure it with 150+ pre-built behaviors or define your own in plain language, either the built-in `default` or a JSON `BatchConfig` — and behaviors can be pulled from a catalog of ready-made presets.

|                                                    | Batch                                                   | Streaming                                            |
| -------------------------------------------------- | ------------------------------------------------------- | ---------------------------------------------------- |
| **Use case**                                       | Analyze a complete recording                            | Analyze a live conversation in real time             |
| **Protocol**                                       | HTTP POST (multipart upload)                            | WebSocket                                            |
| **Configuration**                                  | `config` form field — `default` or a JSON `BatchConfig` | First text frame — `default` or a JSON `BatchConfig` |
| **Output**                                         | Full `BatchResponse` after processing                   | Discrete events emitted as results are produced      |
| **Max file size**                                  | 100 MB                                                  | — (streaming)                                        |
| **Transcription + diarization**                    | ✓                                                       | ✓                                                    |
| **Conversation-type & participant-role inference** | ✓                                                       | ✓                                                    |
| **Behavior detection (with presets)**              | ✓                                                       | ✓                                                    |
| **Topics, topic sentiment, summary**               | ✓                                                       | ✓                                                    |

For a side-by-side comparison with the other Modulate capabilities, see [Which API should I use?](/guides/which-api).

## Configuration

Both endpoints take the same configuration: either the literal string `default` to use the built-in configuration, or a JSON `BatchConfig` describing the conversation types, participant roles, behaviors, STT options, and which aggregate outputs (topics, sentiments, summary) to produce. The full `BatchConfig` schema is rendered on the [Batch](/api-reference/velma/batch) reference.

Behaviors can be specified inline or referenced from a catalog of presets using the `preset:<identifier>` syntax. List the available presets with [List behavior presets](/api-reference/velma/presets).

## Authentication

Batch uses the `X-API-Key` header. Streaming uses an `api_key` query parameter at connection time. See [Authentication and rate limits](/guides/authentication).


# List behavior presets
Source: https://docs.modulate.ai/api-reference/velma/presets

api/velma_2_batch.yaml GET /api/velma-2-batch/list-presets
List the catalog of behavior presets that can be referenced from a BatchConfig using the preset:<identifier> syntax.



# Velma Streaming
Source: https://docs.modulate.ai/api-reference/velma/streaming

Real-time conversation analysis over WebSocket — stream audio and receive clips, conversation type, participant roles, behaviors, topics, sentiment, and a summary as they are produced.

Real-time conversation analysis over WebSocket. Stream audio to the server and receive analysis events as they are produced: transcribed clips, the inferred conversation type, participant roles, behavior detections, topics, per-speaker topic sentiment, and a summary — terminating with a final `done` event.

## Endpoint

```text theme={null}
wss://platform.modulate.ai/api/velma-2-streaming
```

## Authentication

Pass your API key as a query parameter when opening the connection.

```text theme={null}
wss://platform.modulate.ai/api/velma-2-streaming?api_key=YOUR_API_KEY
```

<Warning>
  Unlike the batch endpoint, the streaming API does not use an `X-API-Key` header. The key must be in the query string at connection time.
</Warning>

See [Authentication and rate limits](/guides/authentication) for how to obtain and manage API keys.

## Connection parameters

Connection parameters carry only the API key and audio-format hints. All analysis configuration is sent in the [config frame](#configuration), not the query string.

| Parameter      | Type    | Required         | Description                                                                                                                             |
| -------------- | ------- | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `api_key`      | string  | Yes              | Your API key                                                                                                                            |
| `audio_format` | string  | Raw formats only | Audio encoding. Omit for self-describing formats; required for raw/headerless formats. May optionally be set to override auto-detection |
| `sample_rate`  | integer | Raw formats only | Sample rate in Hz. Required for raw formats; must not be set otherwise                                                                  |
| `num_channels` | integer | Raw formats only | Number of channels (1–8). Required for raw formats; must not be set otherwise                                                           |

## Supported audio formats

**Self-describing formats** (auto-detected from file headers — no extra parameters needed):
AAC, AIFF, FLAC, MP3, OGG, WAV, WebM

<Note>
  **OGG / Opus:** OGG is a container that may carry Opus-encoded audio. Pass `audio_format=ogg`, not `audio_format=opus`.
</Note>

**Raw / headerless formats** (require `audio_format`, `sample_rate`, and `num_channels`):
`s8`, `s16le`, `s16be`, `s24le`, `s24be`, `s32le`, `s32be`, `u8`, `u16le`, `u16be`, `u24le`, `u24be`, `u32le`, `u32be`, `f32le`, `f32be`, `f64le`, `f64be`, `mulaw`, `alaw`

Valid sample rates: `8000`, `11025`, `16000`, `22050`, `32000`, `44100`, `48000`, `96000`

## Configuration

After the connection opens, send exactly **one text frame before any audio**:

* The literal string `default` to use the built-in default configuration, **or**
* A JSON-encoded `BatchConfig` describing the conversation types, participant roles, behaviors, STT options, and which aggregate outputs (topics, sentiments, summary) to produce.

The `BatchConfig` schema is identical to the batch endpoint — see the [Batch](/api-reference/velma/batch) reference for the full field list. Behaviors may be referenced from the preset catalog with the `preset:<identifier>` syntax; list available presets with [List behavior presets](/api-reference/velma/presets).

```json theme={null}
{
  "behaviors": ["preset:empathy", "preset:complaints"],
  "stt": { "speaker_diarization": true, "emotion_signal": true },
  "produce_topics": true,
  "produce_topic_sentiments": true,
  "produce_summary": true
}
```

<Warning>
  You must send the config frame before any audio. Sending a binary audio frame before the config frame is a protocol error and closes the connection with code `1003`.
</Warning>

## Connection flow

1. Connect to the WebSocket endpoint with `api_key` (and `audio_format`, `sample_rate`, `num_channels` for raw formats).
2. Send one text frame: either the literal string `default` or a JSON-encoded `BatchConfig`.
3. Stream audio as **binary** WebSocket frames. Frames can be any size.
4. Receive analysis events as JSON text frames as results are produced.
5. Send an **empty text frame** (`""`) to signal end of audio.
6. Receive a final `done` event with the total audio duration.
7. The connection closes automatically.

## Server events

The server emits JSON text frames as results are produced. Every event carries a `type` discriminator. The payload objects (clip, conversation-type pick, participant-role pick, behavior detection, topic sentiment) use the same schemas the batch endpoint returns — see the [Batch](/api-reference/velma/batch) reference for full field definitions. The `partial_clip` and `clip_update` payloads are streaming-only and documented below.

| `type`               | Payload key       | Description                                                                                                            |
| -------------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `clip`               | `clip`            | A transcribed clip with speaker label, timing, and optional emotion / accent / deepfake signals                        |
| `partial_clip`       | `partial_clip`    | An in-progress clip streamed while an utterance is still being spoken, before it finalizes                             |
| `clip_update`        | `clip_update`     | Refined values for a previously finalized clip                                                                         |
| `conversation_type`  | `pick`            | The inferred or default conversation-type classification for the session                                               |
| `participant_role`   | `pick`            | The inferred or default role for a speaker                                                                             |
| `behavior_detection` | `detection`       | A per-behavior detection result                                                                                        |
| `topics`             | `topics`          | The aggregated list of conversation topics; each event fully replaces the previous one                                 |
| `topic_sentiment`    | `topic_sentiment` | Per-speaker sentiment for one aggregated topic; a later event supersedes an earlier one for the same topic and speaker |
| `summary`            | `text`            | A free-form summary of the conversation; each event fully replaces the previous one                                    |
| `done`               | `duration_ms`     | Streaming completed; carries the total audio duration                                                                  |
| `error`              | `error`           | An error occurred; the connection closes after this event                                                              |

### `clip`

A transcribed clip. Emitted progressively as speech is processed.

```json theme={null}
{
  "type": "clip",
  "clip": {
    "clip_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "text": "Thanks for calling support, how can I help?",
    "start_ms": 0,
    "duration_ms": 3200,
    "speaker_label": "speaker_1",
    "language": "en",
    "emotion": null,
    "accent": null,
    "deepfake_score": null
  }
}
```

The `emotion`, `accent`, and `deepfake_score` fields are `null` unless the corresponding STT options are enabled in the config frame.

### `partial_clip`

An in-progress clip streamed while an utterance is still being spoken, before it finalizes. Multiple partials may be emitted for the same `clip_uuid` as the utterance grows; the eventual `clip` event for that utterance reuses the same `clip_uuid`, so a run of partials can be correlated with its final clip.

```json theme={null}
{
  "type": "partial_clip",
  "partial_clip": {
    "clip_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "text": "Thanks for calling support,",
    "start_ms": 0,
    "end_ms": 1800,
    "speaker_label": "speaker_1",
    "emotion": null,
    "accent": null,
    "deepfake_score": null
  }
}
```

`end_ms` is the current end of the in-progress utterance and grows as the utterance extends; it is `null` until available. The finalized clip reports `duration_ms` instead of `end_ms`. `speaker_label` is `null` until diarization resolves the speaker. `emotion`, `accent`, and `deepfake_score` carry the latest in-progress values, or `null` when no value is available yet.

### `clip_update`

Refined values for a previously finalized clip; `clip_update.clip_uuid` matches the `clip_uuid` of an earlier `clip` event. A finalized clip may receive any number of `clip_update` events (including none), each emitted after that clip's `clip` event — possibly interleaved with events for other clips — and always before the `done` event on clean completion. For each field present, the latest received value supersedes the value on the `clip` event and on any earlier `clip_update` for that clip.

```json theme={null}
{
  "type": "clip_update",
  "clip_update": {
    "clip_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "emotion": "Calm",
    "accent": "American"
  }
}
```

### `conversation_type`

The conversation-type classification for the session.

```json theme={null}
{
  "type": "conversation_type",
  "pick": {
    "conversation_type_uuid": "8f1d2c3b-4a5e-6f70-8192-a3b4c5d6e7f8",
    "name": "Customer support call",
    "confidence": 0.92,
    "selection_source": "inferred",
    "detail": "Caller is seeking help resolving a billing issue.",
    "reasoning": "The agent greets the caller and offers assistance with an account problem."
  }
}
```

`selection_source` is one of `inferred`, `auto_selected_single_option`, or `default`.

### `participant_role`

A role assignment for one speaker. Emitted once per identified speaker.

```json theme={null}
{
  "type": "participant_role",
  "pick": {
    "speaker_label": "speaker_1",
    "participant_role_uuid": "2b7c9d10-1e2f-3a4b-5c6d-7e8f90a1b2c3",
    "name": "Support agent",
    "confidence": 0.88,
    "selection_source": "inferred",
    "detail": "Speaker offers assistance and asks diagnostic questions.",
    "reasoning": "Greets on behalf of the company and drives the troubleshooting."
  }
}
```

### `behavior_detection`

A per-behavior detection result. Emitted once per configured behavior.

```json theme={null}
{
  "type": "behavior_detection",
  "detection": {
    "behavior_uuid": "c4d5e6f7-8091-a2b3-c4d5-e6f708192a3b",
    "behavior_name": "Empathy",
    "speaker_label": "speaker_1",
    "detected": true,
    "confidence": 0.81,
    "evidence_clip_uuids": ["a1b2c3d4-e5f6-7890-abcd-ef1234567890"],
    "definitive_clip_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "reasoning": "Agent acknowledges the customer's frustration before resolving the issue."
  }
}
```

### `topics`

The aggregated list of conversation topics. May be emitted more than once as the stream progresses; each event fully replaces the previous `topics` event, so always treat the latest as authoritative and never merge with earlier ones.

```json theme={null}
{
  "type": "topics",
  "topics": ["billing", "refund policy", "account access"]
}
```

### `topic_sentiment`

Per-speaker sentiment for one aggregated topic, keyed by topic and speaker. May be emitted more than once as the stream progresses; a later event supersedes an earlier one for the same topic and speaker.

```json theme={null}
{
  "type": "topic_sentiment",
  "topic_sentiment": {
    "topic": "billing",
    "speaker_label": "speaker_2",
    "sentiment_score": -0.4,
    "sentiment_label": "negative"
  }
}
```

`sentiment_score` ranges from `-1.0` (most negative) to `1.0` (most positive).

### `summary`

A free-form summary of the conversation. May be emitted more than once as the stream progresses; each event fully replaces the previous `summary` event, so always treat the latest as authoritative.

```json theme={null}
{
  "type": "summary",
  "text": "The customer called about a duplicate charge. The agent confirmed the error, issued a refund, and explained the billing cycle."
}
```

### `done`

Sent after all audio has been processed, in response to the end-of-stream signal.

```json theme={null}
{
  "type": "done",
  "duration_ms": 45000
}
```

### `error`

Sent if processing fails. The connection closes after this event.

```json theme={null}
{
  "type": "error",
  "error": "Internal server error"
}
```

## WebSocket close codes

| Code   | Meaning                                                                                                                                         |
| ------ | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `1000` | Normal closure after a successful `done` message                                                                                                |
| `1003` | Protocol error — invalid or incomplete config, audio sent before the config frame, or an unsupported audio format / sample rate / channel count |
| `4003` | Request could not be validated, or is not permitted (auth failure, missing model access)                                                        |
| `4029` | Insufficient credits, or concurrent-connection limit exceeded                                                                                   |

An `error` JSON message is sent before the connection closes (except on `1000`).

## Rate limits

* Concurrent connection limits apply per model.
* Connections that exceed limits are rejected during the WebSocket handshake with close code `4029`.

See [Authentication and rate limits](/guides/authentication) for retry guidance.

## Examples

The examples below send a JSON `BatchConfig` as the config frame. To use the built-in defaults instead, send the literal string `"default"` in place of the JSON.

<Tabs>
  <Tab title="Python (aiohttp)">
    ```python theme={null}
    import asyncio
    import json
    import aiohttp

    API_KEY = "YOUR_API_KEY"
    AUDIO_FILE = "conversation.ogg"
    CHUNK_SIZE = 8192

    # Either the string "default" or a JSON-encoded BatchConfig.
    CONFIG = json.dumps({
        "behaviors": ["preset:empathy", "preset:complaints"],
        "stt": {"speaker_diarization": True, "emotion_signal": True},
        "produce_topics": True,
        "produce_topic_sentiments": True,
        "produce_summary": True,
    })

    async def analyze_streaming():
        url = f"wss://platform.modulate.ai/api/velma-2-streaming?api_key={API_KEY}"

        async with aiohttp.ClientSession() as session:
            async with session.ws_connect(url) as ws:
                # 1. Send the config frame before any audio.
                await ws.send_str(CONFIG)

                async def send_audio():
                    with open(AUDIO_FILE, "rb") as f:
                        while chunk := f.read(CHUNK_SIZE):
                            await ws.send_bytes(chunk)
                            await asyncio.sleep(CHUNK_SIZE / 4000)
                    await ws.send_str("")  # 2. Signal end of audio.

                send_task = asyncio.create_task(send_audio())

                try:
                    async for msg in ws:
                        if msg.type == aiohttp.WSMsgType.TEXT:
                            event = json.loads(msg.data)
                            etype = event["type"]
                            if etype == "clip":
                                c = event["clip"]
                                print(f"[{c['speaker_label']}] {c['text']}")
                            elif etype == "partial_clip":
                                pc = event["partial_clip"]
                                print(f"[partial {pc['clip_uuid'][:8]}] {pc['text']}")
                            elif etype == "clip_update":
                                cu = event["clip_update"]
                                print(f"[update {cu['clip_uuid'][:8]}] emotion={cu.get('emotion')} accent={cu.get('accent')}")
                            elif etype == "conversation_type":
                                print(f"Conversation type: {event['pick']['name']}")
                            elif etype == "participant_role":
                                p = event["pick"]
                                print(f"Role for {p['speaker_label']}: {p['name']}")
                            elif etype == "behavior_detection":
                                d = event["detection"]
                                print(f"Behavior {d['behavior_name']}: detected={d['detected']}")
                            elif etype == "topics":
                                print(f"Topics: {', '.join(event['topics'])}")
                            elif etype == "topic_sentiment":
                                ts = event["topic_sentiment"]
                                print(f"Sentiment ({ts['topic']}, {ts['speaker_label']}): {ts['sentiment_label']}")
                            elif etype == "summary":
                                print(f"Summary: {event['text']}")
                            elif etype == "done":
                                print(f"Done. Duration: {event['duration_ms']}ms")
                                break
                            elif etype == "error":
                                print(f"Error: {event['error']}")
                                break
                        elif msg.type in (
                            aiohttp.WSMsgType.ERROR,
                            aiohttp.WSMsgType.CLOSE,
                            aiohttp.WSMsgType.CLOSED,
                        ):
                            break
                finally:
                    if not send_task.done():
                        send_task.cancel()

    asyncio.run(analyze_streaming())
    ```
  </Tab>

  <Tab title="JavaScript (Node.js)">
    ```javascript theme={null}
    const WebSocket = require("ws");
    const fs = require("fs");

    const API_KEY = "YOUR_API_KEY";
    const AUDIO_FILE = "conversation.ogg";
    const CHUNK_SIZE = 8192;

    // Either the string "default" or a JSON-encoded BatchConfig.
    const CONFIG = JSON.stringify({
      behaviors: ["preset:empathy", "preset:complaints"],
      stt: { speaker_diarization: true, emotion_signal: true },
      produce_topics: true,
      produce_topic_sentiments: true,
      produce_summary: true,
    });

    const url = new URL("wss://platform.modulate.ai/api/velma-2-streaming");
    url.searchParams.set("api_key", API_KEY);

    const ws = new WebSocket(url.toString());

    ws.on("open", () => {
      // 1. Send the config frame before any audio.
      ws.send(CONFIG);

      // 2. Stream audio as binary frames, then signal end of audio.
      const stream = fs.createReadStream(AUDIO_FILE, { highWaterMark: CHUNK_SIZE });
      stream.on("data", (chunk) => ws.send(chunk));
      stream.on("end", () => ws.send(""));
    });

    ws.on("message", (data) => {
      const event = JSON.parse(data.toString());
      switch (event.type) {
        case "clip":
          console.log(`[${event.clip.speaker_label}] ${event.clip.text}`);
          break;
        case "partial_clip":
          console.log(`[partial ${event.partial_clip.clip_uuid.slice(0, 8)}] ${event.partial_clip.text}`);
          break;
        case "clip_update": {
          const cu = event.clip_update;
          console.log(`[update ${cu.clip_uuid.slice(0, 8)}] emotion=${cu.emotion} accent=${cu.accent}`);
          break;
        }
        case "conversation_type":
          console.log(`Conversation type: ${event.pick.name}`);
          break;
        case "participant_role":
          console.log(`Role for ${event.pick.speaker_label}: ${event.pick.name}`);
          break;
        case "behavior_detection":
          console.log(`Behavior ${event.detection.behavior_name}: detected=${event.detection.detected}`);
          break;
        case "topics":
          console.log(`Topics: ${event.topics.join(", ")}`);
          break;
        case "topic_sentiment": {
          const ts = event.topic_sentiment;
          console.log(`Sentiment (${ts.topic}, ${ts.speaker_label}): ${ts.sentiment_label}`);
          break;
        }
        case "summary":
          console.log(`Summary: ${event.text}`);
          break;
        case "done":
          console.log(`Done. Duration: ${event.duration_ms}ms`);
          ws.close();
          break;
        case "error":
          console.error(`Error: ${event.error}`);
          ws.close();
          break;
      }
    });

    ws.on("error", (err) => console.error("WebSocket error:", err.message));
    ```
  </Tab>
</Tabs>

<Note>
  WebSocket APIs cannot be tested with cURL. For command-line testing, use [`websocat`](https://github.com/vi/websocat).
</Note>

## Related

* [Velma overview](/api-reference/velma/overview) — what Velma analyzes and when to use batch vs streaming
* [Velma Batch](/api-reference/velma/batch) — the `BatchConfig` and event payload schemas in full
* [List behavior presets](/api-reference/velma/presets) — discover behavior preset identifiers
* [Authentication and rate limits](/guides/authentication)


# Datasets
Source: https://docs.modulate.ai/benchmarks/datasets

Every public dataset behind Modulate's published benchmark results, the attack family or acoustic condition each one stresses, and the conditions none of them cover.

*Reviewed as of 19 August 2026.*

Modulate's published results are measured on the datasets its benchmarks define. Neither corpus set is chosen by Modulate.

## Speech DF Arena: 14 deepfake datasets

Speech DF Arena scores deepfake detection across 14 evaluation sets spanning synthesis families, languages, and channel conditions.

| Dataset            | Attack family                              | Condition it stresses                                                                                                                                  |
| ------------------ | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| ASVspoof 2019      | Text-to-speech and voice conversion        | Studio-clean synthesis. The long-standing baseline, and the set most detectors are tuned against.                                                      |
| ASVspoof 2021 LA   | Text-to-speech and voice conversion        | The 2019 attacks after transmission through telephony codecs. Separates codec robustness from detection ability.                                       |
| ASVspoof 2021 DF   | Compressed synthetic speech                | Varied lossy encoders and bitrates, as media re-encoding would apply.                                                                                  |
| ASVspoof 2024 Eval | Crowdsourced and adversarial synthesis     | Newer generation methods over non-studio source recordings.                                                                                            |
| Fake or Real       | Commercial text-to-speech                  | Synthesis from deployed commercial systems, across mixed recording conditions.                                                                         |
| Codecfake          | Neural audio codec resynthesis             | Speech reconstructed through a learned codec rather than a vocoder. Genuine speech enters the pipeline, so there are no synthesis artefacts to key on. |
| ADD 2022 Track 1   | Mandarin full-utterance fakes              | Low-quality and noisy Mandarin audio.                                                                                                                  |
| ADD 2022 Track 3   | Adversarial Mandarin fakes                 | Attacks constructed to evade detection rather than to sound natural.                                                                                   |
| ADD 2023 Round 1   | Mandarin deepfake detection                | Second-edition challenge audio, first evaluation round.                                                                                                |
| ADD 2023 Round 2   | Mandarin deepfake detection                | Second evaluation round of the same challenge.                                                                                                         |
| DFADD              | Diffusion and flow-matching text-to-speech | Generation families that postdate most detectors' training data.                                                                                       |
| LibriVoc           | Vocoder artefacts                          | Multiple vocoders over read speech, isolating the vocoder as the only synthetic signal.                                                                |
| SONAR              | Recent end-to-end text-to-speech           | Current-generation systems, including synthesis with no separate vocoder stage.                                                                        |
| In The Wild        | Real-world deepfakes                       | Deepfaked speech of public figures collected from social media. No controlled synthesis pipeline and no matching training distribution.                |

In The Wild is the only set drawn from deepfakes made to deceive rather than to populate a corpus. Codecfake is the only set whose audio starts as genuine speech, which defeats detectors keyed to synthesis artefacts.

Results: [Deepfake Detection](/benchmarks/deepfake-detection).

## Open ASR Leaderboard: 8 transcription datasets

The Open ASR Leaderboard scores English transcription across eight corpora, applying one text normaliser to every entrant.

| Dataset           | Speech type                         | Condition it stresses                                                                                                     |
| ----------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| AMI               | Multi-party meetings                | Far-field microphones, overlapping speech, spontaneous turn-taking. The hardest set on the leaderboard for every entrant. |
| Earnings-22       | Earnings calls                      | Telephony-grade audio, accented English, dense financial vocabulary and named entities.                                   |
| GigaSpeech        | Mixed podcast, audiobook, and video | Broad-domain spontaneous speech across recording qualities.                                                               |
| LibriSpeech clean | Read audiobooks                     | Well-recorded read speech. Near-saturated across entrants.                                                                |
| LibriSpeech other | Read audiobooks                     | The harder speaker split of the same corpus.                                                                              |
| SPGISpeech        | Financial calls                     | Long-form professional speech with heavy domain vocabulary.                                                               |
| TED-LIUM          | Conference talks                    | Prepared single-speaker delivery, varied accents.                                                                         |
| VoxPopuli         | European Parliament proceedings     | Non-native English accents and parliamentary register.                                                                    |

Earnings-22 and VoxPopuli are the two sets closest to contact-centre audio. Modulate reports that pair as a named subset alongside the full leaderboard result.

Results: [Transcription](/benchmarks/transcription).

## What these datasets do not cover

* **8 kHz telephony throughout.** ASVspoof 2021 LA applies telephony codecs and Earnings-22 is telephony-grade, but most sets in both benchmarks are wideband. A pipeline running narrowband PCM end to end is outside the measured distribution.
* **Non-speech audio between speech.** IVR prompts, hold music, ringback, and transfer tones are absent from every set listed here.
* **Streaming.** Both benchmarks score complete files. Neither measures partial-result latency or accuracy on a live socket.
* **Languages beyond the sets listed.** Speech DF Arena covers English and Mandarin. The Open ASR Leaderboard is English only. Neither speaks to the rest of the languages [Multilingual Transcription](/get-started/stt) accepts.
* **Speaker diarization, emotion, accent, and PII/PHI.** No public benchmark in this tab scores them. Modulate evaluates them internally.


# Deepfake Detection results
Source: https://docs.modulate.ai/benchmarks/deepfake-detection

Speech DF Arena results for Modulate's Deepfake Detection model: pooled and average EER across 14 datasets, competitor scores, and deployment characteristics.

*Reviewed as of 19 August 2026.*

[Deepfake Detection](/get-started/deepfake) is ranked 1st on [Speech DF Arena](https://huggingface.co/spaces/Speech-Arena-2025/Speech-DF-Arena), with an average EER of 1.104% across the arena's [14 evaluation datasets](/benchmarks/datasets#speech-df-arena-14-deepfake-datasets).

<Note>
  **Snapshot: 19 August 2026.** Speech DF Arena accepts submissions continuously, so this standing is accurate on that date and not after. The [live leaderboard](https://huggingface.co/spaces/Speech-Arena-2025/Speech-DF-Arena) is authoritative. Modulate's row is listed under a legacy `VELMA-2` identifier rather than the canonical model name.
</Note>

## Scores

Top four systems at the snapshot date. Lower is better in both EER columns.

| System                          | Average EER | Pooled EER | Detection accuracy |
| ------------------------------- | ----------- | ---------- | ------------------ |
| **Modulate Deepfake Detection** | **1.104%**  | **1.1%**   | **98.9%**          |
| Hiya Authenticity Verification  | 2.113%      | 2.324%     | 97.9%              |
| Resemble Detect 3B Omni         | 2.570%      | 2.099%     | 97.4%              |
| Whispeak                        | 3.05%       | 3.00%      | 96.9%              |

Detection accuracy is `100% - average EER`. It restates the first column.

The two EER columns order these systems differently. Resemble Detect 3B Omni posts the second-best pooled EER and the third-best average EER; Hiya is the reverse. Deepfake Detection leads on both, so one threshold holds across all 14 datasets. [Pooled EER and average EER](/benchmarks/methodology#pooled-eer-and-average-eer) covers the distinction.

### At an equal-error operating point

At the equal-error threshold the same rate applies in both error directions. Each figure below is simultaneously the missed synthetic calls per 1,000 synthetic calls and the false alarms per 1,000 genuine calls.

| System                          | Errors per 1,000 calls |
| ------------------------------- | ---------------------- |
| **Modulate Deepfake Detection** | **\~11**               |
| Hiya Authenticity Verification  | \~21                   |
| Resemble Detect 3B Omni         | \~26                   |
| Whispeak                        | \~31                   |

This is arithmetic on the average EER column, not a separate measurement. The next-best system by average EER produces roughly 1.9 times as many errors at this operating point.

<Warning>
  **Worth knowing:** production systems rarely run at the equal-error threshold. Moving the threshold to catch more synthetic calls raises false alarms on genuine ones, and the published EER does not describe that curve. Set the threshold against your own labelled recordings, as [voice fraud screening](/get-started/voice-fraud-screening) describes.
</Warning>

## Deployment characteristics

| Property                    | Modulate Deepfake Detection            | Leaderboard competitors |
| --------------------------- | -------------------------------------- | ----------------------- |
| Parameter count             | 316 million                            | Over 1 billion          |
| Minimum audio for a verdict | 2.5 seconds                            | 5 to 30 seconds         |
| Published price             | \$0.25 per hour                        | $29 to $120 per hour    |
| Streaming support           | Yes, per-frame verdicts over WebSocket | Varies                  |

Minimum audio duration determines whether a model can gate a live authentication flow. Deepfake Detection returns a verdict per 192 ms frame once 2.5 seconds of speech has accumulated.

Competitor prices are list prices at the snapshot date. The arena does not measure them.

## What the arena does not measure

* **Streaming behaviour.** The arena scores complete files. The per-frame latency of [Deepfake Detection streaming](/api-reference/svd/streaming) is not part of this result.
* **Narrowband telephony end to end.** ASVspoof 2021 LA applies telephony codecs. Most arena datasets are wideband.
* **Non-speech audio.** Hold music, IVR prompts, and ringback are absent from every arena dataset. In production those frames return `verdict: "no-content"` and are excluded from scoring.
* **Languages beyond English and Mandarin.**

## Sources

* [Speech DF Arena leaderboard](https://huggingface.co/spaces/Speech-Arena-2025/Speech-DF-Arena), Hugging Face. Live table.
* [Speech DF Arena: A Leaderboard for Speech DeepFake Detection Models](https://arxiv.org/abs/2509.02859), arXiv:2509.02859. Benchmark design, dataset selection, metric definitions.
* [speech\_df\_arena toolkit](https://github.com/Speech-Arena/speech_df_arena), GitHub. Evaluation harness and protocol format.


# Our approach to benchmarking
Source: https://docs.modulate.ai/benchmarks/methodology

How Modulate selects benchmarks and datasets, which metrics it reports, and the failure modes that make a leaderboard score misleading.

*Reviewed as of 19 August 2026.*

Modulate publishes results that a third party can reproduce. Every figure in this tab is measured on a public dataset, by a public harness, against a metric defined before the run.

## Why third-party benchmarks

A vendor-run evaluation selects its own test set, and that selection is not visible in the published number. A leaderboard fixes the datasets, the protocol files, the thresholds, and the scoring code, identically for every entrant.

| Property           | Third-party leaderboard                | Vendor-run evaluation                                     |
| ------------------ | -------------------------------------- | --------------------------------------------------------- |
| Test set selection | Fixed by the maintainer                | Chosen by the vendor                                      |
| Scoring code       | Published, shared across entrants      | Unpublished                                               |
| Competitor scores  | Measured under the same protocol       | Quoted from competitor marketing, or re-run by the vendor |
| Reproducibility    | Any third party can re-run the harness | None                                                      |

Internal evaluation covers conditions no public benchmark measures, notably telephony-bandwidth audio and the conversational domains Modulate's customers run. Results in this tab that are internal rather than third-party say so on the page.

## Selecting datasets

Three properties determine whether a benchmark result is informative:

* **Attack and acoustic coverage.** A deepfake score measured only on text-to-speech does not describe voice conversion, neural codec resynthesis, or diffusion synthesis. A transcription score measured only on read audiobooks does not describe meetings.
* **Out-of-domain datasets.** Speech DF Arena's paper reports many systems posting low error in domain and high error out of domain. Modulate reports per-dataset spread alongside the average for that reason.
* **No overlap with training data.** Modulate does not train on the evaluation splits of the benchmarks it submits to.

[Datasets](/benchmarks/datasets) lists every corpus behind the published results and what each one stresses.

## Metrics

### EER and accuracy are one measurement

Equal Error Rate is the false-positive rate at the threshold where false positives and false negatives are equal. One threshold, one number.

Detection accuracy as quoted on a deepfake leaderboard is `100% - EER`. Modulate's 1.104% average EER and its 98.9% average accuracy are the same result stated two ways, not two pieces of evidence.

### Pooled EER and average EER

Speech DF Arena reports both. They are not interchangeable.

| Metric      | Computation                                                          | Property measured                                                              |
| ----------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| Average EER | Per-dataset EER, then the mean across datasets                       | Consistency. Each dataset gets an equal vote and its own threshold.            |
| Pooled EER  | One global threshold across all scores from every dataset in the run | Deployability. Whether a single production threshold holds everywhere at once. |

The two can order the same systems differently. In the 19 August 2026 snapshot, Resemble Detect 3B Omni posts a better pooled EER than Hiya Authenticity Verification (2.099% against 2.324%) and a worse average EER (2.570% against 2.113%).

A system with a strong average and a weak pooled score is calibrated per dataset but needs a different threshold for each, which a production deployment cannot supply. Modulate reports both on every deepfake result.

### WER

Word Error Rate is the sum of substitutions, insertions, and deletions over reference word count. It weights every word equally. A transcript that renders a sentence correctly except the account number scores better than one that drops two filler words.

WER is sensitive to text normalisation, so cross-vendor comparison is valid only within a single harness. The Open ASR Leaderboard applies one normaliser to every entrant. Figures taken from two vendors' own published numbers are not comparable, and this tab does not mix them.

## Failure modes

* **Benchmark overfitting.** Repeated submission against a fixed public test set converges on that test set. Out-of-domain spread is the check: tight per-dataset results across unseen corpora indicate generalisation, an average carried by two strong datasets does not.
* **Cherry-picked subsets.** An average across a chosen pair of datasets is not a leaderboard average. Subset figures in this tab name their datasets in the same sentence as the number.
* **Stale standings.** A leaderboard position is a snapshot. Standings and their measurement dates are on [Benchmarks](/benchmarks/overview).
* **Size and latency omitted.** A system needing 30 seconds of speech cannot gate a live authentication flow whatever its EER. Results pages report parameter count, minimum audio duration, and price alongside accuracy.
* **Derived comparisons presented as measurements.** Converting an EER into missed detections per thousand calls is arithmetic on a published figure. Pages that do it show the calculation.

## Reproducing a result

<Steps>
  <Step title="Get the harness">
    Speech DF Arena publishes its toolkit at [github.com/Speech-Arena/speech\_df\_arena](https://github.com/Speech-Arena/speech_df_arena). The Open ASR Leaderboard publishes its evaluation code alongside the [leaderboard space](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard).
  </Step>

  <Step title="Assemble the datasets">
    Each benchmark defines its own protocol format. Speech DF Arena expects a `protocol.csv` per dataset carrying absolute file paths and a `spoof` or `bonafide` label per utterance.
  </Step>

  <Step title="Score against the endpoint">
    Call the model per utterance and write its score in the format the harness expects. Both leaderboards score utterance-level output, so the per-frame results from [Deepfake Detection](/get-started/deepfake) need aggregating to one score per file.
  </Step>

  <Step title="Compare against the published figure">
    Report pooled and average EER, or per-dataset and average WER, with the snapshot date of the leaderboard being compared against.
  </Step>
</Steps>

<Note>
  **Worth knowing:** a full harness run makes one API call per evaluation utterance across corpora totalling tens of thousands of files. [Contact support](/support) about evaluation credits before starting one.
</Note>


# Benchmarks
Source: https://docs.modulate.ai/benchmarks/overview

Third-party benchmark results for Modulate's models, the datasets they were measured on, and the methodology behind each number.

*Reviewed as of 19 August 2026.*

Modulate submits its models to public benchmarks and publishes the results here, with the datasets behind them and the methodology used to read them.

Every figure comes from a benchmark run by a third party. Figures derived from a published one rather than measured directly are labelled, with the arithmetic shown.

## Current standings

| Model                                          | Benchmark                                         | Headline metric                                | Standing          | Snapshot       |
| ---------------------------------------------- | ------------------------------------------------- | ---------------------------------------------- | ----------------- | -------------- |
| [Deepfake Detection](/get-started/deepfake)    | [Speech DF Arena](/benchmarks/deepfake-detection) | 1.104% average EER across 14 datasets          | 1st of 18 systems | 19 August 2026 |
| [English Fast Transcription](/get-started/stt) | [Open ASR Leaderboard](/benchmarks/transcription) | 7.80% average WER on Earnings-22 and VoxPopuli | 1st of 88 systems | 19 August 2026 |

## How to read these pages

<CardGroup>
  <Card title="Our approach" icon="compass" href="/benchmarks/methodology">
    Which benchmarks Modulate submits to, how metrics are chosen, and the failure modes that make a good score meaningless.
  </Card>

  <Card title="Datasets" icon="database" href="/benchmarks/datasets">
    Every dataset behind the results, what each one stresses, and what none of them cover.
  </Card>

  <Card title="Results" icon="chart-column" href="/benchmarks/deepfake-detection">
    Per-benchmark results with competitor scores, snapshot dates, and source links.
  </Card>
</CardGroup>


# Transcription results
Source: https://docs.modulate.ai/benchmarks/transcription

Open ASR Leaderboard results for Modulate's English Fast Transcription model, per-dataset WER, and published price per hour across transcription vendors.

*Reviewed as of 19 August 2026.*

[English Fast Transcription](/get-started/stt) is ranked 1st of 88 systems on the [Open ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard), which scores English transcription across [eight corpora](/benchmarks/datasets#open-asr-leaderboard-8-transcription-datasets) under a single text normaliser.

## Word error rate

| Measurement                   | Datasets                  | WER   |
| ----------------------------- | ------------------------- | ----- |
| Contact-centre subset average | Earnings-22 and VoxPopuli | 7.80% |
| Meeting audio                 | AMI                       | 14.9% |

## Published price per hour

List prices as of 19 August 2026. The leaderboard does not measure them.

### Batch

| Provider and model         | Price per hour |
| -------------------------- | -------------- |
| **Modulate transcription** | **\$0.03**     |
| Grok STT                   | \$0.10         |
| AssemblyAI Universal-3 Pro | \$0.21         |
| ElevenLabs Scribe v2       | \$0.22         |
| Speechmatics Enhanced      | \$0.24         |
| Deepgram Nova-3            | \$0.31         |
| OpenAI GPT-4o-transcribe   | \$0.36         |

### Streaming

| Provider and model         | Price per hour |
| -------------------------- | -------------- |
| **Modulate transcription** | **\$0.06**     |
| Grok                       | \$0.20         |
| Speechmatics Enhanced      | \$0.24         |
| Deepgram Nova-3            | \$0.35         |
| OpenAI GPT-4o-transcribe   | \$0.36         |
| ElevenLabs Scribe v2       | \$0.39         |
| AssemblyAI Universal-3-Pro | \$0.45         |

<Note>
  **Worth knowing:** these are prices per audio hour and exclude enrichment flags. `speaker_diarization`, `time_stamps`, and PII/PHI tagging each change cost and latency, as [Transcription](/get-started/stt) documents per flag.
</Note>

## Reading WER against your own requirement

WER counts substitutions, insertions, and deletions equally, with no notion of which words carry the decision. A transcript that renders a sentence correctly except the account number scores better than one that drops two filler words. An application that reads names, amounts, or identifiers out of the transcript needs accuracy measured on those spans.

Normalisation determines comparability. The leaderboard applies one normaliser to every entrant. Figures taken from two vendors' own marketing were normalised differently, and this tab does not mix them.

## Sources

* [Open ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard), Hugging Face. Live table and evaluation code.
* [Earnings-21: A Practical Benchmark for ASR in the Wild](https://arxiv.org/abs/2104.11348), arXiv:2104.11348. Background on the earnings-call corpora.
* [ESB: A Benchmark For Multi-Domain End-to-End Speech Recognition](https://arxiv.org/abs/2210.13352), arXiv:2210.13352. Dataset selection and normalisation rationale.


# FAQ
Source: https://docs.modulate.ai/faq

Frequently asked questions about authentication, models, audio formats, pricing, rate limits, streaming, errors, privacy, and support.

## Authentication and API keys

**How do I get an API key?** [Create a free account](https://platform.modulate.ai/signup-request) and your API key will be available in the dashboard after sign-up.

**How do I authenticate my requests?** Authentication works differently depending on the API type.

For REST endpoints, pass your key in the `X-API-Key` header:

```bash theme={null}
X-API-Key: your_api_key_here
```

For WebSocket endpoints, pass it as a query parameter at connection time:

```text theme={null}
wss://platform.modulate.ai/api/velma-2-stt-streaming?api_key=your_api_key_here
```

**Is it safe to put my API key in code?**

<Warning>
  Never commit credentials to source control. API keys pushed to a repository — even briefly — should be considered compromised. Rotate the key immediately if this happens.
</Warning>

Store your key in a `.env` file, load it at runtime via `python-dotenv` or your environment's secret manager, and ensure `.env` is in your `.gitignore`. The [Quick start](/quickstart) guide covers this setup in full.

## Models and capabilities

**What models are available?**

| Model                                 | API type  | Primary use                                                                   |
| ------------------------------------- | --------- | ----------------------------------------------------------------------------- |
| Multilingual Transcription Batch      | REST      | Transcription with the full feature set                                       |
| Multilingual Transcription Streaming  | WebSocket | Real-time transcription with per-utterance results                            |
| English Fast Transcription Batch      | REST      | High-throughput English transcription at lowest cost                          |
| English Fast Transcription Streaming  | WebSocket | Low-latency English transcription with partials every \~1.5s                  |
| Multilingual Fast Transcription Batch | REST      | Fast multilingual transcription with optional language declaration            |
| Deepfake Detection Batch              | REST      | Analyzing recorded audio files for synthetic speech                           |
| Deepfake Detection Streaming          | WebSocket | Live deepfake detection with results from 500ms onward                        |
| Emotion Detection Batch               | REST      | Classify the emotional tone of an audio file                                  |
| Accent Detection Batch                | REST      | Classify the speaker accent of an audio file                                  |
| PII/PHI Redaction Batch               | REST      | Transcription with PII/PHI text redaction and audio silencing                 |
| PII/PHI Redaction Streaming           | WebSocket | Real-time PII/PHI redaction — redacted transcript and MP3 clips per utterance |
| Music & Speech Detection Batch        | REST      | Classify audio as music, speech, or neither                                   |
| Music & Speech Detection Streaming    | WebSocket | Real-time audio classification as music, speech or neither                    |
| AI Music Detection Batch              | REST      | Detect AI-generated music in audio                                            |
| AI Music Detection Streaming          | WebSocket | Real-time detection of AI-generated music in audio                            |
| Language Detection                    | REST      | Identify the spoken language of an audio file                                 |
| Language Detection                    | REST      | Identify the spoken language of an audio file                                 |

**What languages does transcription support?** The Multilingual Transcription batch and streaming models [support 100 languages](/get-started/language-detection#supported-languages) with automatic language detection. Language is detected per-utterance, so code-switching within a single file is handled automatically. Multilingual Fast Transcription (batch) also transcribes any supported language — declare it with the optional `language` parameter or let it be detected automatically. The English Fast Transcription model is English-only.

**What optional enrichments are available on transcription?** The Multilingual Transcription batch and streaming models support the following optional fields, configurable per request:

| Parameter             | What it adds                                                | Default |
| --------------------- | ----------------------------------------------------------- | ------- |
| `speaker_diarization` | Distinct speaker labels per utterance                       | `true`  |
| `emotion_signal`      | Detected emotional tone (e.g. Neutral, Happy, Frustrated)   | `false` |
| `accent_signal`       | Detected accent (e.g. American, British, Indian)            | `false` |
| `deepfake_signal`     | Deepfake score per utterance (0.0 = human, 1.0 = synthetic) | `false` |
| `pii_phi_tagging`     | PII/PHI wrapped with tags in transcript text                | `false` |

English Fast Transcription (batch) supports two independent opt-in flags, both defaulting to `false`, and none of the other enrichments:

| Parameter             | What it adds                                                                      | Default |
| --------------------- | --------------------------------------------------------------------------------- | ------- |
| `time_stamps`         | A `words` array giving each word a start time, end time, and alignment confidence | `false` |
| `speaker_diarization` | An `utterances` array of speaker-labeled turns with text and timing               | `false` |

`words` times are in seconds, while `utterances` times are in milliseconds; both are measured from the start of the file. See [English Fast Transcription (batch)](/get-started/stt#english-fast-transcription-batch) for the full response shape.

**What's the difference between PII/PHI tagging and PII/PHI redaction?**

These are two separate capabilities with different outputs:

* **PII/PHI tagging** (`pii_phi_tagging=true` on Multilingual Transcription batch or streaming) — identifies PII/PHI spans in the transcript and wraps them with tags. The original text content is preserved. Use this when downstream systems need to detect or handle sensitive spans while retaining the full transcript.
* **PII/PHI redaction** (PII/PHI Redaction APIs) — replaces each detected PII/PHI span with an empty marker tag (e.g. `<pii:name></pii:name>`, `<pii:ssn></pii:ssn>`, `<phi></phi>`) in the transcript **and** silences the corresponding audio ranges in the returned MP3. Use this when the audio itself must be clean — for example, recordings that will be shared, archived, or reviewed by parties who should not hear sensitive information.

**What types of PII/PHI get redacted?** Currently, every entity type the model can detect is redacted, covering both personal information (PII) and health information (PHI). For more detail, see the PII/PHI Redaction Batch reference in the **API Reference** tab.

**Can I configure which PII/PHI tags are enabled?** Not at this time, but it's on our roadmap. [Reach out to Support](/support) to let us know what your specific needs are.

**How accurate is the deepfake detection?** The Deepfake Detection model scores 1.104% average EER across the 14 datasets of the Hugging Face Speech DF Arena benchmark, ranked 1st as of 11 March 2026. Quoted as accuracy that is the same figure restated: 98.9%. [Deepfake Detection results](/benchmarks/deepfake-detection) carries the competitor comparison, both EER metrics, and the live source link.

**What does the deepfake `confidence` score mean?** Confidence represents how certain the model is in its verdict, on a scale of 0 to 1. A frame with `verdict: "synthetic"` and `confidence: 0.97` means the model is highly confident that segment contains AI-generated speech. A `no-content` verdict indicates the frame is silent or contains no usable audio — these frames are not sent through the model and always return `confidence: 1.0`.

## Audio formats and file requirements

**What audio formats are supported?** Most batch models accept AAC, AIFF, FLAC, MOV, MP3, MP4, OGG, Opus, WAV, and WebM. The accepted set differs per endpoint, and each capability page lists its own in full. Music & Speech Detection and AI Music Detection accept a different batch set, and the streaming endpoints each accept their own mix of container and raw formats.

Streaming endpoints split into two groups. Transcription, Redaction, and Velma Triage auto-detect self-describing containers, so `audio_format` is optional. Deepfake Detection, Music & Speech Detection, AI Music Detection, and English Fast Transcription require `audio_format` on every connection, including containers, and omitting it closes the connection with `1003`.

Raw PCM, mu-law, and A-law formats are headerless everywhere they are accepted, so they always require `sample_rate` and `num_channels` alongside `audio_format`.

**Is there a file size limit?** 100 MB for all batch endpoints.

**Is there a minimum audio length?** For deepfake detection, audio must be at least 0.5 seconds. Files shorter than this are rejected with a `422`. For transcription, very short clips may return empty or minimal results.

**What is the recommended audio length for deepfake detection?** 4–60 seconds is the recommended range. Files shorter than one full 4-second analysis window are padded before inference. Leading and trailing silence is trimmed automatically — frame timestamps reflect positions in the original file.

## Pricing and billing

**How is usage billed?** Billing is credit-based and priced per hour of audio processed. Prices range from \$0.01/hour to \$1.25/hour depending on the model. Review pricing for all models here: [platform.modulate.ai/pricing](https://platform.modulate.ai/pricing)

**Is there a free tier?** Yes. Free credits are included when you create an account — no credit card required to get started.

**Where can I monitor my usage?** Real-time usage and billing details are in the [Usage dashboard](https://platform.modulate.ai/dashboard/usage).

## Rate limits

**What rate limits apply?** Concurrency is capped per model: the number of simultaneous in-flight requests or active WebSocket connections against one endpoint. The default is 3. Credits are tracked separately and are not a rate limit.

**What happens when I hit the concurrency cap?** REST endpoints return `429`. WebSocket connections are rejected at the handshake with close code `4030`, or `4029` on endpoints that report every limit condition as one code. The [Usage dashboard](https://platform.modulate.ai/dashboard/usage) shows current usage.

<Tip>
  **Production recommendation:** bound parallel requests with a semaphore sized to the cap. Retrying into a full queue does not help. Caps are per model, so spreading load across models raises total throughput.
</Tip>

**Can the cap be increased?** Contact [support@modulate.ai](mailto:support@modulate.ai) with the model and expected traffic.

## Streaming (WebSocket)

**How do I signal the end of my audio stream?** Send an empty text frame (`""`) on the open connection. The server will drain any buffered audio, deliver outstanding results, send a `done` message, then close the connection cleanly.

**What does the `done` message look like?**

<Accordion title="Multilingual Transcription streaming done">
  ```json theme={null}
  {
    "type": "done",
    "duration_ms": 45000
  }
  ```
</Accordion>

<Accordion title="Deepfake streaming done">
  ```json theme={null}
  {
    "type": "done",
    "duration_ms": 12500,
    "frame_count": 10
  }
  ```
</Accordion>

**What are partial results in Multilingual Transcription streaming?** When `partial_results=true`, the server emits `partial_utterance` messages while speech is in progress, before the utterance is finalized. Each partial replaces the previous one — the final `utterance` message supersedes all preceding partials for that segment. When the emotion, accent, or deepfake signals are enabled, each partial also carries the latest interim `emotion`, `accent`, and `deepfake_score` values (`null` until a value is available). Useful for live caption rendering where low perceived latency matters.

**What WebSocket close codes should I handle?**

<Accordion title="WebSocket close code reference">
  | Code   | Meaning                                                                            |
  | ------ | ---------------------------------------------------------------------------------- |
  | `1000` | Normal closure — received after `done`, connection finished cleanly                |
  | `1003` | Invalid query parameters (bad `audio_format`, `sample_rate`, or `num_channels`)    |
  | `1011` | Internal server error during streaming                                             |
  | `4001` | Invalid API key (Multilingual Transcription streaming)                             |
  | `4002` | Audio could not be decoded or doesn't match the declared format                    |
  | `4003` | Authentication failed or model access not enabled for your organization            |
  | `4029` | Insufficient credits, or a limit condition the endpoint does not report separately |
</Accordion>

## Errors

**What does a `503` response mean?** The inference server is temporarily overloaded. Wait a moment and retry. For production workloads, implement exponential backoff with jitter rather than an immediate retry loop.

**What does a `504` response mean?** The request timed out — batch processing has a 60-second limit. This is uncommon for typical audio lengths. If you see it consistently on files within the recommended size range, contact [support@modulate.ai](mailto:support@modulate.ai).

**My file was rejected with `422`. Why?** The audio is too short for analysis. Deepfake detection requires a minimum of 0.5 seconds. Check the actual duration of your file — empty or near-silent files sometimes report a longer duration than their usable content.

## Privacy and data

**Does Modulate store the audio or outputs?** By default, no data is stored when using our APIs.

**Does Modulate sell the audio I send through the API?** No. Modulate does not sell personal data, including audio submitted through the API. Audio processed via the platform is used solely to deliver the service and, in specific cases, to improve Modulate's models — see the retention and training questions below for details.

**How long is my audio retained after I send it?** Any audio submitted through the platform API *and* is marked for storage by the user is retained for **35 days** from the date of upload, after which it is permanently deleted. **By default, no audio is stored at all.** Enterprise customers with annual commitment agreements can negotiate custom retention periods through their account representative. Self-service customers cannot configure retention periods.

If you need to delete specific audio before the 35-day period expires, you can do so through the platform interface or API.

**Is my audio used to train Modulate's AI models?** It depends on your account type:

* **Self-service (pay-as-you-go) customers** — audio that *is marked for storage* by the user may be used by Modulate to train and improve its models.
* **Enterprise customers (annual commitment)** — participation in model training is optional. To opt out, contact [legal@modulate.ai](mailto:legal@modulate.ai). Opting out does not affect the quality or functionality of your API results.

When audio is used for training, Modulate extracts acoustic and linguistic patterns to improve model performance. Customer audio is never sold or used for purposes unrelated to platform improvement.

**What data does Modulate collect about my API usage?** Modulate collects account identifiers, API usage logs, session data, and any metadata you provide. Usage metadata — such as timestamps, API call counts, and conversation counts — is retained separately from audio for operational and billing purposes.

**Who is responsible for privacy compliance when I use the API to process my users' audio?** When you use the Modulate platform to analyze audio from your own end users, you are the data controller and Modulate acts as a data processor on your behalf. This means you are responsible for:

* Providing required privacy notices to your end users before collecting audio
* Obtaining any necessary consents for recording and analysis
* Establishing a lawful basis for processing under applicable data protection laws
* Complying with audio recording laws in your jurisdiction (wiretapping statutes, consent-to-record requirements, biometric data regulations)
* Responding to your end users' data rights requests (access, deletion, correction)

If your end users submit data rights requests related to audio you processed through the API, those requests should come to you as the data controller. You can then request Modulate's assistance, and Modulate will cooperate with verified requests.

**Does Modulate offer a Data Processing Agreement (DPA)?** Yes. If you are subject to GDPR, UK GDPR, or other regulations requiring a DPA, Modulate offers a standard agreement that includes appropriate data protection terms, security commitments, and Standard Contractual Clauses for international transfers. Contact [legal@modulate.ai](mailto:legal@modulate.ai) to request a DPA.

**Where is my data processed and stored?** Audio and account data are primarily processed and stored in the United States. For customers transferring data from the EU, UK, or other jurisdictions with cross-border transfer requirements, Modulate implements appropriate safeguards including Standard Contractual Clauses.

**How do I exercise my privacy rights or submit a data request?** Contact [privacy@modulate.ai](mailto:privacy@modulate.ai) for access, correction, or deletion requests. Modulate aims to respond within 30 days. Because audio is processed on behalf of platform customers, Modulate may direct end-user requests to the relevant customer (data controller) in some cases.

To opt out of marketing communications, click the unsubscribe link in any Modulate email or contact [privacy@modulate.ai](mailto:privacy@modulate.ai) directly.

<Note>
  This section covers the details most relevant to API users. For complete information — including cookie practices, third-party data sharing, and regional rights — see Modulate's [full Privacy Policy](https://www.modulate.ai/privacy-policy) and the [Velma Services Privacy Addendum](https://www.modulate.ai/privacy-policy#velma-services-privacy-addendum).
</Note>

## Support

**How do I get help?** Check out our [Support](/support) page.

**Where can I learn more about Modulate?** Visit [modulate.ai](https://modulate.ai) to learn about Modulate's mission, or try [Velma Preview](https://preview.modulate.ai) to explore voice analysis in the browser without writing any code.


# Accent Detection
Source: https://docs.modulate.ai/get-started/accent

Classify the speaker accent of an audio file as a whole-file label plus a per-window time series, in one synchronous call.

Accent Detection classifies the regional or national accent of the speech from the voice signal. Classification is acoustic, based on how the speech sounds rather than the words used.

The endpoint produces no transcript, diarization, or enrichment data.

## Accent Detection (batch)

Returns `application/json`.

| Field                       | Type    | Contents                                                                                                                 |
| --------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------ |
| `accent`                    | string  | The label for the whole file. Always present, never null, at any file length.                                            |
| `time_series`               | array   | Consecutive fixed-length windows over the file, each with its own label. Empty when the file is shorter than one window. |
| `time_series[].start_ms`    | integer | Window start, in milliseconds from the beginning of the file.                                                            |
| `time_series[].duration_ms` | integer | Window length.                                                                                                           |
| `time_series[].accent`      | string  | The label for that window.                                                                                               |

There is no top-level duration field. Derive audio length from the last window if it is needed.

### Try it

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://platform.modulate.ai/api/velma-2-accent-batch \
    -H "X-API-Key: $MODULATE_API_KEY" \
    -F "upload_file=@audio.mp3"
  ```

  ```python Python theme={null}
  import os, requests

  response = requests.post(
      "https://platform.modulate.ai/api/velma-2-accent-batch",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
      files={"upload_file": open("audio.mp3", "rb")},
  )
  response.raise_for_status()
  result = response.json()

  print(f"Overall: {result['accent']}")
  for window in result["time_series"]:
      end = window["start_ms"] + window["duration_ms"]
      print(f"  {window['start_ms']}-{end} ms: {window['accent']}")
  ```

  ```javascript JavaScript theme={null}
  import fs from "fs";
  import FormData from "form-data";

  const form = new FormData();
  form.append("upload_file", fs.createReadStream("audio.mp3"), { filename: "audio.mp3" });

  const response = await fetch(
    "https://platform.modulate.ai/api/velma-2-accent-batch",
    {
      method: "POST",
      headers: { "X-API-Key": process.env.MODULATE_API_KEY, ...form.getHeaders() },
      body: form,
    }
  );

  const result = await response.json();
  console.log(`Overall: ${result.accent}`);
  for (const w of result.time_series) {
    console.log(`  ${w.start_ms}-${w.start_ms + w.duration_ms} ms: ${w.accent}`);
  }
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "accent": "British",
    "time_series": [
      { "start_ms": 0, "duration_ms": 15000, "accent": "British" },
      { "start_ms": 15000, "duration_ms": 15000, "accent": "American" }
    ]
  }
  ```
</Accordion>

### What you can configure

| Form field           | Default    | Effect                                                                                    |
| -------------------- | ---------- | ----------------------------------------------------------------------------------------- |
| `upload_file`        | *required* | The audio to analyze.                                                                     |
| `use_ensemble`       | `false`    | Set `true` for a slower, more thorough analysis. Labels may change and latency increases. |
| `training_permitted` | `true`     | Set `false` to exclude this request's audio from model improvement.                       |

### Audio formats

Accepted extensions: `.aac`, `.aiff`, `.flac`, `.mov`, `.mp3`, `.mp4`, `.ogg`, `.opus`, `.wav`, `.webm`.

Maximum file size is 100 MB. Empty files are rejected with `400`.

### Labels

`American`, `British`, `Australian`, `Southern`, `Indian`, `Irish`, `Scottish`, `Eastern_European`, `African`, `Asian`, `Latin_American`, `Middle_Eastern`, `Unknown`.

The same set is used by the `accent_signal` enrichment on Multilingual Transcription.

### Working with the time series

Windows are consecutive and cover the file from the start. A trailing remainder shorter than one full window is dropped. A file shorter than a single window returns an empty `time_series`, but `accent` is still present, so the whole-file label is safe to rely on at any length.

Results for a speaker with a consistent accent are typically stable across windows, and vary more on short or acoustically difficult segments.

```python theme={null}
result = response.json()

overall = result["accent"]

if result["time_series"]:
    shifts = [w["accent"] for w in result["time_series"]]
    print(f"{overall} overall; window by window: {shifts}")
else:
    print(f"{overall} (file shorter than one window)")
```

## Accent alongside a transcript

Multilingual Transcription accepts `accent_signal=true`, which attaches an `accent` label to every utterance from the same label set, in the same call as the transcript. Use that when a transcript is also needed. This endpoint exists for whole-file classification with no transcript. See [Transcription](/get-started/stt).

## API reference

* [Accent Detection Batch](/api-reference/accent/batch)


# AI Music Detection
Source: https://docs.modulate.ai/get-started/ai-music-detection

Detect AI-generated music, as a clip-level verdict with per-window vocal and instrumental breakdowns. Batch and streaming.

AI Music Detection determines whether music was generated by a model. Vocals and instrumentals are scored separately, because a track can pair an AI voice with a human backing track or the reverse, and a single number would hide that.

This is not [Music & Speech Detection](/get-started/music-detection), which separates music from speech. That model answers what kind of sound is present. This one answers where the music came from.

|          | Batch                                   | Streaming                                         |
| -------- | --------------------------------------- | ------------------------------------------------- |
| Protocol | HTTP POST                               | WebSocket                                         |
| Input    | A complete file                         | Live audio frames                                 |
| Output   | Clip verdict plus every window, at once | Windows as they are scored, then the clip verdict |
| Use case | Screening uploads, auditing a catalog   | Monitoring a live broadcast or user stream        |

## Reading the fields

Two families of field look alike and mean different things.

| Suffix           | Meaning                                                  |
| ---------------- | -------------------------------------------------------- |
| `_percentage`    | How much of the audio **contains** that kind of content. |
| `_ai_percentage` | How much of it reads as **AI-generated**.                |

A track can be 90% vocal and 5% AI-vocal. The first is about content, the second about origin.

Vocal and instrumental are scored independently per window rather than routed to one or the other. A window with enough vocal content receives `vocal_ai_probability` and `vocal_ai_confidence`. A window that is not mostly silence receives `instrumental_ai_probability` and `instrumental_ai_confidence`. Either pair is `null` when the window lacks enough of that content to score, so a window can carry both pairs, one, or neither.

<Warning>
  Guard for `null` on `vocal_ai_probability`, `vocal_ai_confidence`, `instrumental_ai_probability`, and `instrumental_ai_confidence` before formatting them. Any per-window renderer that assumes a number will print garbage on a silent or instrumental-only window.
</Warning>

## AI Music Detection (batch)

Returns `application/json`. Every field is always present, though the four per-window AI fields can be `null`.

| Field                        | Type   | Contents                                                                                                                                                                                |
| ---------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `filename`                   | string | The submitted filename, or an empty string if none was sent.                                                                                                                            |
| `duration_s`                 | number | Total duration in **seconds**, not milliseconds.                                                                                                                                        |
| `primary_verdict`            | string | `ai-vocal-music`, `ai-instrumental`, or `not-ai-music`.                                                                                                                                 |
| `vocal_percentage`           | number | Clip-level average share of vocal content, 0 to 100.                                                                                                                                    |
| `vocal_ai_percentage`        | number | Clip-level AI-vocal score, 0 to 100. Each scored window's probability-weighted duration as a share of the full clip. Unscored windows contribute zero but still count toward the total. |
| `vocal_ai_confidence`        | number | Average confidence across windows scored for AI vocals. Not diluted by unscored windows.                                                                                                |
| `instrumental_percentage`    | number | Clip-level average share of instrumental content, 0 to 100.                                                                                                                             |
| `instrumental_ai_percentage` | number | Clip-level AI-instrumental score, 0 to 100, duration-weighted. Mostly-silent windows do not contribute.                                                                                 |
| `instrumental_ai_confidence` | number | Average confidence across windows scored for AI instrumentals. Zero if none were scored.                                                                                                |
| `silence_percentage`         | number | Clip-level average share containing neither vocal nor instrumental content.                                                                                                             |
| `latency_ms`                 | number | Server-side inference time.                                                                                                                                                             |
| `windows`                    | array  | Per-window breakdown, each covering 4 seconds.                                                                                                                                          |

| `primary_verdict` | Meaning                                                                                  |
| ----------------- | ---------------------------------------------------------------------------------------- |
| `ai-vocal-music`  | AI-generated music with a detected synthetic voice. Covers AI songs and AI vocal tracks. |
| `ai-instrumental` | AI-generated instrumental music with no detectable synthetic voice.                      |
| `not-ai-music`    | The clip does not appear to contain AI-generated music.                                  |

Each entry in `windows` carries `start_time_ms`, `end_time_ms`, `vocal_percentage`, `vocal_ai_probability`, `vocal_ai_confidence`, `instrumental_percentage`, `instrumental_ai_probability`, `instrumental_ai_confidence`, and `silence_percentage`.

### Try it

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://platform.modulate.ai/api/velma-2-ai-music-detection-batch \
    -H "X-API-Key: $MODULATE_API_KEY" \
    -F "upload_file=@audio.mp3"
  ```

  ```python Python theme={null}
  import os, requests

  response = requests.post(
      "https://platform.modulate.ai/api/velma-2-ai-music-detection-batch",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
      files={"upload_file": open("audio.mp3", "rb")},
  )
  response.raise_for_status()
  result = response.json()

  print(f"Verdict: {result['primary_verdict']}")
  print(f"Vocal AI: {result['vocal_ai_percentage']}% "
        f"({result['vocal_ai_confidence']:.0%} confidence)")
  print(f"Instrumental AI: {result['instrumental_ai_percentage']}% "
        f"({result['instrumental_ai_confidence']:.0%} confidence)")

  def pct(value):
      return "n/a" if value is None else f"{value:.2f}"

  for w in result["windows"]:
      print(f"  {w['start_time_ms']}ms - {w['end_time_ms']}ms  "
            f"vocal_ai={pct(w['vocal_ai_probability'])}  "
            f"instr_ai={pct(w['instrumental_ai_probability'])}")
  ```

  ```javascript JavaScript theme={null}
  import fs from "fs";
  import FormData from "form-data";

  const form = new FormData();
  form.append("upload_file", fs.createReadStream("audio.mp3"), { filename: "audio.mp3" });

  const response = await fetch(
    "https://platform.modulate.ai/api/velma-2-ai-music-detection-batch",
    {
      method: "POST",
      headers: { "X-API-Key": process.env.MODULATE_API_KEY, ...form.getHeaders() },
      body: form,
    }
  );

  const result = await response.json();
  const pct = (v) => (v == null ? "n/a" : v.toFixed(2));

  console.log(`Verdict: ${result.primary_verdict}`);
  for (const w of result.windows) {
    console.log(
      `${w.start_time_ms}ms - ${w.end_time_ms}ms  ` +
      `vocal_ai=${pct(w.vocal_ai_probability)}  instr_ai=${pct(w.instrumental_ai_probability)}`
    );
  }
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "filename": "my_audio.mp3",
    "duration_s": 89.28,
    "primary_verdict": "ai-vocal-music",
    "vocal_percentage": 87.5,
    "vocal_ai_percentage": 56.5,
    "vocal_ai_confidence": 0.96,
    "instrumental_percentage": 64.3,
    "instrumental_ai_percentage": 10.5,
    "instrumental_ai_confidence": 0.95,
    "silence_percentage": 3.51,
    "latency_ms": 1333.0,
    "windows": [
      {
        "start_time_ms": 0,
        "end_time_ms": 4000,
        "vocal_percentage": 100.0,
        "vocal_ai_probability": 0.97,
        "vocal_ai_confidence": 0.97,
        "instrumental_percentage": 79.0,
        "instrumental_ai_probability": 0.42,
        "instrumental_ai_confidence": 0.16,
        "silence_percentage": 0.0
      },
      {
        "start_time_ms": 4000,
        "end_time_ms": 8000,
        "vocal_percentage": 0.0,
        "vocal_ai_probability": null,
        "vocal_ai_confidence": null,
        "instrumental_percentage": 82.0,
        "instrumental_ai_probability": 0.88,
        "instrumental_ai_confidence": 0.76,
        "silence_percentage": 18.0
      }
    ]
  }
  ```
</Accordion>

### What you can configure

| Form field    | Default    | Effect                                                         |
| ------------- | ---------- | -------------------------------------------------------------- |
| `upload_file` | *required* | The audio to analyze. This endpoint takes no other parameters. |

### Audio formats

Accepted extensions include `.mp3`, `.wav`, `.flac`, `.m4a`, `.mp4`, `.ogg`, `.opus`, `.webm`, `.aac`, `.aiff`, and `.mov`, plus 86 others.

<Accordion title="All 97 accepted extensions">
  `.3g2`, `.3ga`, `.3gp`, `.3gpp`, `.8svx`, `.aa3`, `.aac`, `.ac3`, `.act`, `.adts`, `.aif`,
  `.aifc`, `.aiff`, `.alac`, `.amb`, `.amr`, `.ape`, `.asf`, `.at3`, `.au`, `.avi`, `.avr`,
  `.awb`, `.bwf`, `.c2`, `.caf`, `.dss`, `.dts`, `.dtshd`, `.eac3`, `.ec3`, `.f4a`, `.f4b`,
  `.flac`, `.flv`, `.gsm`, `.iff`, `.m2a`, `.m2ts`, `.m4a`, `.m4b`, `.m4r`, `.m4v`, `.mka`,
  `.mkv`, `.mlp`, `.mmf`, `.mov`, `.mp+`, `.mp1`, `.mp2`, `.mp3`, `.mp4`, `.mpa`, `.mpc`, `.mpeg`,
  `.mpg`, `.mpga`, `.mpp`, `.mts`, `.mxf`, `.nist`, `.oga`, `.ogg`, `.ogx`, `.oma`, `.omg`,
  `.opus`, `.paf`, `.pvf`, `.qcp`, `.ra`, `.rf64`, `.rka`, `.rm`, `.rmvb`, `.sf`, `.shn`, `.snd`,
  `.sph`, `.spx`, `.svx`, `.tak`, `.thd`, `.ts`, `.tta`, `.vob`, `.voc`, `.vqf`, `.w64`, `.wav`,
  `.wave`, `.weba`, `.webm`, `.wma`, `.wmv`, `.wv`
</Accordion>

Maximum file size is 100 MB. Files above it are rejected with `413`. Empty files are rejected with `400`.

### Accuracy

Use `primary_verdict` when judging a whole song or segment. Per-window results are lower accuracy by construction, each scored from four seconds of context.

Heavily processed or high-production tracks are sometimes labelled AI-generated. This is a known gap targeted by future model updates.

## AI Music Detection (streaming)

One `window` message per completed 4-second window, then a `done` message with the clip verdict and percentages.

| Message  | Payload                                                                                                                             |
| -------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `window` | `window` object, same shape as a batch window entry.                                                                                |
| `done`   | `duration_ms`, `window_count`, `primary_verdict`, and the six clip-level percentages and confidences. The server closes after this. |
| `error`  | `error` string. The server closes after this.                                                                                       |

The `done` message is not a summary of the windows already sent. Its instrumental score is recomputed at end of stream from the full accumulated audio, which is more context than any single window receives, so it can differ from what the live windows suggested and is the more reliable number.

Send audio as binary WebSocket frames of any size. Send an empty text frame (`""`) to end the stream. The server then flushes remaining audio, emits outstanding windows, runs the final instrumental analysis, sends `done`, and closes.

### Try it

<CodeGroup>
  ```bash websocat theme={null}
  websocat "wss://platform.modulate.ai/api/velma-2-ai-music-detection-streaming?api_key=$MODULATE_API_KEY&audio_format=mp3" \
    --binary - < audio.mp3
  ```

  ```python Python theme={null}
  import os, asyncio, json, websockets

  API_KEY = os.environ["MODULATE_API_KEY"]
  AUDIO_FILE = "audio.mp3"
  CHUNK_SIZE = 65536

  def pct(value):
      return "n/a" if value is None else f"{value:.2f}"

  async def stream():
      url = (
          f"wss://platform.modulate.ai/api/velma-2-ai-music-detection-streaming"
          f"?api_key={API_KEY}&audio_format=mp3"
      )
      async with websockets.connect(url) as ws:
          async def send():
              with open(AUDIO_FILE, "rb") as f:
                  while chunk := f.read(CHUNK_SIZE):
                      await ws.send(chunk)
              await ws.send("")  # end of stream

          async def receive():
              async for message in ws:
                  msg = json.loads(message)
                  if msg["type"] == "window":
                      w = msg["window"]
                      print(f"  {w['start_time_ms']}ms - {w['end_time_ms']}ms  "
                            f"vocal_ai={pct(w['vocal_ai_probability'])}  "
                            f"instr_ai={pct(w['instrumental_ai_probability'])}")
                  elif msg["type"] == "done":
                      print(f"Done: {msg['duration_ms']}ms, {msg['window_count']} windows")
                      print(f"Verdict: {msg['primary_verdict']}  "
                            f"vocal_ai={msg['vocal_ai_percentage']}%  "
                            f"instr_ai={msg['instrumental_ai_percentage']}%")
                      break
                  elif msg["type"] == "error":
                      raise RuntimeError(msg["error"])

          await asyncio.gather(send(), receive())

  asyncio.run(stream())
  ```

  ```javascript JavaScript theme={null}
  import { WebSocket } from "ws";
  import { createReadStream } from "fs";

  const url =
    `wss://platform.modulate.ai/api/velma-2-ai-music-detection-streaming` +
    `?api_key=${process.env.MODULATE_API_KEY}&audio_format=mp3`;

  const ws = new WebSocket(url);
  const pct = (v) => (v == null ? "n/a" : v.toFixed(2));

  ws.on("open", () => {
    const stream = createReadStream("audio.mp3", { highWaterMark: 65536 });
    stream.on("data", (chunk) => ws.send(chunk));
    stream.on("end", () => ws.send(""));
  });

  ws.on("message", (data) => {
    const msg = JSON.parse(data);
    if (msg.type === "window") {
      const w = msg.window;
      console.log(
        `${w.start_time_ms}ms - ${w.end_time_ms}ms  ` +
        `vocal_ai=${pct(w.vocal_ai_probability)}  instr_ai=${pct(w.instrumental_ai_probability)}`
      );
    } else if (msg.type === "done") {
      console.log(`Verdict: ${msg.primary_verdict}`);
      ws.close();
    } else if (msg.type === "error") {
      throw new Error(msg.error);
    }
  });
  ```
</CodeGroup>

<Accordion title="Messages received">
  ```json theme={null}
  { "type": "window", "window": { "start_time_ms": 0, "end_time_ms": 4000, "vocal_percentage": 87.5, "vocal_ai_probability": 0.97, "vocal_ai_confidence": 0.97, "instrumental_percentage": 64.3, "instrumental_ai_probability": 0.42, "instrumental_ai_confidence": 0.16, "silence_percentage": 3.5 } }
  { "type": "window", "window": { "start_time_ms": 4000, "end_time_ms": 8000, "vocal_percentage": 0.0, "vocal_ai_probability": null, "vocal_ai_confidence": null, "instrumental_percentage": 82.0, "instrumental_ai_probability": 0.88, "instrumental_ai_confidence": 0.76, "silence_percentage": 18.0 } }
  { "type": "done", "duration_ms": 89280, "window_count": 22, "primary_verdict": "ai-vocal-music", "vocal_percentage": 87.5, "vocal_ai_percentage": 56.5, "vocal_ai_confidence": 0.96, "instrumental_percentage": 64.3, "instrumental_ai_percentage": 42.0, "instrumental_ai_confidence": 0.95, "silence_percentage": 3.51 }
  ```
</Accordion>

WebSocket endpoints cannot be exercised with cURL. For command-line testing use [websocat](https://github.com/vi/websocat).

### What you can configure

| Query parameter | Default            | Effect                                                                                            |
| --------------- | ------------------ | ------------------------------------------------------------------------------------------------- |
| `api_key`       | *required*         | The API key. WebSocket connections authenticate through the query string, not a header.           |
| `audio_format`  | *required*         | How the server decodes the bytes sent. Required on every connection, including container formats. |
| `sample_rate`   | *required for raw* | Sample rate in Hz. Required for raw formats. Must not be sent for container formats.              |
| `num_channels`  | *required for raw* | 1 to 8. Required for raw formats. Must not be sent for container formats.                         |

### Audio formats

<Warning>
  `audio_format` is required on every connection, including self-describing containers. Omitting it closes the connection with code `1003`.
</Warning>

**Container formats.** Accepted values include `mp3`, `wav`, `flac`, `m4a`, `mp4`, `ogg`, `opus`, `webm`, `aac`, `aiff`, `wma`, `amr`, and `au`, plus 67 others. The stream header carries sample rate and channel count, so `sample_rate` and `num_channels` are ignored if supplied.

<Accordion title="All 80 accepted container values">
  `3g2`, `3ga`, `3gp`, `3gpp`, `8svx`, `aa3`, `aac`, `ac3`, `act`, `adts`, `aif`, `aifc`, `aiff`,
  `amb`, `amr`, `asf`, `at3`, `au`, `avr`, `awb`, `bwf`, `c2`, `caf`, `dss`, `dts`, `dtshd`,
  `eac3`, `ec3`, `f4a`, `f4b`, `flac`, `gsm`, `iff`, `m2a`, `m2ts`, `m4a`, `m4b`, `m4r`, `m4v`,
  `mka`, `mkv`, `mlp`, `mp+`, `mp1`, `mp2`, `mp3`, `mp4`, `mpa`, `mpc`, `mpga`, `mpp`, `mts`,
  `oga`, `ogg`, `ogx`, `oma`, `omg`, `opus`, `paf`, `pvf`, `qcp`, `ra`, `rf64`, `rm`, `rmvb`,
  `snd`, `spx`, `svx`, `thd`, `ts`, `tta`, `voc`, `vqf`, `w64`, `wav`, `wave`, `weba`, `webm`,
  `wma`, `wmv`
</Accordion>

The MP4-family values (`mp4`, `m4a`, `m4b`, `m4r`, `m4v`, `3gp`, `3gpp`, `3ga`, `3g2`, `f4a`, `f4b`) must be sent in a streamable layout. Anything else ends the connection with an audio-processing error.

**Raw formats.** `s8`, `s16le`, `s16be`, `s24le`, `s24be`, `s32le`, `s32be`, `u8`, `u16le`, `u16be`, `u24le`, `u24be`, `u32le`, `u32be`, `f32le`, `f32be`, `f64le`, `f64be`, `mulaw`, `alaw`, `g722`, `vox`. These are headerless, so `sample_rate` and `num_channels` are both required. `g722` and `vox` are mono-only: `num_channels` must be `1`.

`sample_rate` accepts `8000`, `11025`, `16000`, `22050`, `32000`, `44100`, `48000`, `96000`.

To convert a file to raw PCM:

```bash theme={null}
ffmpeg -i audio.mp3 -ar 16000 -ac 1 -f s16le audio.raw
```

## API reference

* [AI Music Detection Batch](/api-reference/ai-music-detection/batch)
* [AI Music Detection Streaming](/api-reference/ai-music-detection/streaming)


# Audio Event Detection
Source: https://docs.modulate.ai/get-started/audio-event-detection

Detect non-speech sound events in an audio file. Returns a probability for every supported event in one synchronous call.

Audio Event Detection identifies non-speech sounds in a recording: instruments, human vocalizations such as laughter and coughing, and environmental noises such as a knock or a gunshot. Every supported event receives a probability on every call, so the response shape does not change with the contents of the audio.

The endpoint produces no transcript, diarization, or PII/PHI tagging.

## Audio Event Detection (batch)

Returns `application/json`.

| Field         | Type    | Contents                                                                                        |
| ------------- | ------- | ----------------------------------------------------------------------------------------------- |
| `probs`       | object  | A probability between 0 and 1 for each of the 42 supported events. Every key is always present. |
| `probs.cry`   | number  | Probability that the audio contains crying. Scored independently of every other key.            |
| `duration_ms` | integer | Length of the processed audio, in milliseconds.                                                 |

<Warning>
  **Worth knowing:** `cry` and the other 41 events are not on the same scale. `cry` is an independent probability, while the other 41 keys are drawn from one shared distribution that sums to 1 across them. Comparing `cry` against `Laughter` compares two different quantities.
</Warning>

Because the 41 shared keys sum to 1, they rank the most prominent event rather than reporting independent detections. A clip containing both applause and laughter splits probability between `Applause` and `Laughter`, so neither reaches the value it would reach alone. Read those keys by taking the highest rather than by thresholding each one. `cry` is independent, so a threshold on `cry` is meaningful.

Key names are case-sensitive and inconsistent in form: `cry` is the only lowercase key, `Hi-hat` is the only key containing a hyphen, and the rest are capitalized with underscores.

### Try it

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://platform.modulate.ai/api/velma-2-audio-event-classifier \
    -H "X-API-Key: $MODULATE_API_KEY" \
    -F "upload_file=@audio.mp3"
  ```

  ```python Python theme={null}
  import os, requests

  response = requests.post(
      "https://platform.modulate.ai/api/velma-2-audio-event-classifier",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
      files={"upload_file": open("audio.mp3", "rb")},
  )
  response.raise_for_status()
  result = response.json()

  probs = result["probs"]

  # The shared keys sum to 1, so rank them rather than thresholding each one.
  shared = {event: p for event, p in probs.items() if event != "cry"}
  top_event, top_prob = max(shared.items(), key=lambda item: item[1])

  print(f"{result['duration_ms']} ms")
  print(f"most prominent event: {top_event} ({top_prob:.3f})")
  print(f"crying: {probs['cry']:.3f}")
  ```

  ```javascript JavaScript theme={null}
  import fs from "fs";
  import FormData from "form-data";

  const form = new FormData();
  form.append("upload_file", fs.createReadStream("audio.mp3"), { filename: "audio.mp3" });

  const response = await fetch(
    "https://platform.modulate.ai/api/velma-2-audio-event-classifier",
    {
      method: "POST",
      headers: { "X-API-Key": process.env.MODULATE_API_KEY, ...form.getHeaders() },
      body: form,
    }
  );

  const result = await response.json();
  const { cry, ...shared } = result.probs;

  // The shared keys sum to 1, so rank them rather than thresholding each one.
  const [topEvent, topProb] = Object.entries(shared).sort((a, b) => b[1] - a[1])[0];

  console.log(`${result.duration_ms} ms`);
  console.log(`most prominent event: ${topEvent} (${topProb.toFixed(3)})`);
  console.log(`crying: ${cry.toFixed(3)}`);
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "probs": {
      "cry": 0.01,
      "Acoustic_guitar": 0.002,
      "Applause": 0.03,
      "Bark": 0.001,
      "Bass_drum": 0.001,
      "Burping_or_eructation": 0.002,
      "Bus": 0.001,
      "Cello": 0.001,
      "Chime": 0.001,
      "Clarinet": 0.001,
      "Computer_keyboard": 0.001,
      "Cough": 0.01,
      "Cowbell": 0.001,
      "Double_bass": 0.001,
      "Drawer_open_or_close": 0.001,
      "Electric_piano": 0.001,
      "Fart": 0.002,
      "Finger_snapping": 0.005,
      "Fireworks": 0.001,
      "Flute": 0.001,
      "Glockenspiel": 0.001,
      "Gong": 0.001,
      "Gunshot_or_gunfire": 0.001,
      "Harmonica": 0.001,
      "Hi-hat": 0.001,
      "Keys_jangling": 0.001,
      "Knock": 0.002,
      "Laughter": 0.913,
      "Meow": 0.001,
      "Microwave_oven": 0.001,
      "Oboe": 0.001,
      "Saxophone": 0.001,
      "Scissors": 0.001,
      "Shatter": 0.001,
      "Snare_drum": 0.001,
      "Squeak": 0.001,
      "Tambourine": 0.001,
      "Tearing": 0.001,
      "Telephone": 0.002,
      "Trumpet": 0.001,
      "Violin_or_fiddle": 0.001,
      "Writing": 0.001
    },
    "duration_ms": 2000
  }
  ```
</Accordion>

### What you can configure

| Form field    | Default    | Effect                |
| ------------- | ---------- | --------------------- |
| `upload_file` | *required* | The audio to analyze. |

There are no other parameters. The set of scored events is fixed, and the response always carries all 42 keys.

### Audio formats

Accepted extensions: `.aac`, `.aiff`, `.flac`, `.mov`, `.mp3`, `.mp4`, `.ogg`, `.opus`, `.wav`, `.webm`.

Maximum file size is 100 MB. Empty files are rejected with `400`, as are files whose audio cannot be decoded.

### Supported events

`cry` is scored independently.

These 41 are drawn from one shared distribution:

`Acoustic_guitar`, `Applause`, `Bark`, `Bass_drum`, `Burping_or_eructation`, `Bus`, `Cello`, `Chime`, `Clarinet`, `Computer_keyboard`, `Cough`, `Cowbell`, `Double_bass`, `Drawer_open_or_close`, `Electric_piano`, `Fart`, `Finger_snapping`, `Fireworks`, `Flute`, `Glockenspiel`, `Gong`, `Gunshot_or_gunfire`, `Harmonica`, `Hi-hat`, `Keys_jangling`, `Knock`, `Laughter`, `Meow`, `Microwave_oven`, `Oboe`, `Saxophone`, `Scissors`, `Shatter`, `Snare_drum`, `Squeak`, `Tambourine`, `Tearing`, `Telephone`, `Trumpet`, `Violin_or_fiddle`, `Writing`.

## API reference

* [Audio Event Detection Batch](/api-reference/audio-event-detection/batch)


# Deepfake Detection
Source: https://docs.modulate.ai/get-started/deepfake

Detect synthetic (AI-generated) voice in recorded files or live audio, as per-frame verdicts with confidence scores.

Deepfake Detection classifies segments of single-speaker audio as naturally produced human speech or synthetic speech. Synthetic speech covers text-to-speech systems, voice cloning, and other AI voice generation. Classification is acoustic: the model analyzes how the audio sounds, not the words spoken.

Audio is scored in windows rather than as a whole, so the response locates synthetic speech in time instead of returning one verdict per file.

<Note>
  Synthetic speech is not itself evidence of harm or deception. The model cannot distinguish someone using an accessibility tool to communicate from someone using voice cloning to defraud. Interpret results against the use case.
</Note>

|          | Batch                                               | Streaming                                             |
| -------- | --------------------------------------------------- | ----------------------------------------------------- |
| Protocol | HTTP POST                                           | WebSocket                                             |
| Input    | A complete file                                     | Live audio frames                                     |
| Output   | All frames at once, after processing                | Frames as they are analyzed, then a summary           |
| Use case | Screening recordings, investigating a reported call | Anti-spoofing during a live voice-authentication flow |

## Deepfake Detection (batch)

Returns `application/json`. Every field is always present.

| Field                    | Type           | Contents                                                |
| ------------------------ | -------------- | ------------------------------------------------------- |
| `filename`               | string or null | The uploaded filename.                                  |
| `duration_ms`            | integer        | Total audio duration.                                   |
| `frames`                 | array          | Per-frame results, ordered, covering the full duration. |
| `frames[].start_time_ms` | integer        | Frame start, relative to the original file.             |
| `frames[].end_time_ms`   | integer        | Frame end, relative to the original file.               |
| `frames[].verdict`       | string         | `synthetic`, `non-synthetic`, or `no-content`.          |
| `frames[].confidence`    | number         | 0 to 1, confidence in the stated verdict.               |

| Verdict         | Meaning                                                                            |
| --------------- | ---------------------------------------------------------------------------------- |
| `synthetic`     | The frame likely contains AI-generated speech.                                     |
| `non-synthetic` | The frame likely contains natural human speech.                                    |
| `no-content`    | The frame is silent or has no usable audio. Always returned with confidence `1.0`. |

`confidence` is confidence in the verdict given, not the probability that the audio is synthetic. A frame with `verdict: "non-synthetic"` and `confidence: 0.97` means the model is 97% confident the speech is natural. It does not mean a 3% chance of being synthetic.

### Try it

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://platform.modulate.ai/api/velma-2-synthetic-voice-detection-batch \
    -H "X-API-Key: $MODULATE_API_KEY" \
    -F "upload_file=@audio.mp3"
  ```

  ```python Python theme={null}
  import os, requests

  response = requests.post(
      "https://platform.modulate.ai/api/velma-2-synthetic-voice-detection-batch",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
      files={"upload_file": open("audio.mp3", "rb")},
  )
  response.raise_for_status()
  result = response.json()

  print(f"{result['duration_ms']}ms, {len(result['frames'])} frames")
  for frame in result["frames"]:
      print(f"  {frame['start_time_ms']}ms - {frame['end_time_ms']}ms  "
            f"{frame['verdict']}  ({frame['confidence']:.0%})")
  ```

  ```javascript JavaScript theme={null}
  import fs from "fs";
  import FormData from "form-data";

  const form = new FormData();
  form.append("upload_file", fs.createReadStream("audio.mp3"), { filename: "audio.mp3" });

  const response = await fetch(
    "https://platform.modulate.ai/api/velma-2-synthetic-voice-detection-batch",
    {
      method: "POST",
      headers: { "X-API-Key": process.env.MODULATE_API_KEY, ...form.getHeaders() },
      body: form,
    }
  );

  const result = await response.json();
  for (const f of result.frames) {
    console.log(`${f.start_time_ms}ms - ${f.end_time_ms}ms  ${f.verdict}  ${f.confidence}`);
  }
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "filename": "audio.mp3",
    "duration_ms": 16000,
    "frames": [
      { "start_time_ms": 0,     "end_time_ms": 4000,  "verdict": "non-synthetic", "confidence": 0.94 },
      { "start_time_ms": 4000,  "end_time_ms": 8000,  "verdict": "non-synthetic", "confidence": 0.91 },
      { "start_time_ms": 8000,  "end_time_ms": 12000, "verdict": "synthetic",     "confidence": 0.87 },
      { "start_time_ms": 12000, "end_time_ms": 16000, "verdict": "no-content",    "confidence": 1.00 }
    ]
  }
  ```
</Accordion>

### What you can configure

| Form field    | Default    | Effect                                                         |
| ------------- | ---------- | -------------------------------------------------------------- |
| `upload_file` | *required* | The audio to analyze. This endpoint takes no other parameters. |

### Audio formats

Accepted extensions include `.mp3`, `.wav`, `.flac`, `.m4a`, `.mp4`, `.ogg`, `.opus`, `.webm`, `.aac`, `.aiff`, and `.mov`, plus 86 others.

<Accordion title="All 97 accepted extensions">
  `.3g2`, `.3ga`, `.3gp`, `.3gpp`, `.8svx`, `.aa3`, `.aac`, `.ac3`, `.act`, `.adts`, `.aif`,
  `.aifc`, `.aiff`, `.alac`, `.amb`, `.amr`, `.ape`, `.asf`, `.at3`, `.au`, `.avi`, `.avr`,
  `.awb`, `.bwf`, `.c2`, `.caf`, `.dss`, `.dts`, `.dtshd`, `.eac3`, `.ec3`, `.f4a`, `.f4b`,
  `.flac`, `.flv`, `.gsm`, `.iff`, `.m2a`, `.m2ts`, `.m4a`, `.m4b`, `.m4r`, `.m4v`, `.mka`,
  `.mkv`, `.mlp`, `.mmf`, `.mov`, `.mp+`, `.mp1`, `.mp2`, `.mp3`, `.mp4`, `.mpa`, `.mpc`, `.mpeg`,
  `.mpg`, `.mpga`, `.mpp`, `.mts`, `.mxf`, `.nist`, `.oga`, `.ogg`, `.ogx`, `.oma`, `.omg`,
  `.opus`, `.paf`, `.pvf`, `.qcp`, `.ra`, `.rf64`, `.rka`, `.rm`, `.rmvb`, `.sf`, `.shn`, `.snd`,
  `.sph`, `.spx`, `.svx`, `.tak`, `.thd`, `.ts`, `.tta`, `.vob`, `.voc`, `.vqf`, `.w64`, `.wav`,
  `.wave`, `.weba`, `.webm`, `.wma`, `.wmv`, `.wv`
</Accordion>

Maximum file size is 100 MB. Empty files are rejected with `400`.

### Behavior and constraints

Audio under 0.5 seconds is rejected with `422`. Audio shorter than one 4-second window is accepted and padded before inference.

Each batch frame covers a 4-second window, and the file is windowed from start to finish.

Leading and trailing silence is trimmed before windowing. Frame timestamps still refer to positions in the original file, so `start_time_ms` and `end_time_ms` line up against the source audio with no offset arithmetic.

`no-content` frames are classified before inference rather than by the model, so silence produces an explicit verdict instead of an arbitrary synthetic or non-synthetic guess. Filter them out when aggregating a clip-level verdict.

Recommended clip length is 4 to 60 seconds.

## Deepfake Detection (streaming)

Frames arrive as JSON text messages while audio streams in. A closing `done` message reports total duration and frame count.

| Message | Payload                                                                                                   |
| ------- | --------------------------------------------------------------------------------------------------------- |
| `frame` | `frame` object with `start_time_ms`, `end_time_ms`, `verdict`, `confidence`. Same shape as a batch frame. |
| `done`  | `duration_ms` and `frame_count`. The server closes after this.                                            |
| `error` | `error` string. The server closes after this.                                                             |

Send audio as binary WebSocket frames of any size; the server buffers and windows internally. Send an empty text frame (`""`) to end the stream. The server then flushes remaining audio, delivers final frames, sends `done`, and closes.

### Try it

<CodeGroup>
  ```bash websocat theme={null}
  websocat "wss://platform.modulate.ai/api/velma-2-synthetic-voice-detection-streaming?api_key=$MODULATE_API_KEY&audio_format=s16le&sample_rate=16000&num_channels=1" \
    --binary - < audio.raw
  ```

  ```python Python theme={null}
  import os, asyncio, json, websockets

  API_KEY = os.environ["MODULATE_API_KEY"]
  AUDIO_FILE = "audio.raw"
  CHUNK_SIZE = 8192

  async def stream():
      url = (
          f"wss://platform.modulate.ai/api/velma-2-synthetic-voice-detection-streaming"
          f"?api_key={API_KEY}&audio_format=s16le&sample_rate=16000&num_channels=1"
      )
      async with websockets.connect(url) as ws:
          async def send():
              with open(AUDIO_FILE, "rb") as f:
                  while chunk := f.read(CHUNK_SIZE):
                      await ws.send(chunk)
              await ws.send("")  # end of stream

          async def receive():
              async for message in ws:
                  msg = json.loads(message)
                  if msg["type"] == "frame":
                      f = msg["frame"]
                      print(f"  {f['start_time_ms']}ms - {f['end_time_ms']}ms  "
                            f"{f['verdict']}  ({f['confidence']:.0%})")
                  elif msg["type"] == "done":
                      print(f"Done: {msg['duration_ms']}ms, {msg['frame_count']} frames")
                      break
                  elif msg["type"] == "error":
                      raise RuntimeError(msg["error"])

          await asyncio.gather(send(), receive())

  asyncio.run(stream())
  ```

  ```javascript JavaScript theme={null}
  import { WebSocket } from "ws";
  import { createReadStream } from "fs";

  const url =
    `wss://platform.modulate.ai/api/velma-2-synthetic-voice-detection-streaming` +
    `?api_key=${process.env.MODULATE_API_KEY}` +
    `&audio_format=s16le&sample_rate=16000&num_channels=1`;

  const ws = new WebSocket(url);

  ws.on("open", () => {
    const stream = createReadStream("audio.raw", { highWaterMark: 8192 });
    stream.on("data", (chunk) => ws.send(chunk));
    stream.on("end", () => ws.send(""));
  });

  ws.on("message", (data) => {
    const msg = JSON.parse(data);
    if (msg.type === "frame") {
      const f = msg.frame;
      console.log(`${f.start_time_ms}ms - ${f.end_time_ms}ms  ${f.verdict}  ${f.confidence}`);
    } else if (msg.type === "done") {
      console.log(`Done: ${msg.duration_ms}ms, ${msg.frame_count} frames`);
      ws.close();
    } else if (msg.type === "error") {
      throw new Error(msg.error);
    }
  });
  ```
</CodeGroup>

<Accordion title="Messages received">
  ```json theme={null}
  { "type": "frame", "frame": { "start_time_ms": 0, "end_time_ms": 4000, "verdict": "non-synthetic", "confidence": 0.94 } }
  { "type": "frame", "frame": { "start_time_ms": 4000, "end_time_ms": 8000, "verdict": "synthetic", "confidence": 0.91 } }
  { "type": "done", "duration_ms": 8000, "frame_count": 2 }
  ```
</Accordion>

WebSocket endpoints cannot be exercised with cURL. For command-line testing use [websocat](https://github.com/vi/websocat).

### What you can configure

| Query parameter | Default            | Effect                                                                                                                              |
| --------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------- |
| `api_key`       | *required*         | The API key. WebSocket connections authenticate through the query string, not a header.                                             |
| `audio_format`  | *required*         | How the server decodes the bytes sent. Required on every connection, including container formats.                                   |
| `sample_rate`   | *required for raw* | Sample rate in Hz. Required for raw formats. Must not be sent for container formats.                                                |
| `num_channels`  | *required for raw* | 1 to 8. Required for raw formats. Must not be sent for container formats. Multi-channel audio is downmixed to mono before analysis. |

### Audio formats

<Warning>
  `audio_format` is required on every connection, including self-describing containers. Omitting it closes the connection with code `1003`. This differs from Transcription and Redaction streaming, which auto-detect containers.
</Warning>

**Container formats.** Accepted values include `mp3`, `wav`, `flac`, `m4a`, `mp4`, `ogg`, `opus`, `webm`, `aac`, `aiff`, `wma`, `amr`, and `au`, plus 67 others. The stream header carries sample rate and channel count, so `sample_rate` and `num_channels` are ignored if supplied.

<Accordion title="All 80 accepted container values">
  `3g2`, `3ga`, `3gp`, `3gpp`, `8svx`, `aa3`, `aac`, `ac3`, `act`, `adts`, `aif`, `aifc`, `aiff`,
  `amb`, `amr`, `asf`, `at3`, `au`, `avr`, `awb`, `bwf`, `c2`, `caf`, `dss`, `dts`, `dtshd`,
  `eac3`, `ec3`, `f4a`, `f4b`, `flac`, `gsm`, `iff`, `m2a`, `m2ts`, `m4a`, `m4b`, `m4r`, `m4v`,
  `mka`, `mkv`, `mlp`, `mp+`, `mp1`, `mp2`, `mp3`, `mp4`, `mpa`, `mpc`, `mpga`, `mpp`, `mts`,
  `oga`, `ogg`, `ogx`, `oma`, `omg`, `opus`, `paf`, `pvf`, `qcp`, `ra`, `rf64`, `rm`, `rmvb`,
  `snd`, `spx`, `svx`, `thd`, `ts`, `tta`, `voc`, `vqf`, `w64`, `wav`, `wave`, `weba`, `webm`,
  `wma`, `wmv`
</Accordion>

The MP4-family values (`mp4`, `m4a`, `m4b`, `m4r`, `m4v`, `3gp`, `3gpp`, `3ga`, `3g2`, `f4a`, `f4b`) must be sent in a streamable layout. Anything else ends the connection with an audio-processing error.

**Raw formats.** `s8`, `s16le`, `s16be`, `s24le`, `s24be`, `s32le`, `s32be`, `u8`, `u16le`, `u16be`, `u24le`, `u24be`, `u32le`, `u32be`, `f32le`, `f32be`, `f64le`, `f64be`, `mulaw`, `alaw`, `g722`, `vox`. These are headerless, so `sample_rate` and `num_channels` are both required. `g722` and `vox` are mono-only: `num_channels` must be `1`.

`sample_rate` accepts `8000`, `11025`, `16000`, `22050`, `32000`, `44100`, `48000`, `96000`.

Common raw configurations:

| Source                         | `audio_format` | `sample_rate` | `num_channels` |
| ------------------------------ | -------------- | ------------- | -------------- |
| Native app default             | `s16le`        | `16000`       | `1`            |
| Web Audio API (`AudioWorklet`) | `f32le`        | `48000`       | `1`            |
| Native stereo capture          | `s16le`        | `48000`       | `2`            |
| Telephony (mu-law)             | `mulaw`        | `8000`        | `1`            |
| Telephony (A-law)              | `alaw`         | `8000`        | `1`            |

`s16le` at 16 kHz mono is passed through without conversion. Every other format is decoded and resampled to 16 kHz mono first. Output is identical either way, so this only affects latency. Capture at `s16le` 16 kHz mono where the pipeline allows it.

To convert a file:

```bash theme={null}
ffmpeg -i audio.mp3 -ar 16000 -ac 1 -f s16le audio.raw
```

### Behavior and constraints

The first prediction is emitted once the minimum audio duration has arrived. Each subsequent prediction follows one second later, with the window growing from time zero. Once the window reaches full length it slides forward, holding a constant size. Verdicts therefore arrive incrementally rather than at end of stream.

Format parameters are validated twice: at connection time, which closes with `1003`, and again while decoding audio, which closes with `4002`. The second catches both undecodable audio and a mismatch between the declared `audio_format` and the bytes actually sent.

## Frame-level detection compared with the transcription signal

Multilingual Transcription accepts `deepfake_signal=true`, which adds a `deepfake_score` to each utterance. That is one score per utterance, available only when transcribing, and `null` for utterances under 0.5 seconds.

The endpoints on this page score every frame across the file, return explicit `no-content` verdicts for silence, and work on live audio without producing a transcript. Use `deepfake_signal` when detection is a secondary signal alongside a transcript. Use these endpoints when detection is the goal.

## Screening with behavior analysis

A synthetic verdict does not say whether the caller did anything wrong, and this page's endpoints cannot tell an AI assistant calling on a customer's behalf from a cloned voice running a fraud script. [Screen a call for voice fraud](/get-started/voice-fraud-screening) pairs these endpoints with [Velma Triage](/get-started/velma), aggregates the frame verdicts into one clip-level value, and combines both outputs into a single decision.

## API reference

* [Deepfake Detection Batch](/api-reference/svd/batch)
* [Deepfake Detection Streaming](/api-reference/svd/streaming)


# Emotion Detection
Source: https://docs.modulate.ai/get-started/emotion

Classify the emotional tone of an audio file as a whole-file label plus a per-window time series, in one synchronous call.

Emotion Detection classifies emotional tone from the voice signal. Classification is acoustic: it reflects how something was said, not the words used. Two recordings of identical text can receive different labels if the delivery differs.

The endpoint produces no transcript, diarization, or enrichment data.

## Emotion Detection (batch)

Returns `application/json`.

| Field                       | Type    | Contents                                                                                                                 |
| --------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------ |
| `emotion`                   | string  | The label for the whole file. Always present, never null, at any file length.                                            |
| `time_series`               | array   | Consecutive fixed-length windows over the file, each with its own label. Empty when the file is shorter than one window. |
| `time_series[].start_ms`    | integer | Window start, in milliseconds from the beginning of the file.                                                            |
| `time_series[].duration_ms` | integer | Window length.                                                                                                           |
| `time_series[].emotion`     | string  | The label for that window.                                                                                               |

There is no top-level duration field. Derive audio length from the last window if it is needed.

### Try it

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://platform.modulate.ai/api/velma-2-emotion-batch \
    -H "X-API-Key: $MODULATE_API_KEY" \
    -F "upload_file=@audio.mp3"
  ```

  ```python Python theme={null}
  import os, requests

  response = requests.post(
      "https://platform.modulate.ai/api/velma-2-emotion-batch",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
      files={"upload_file": open("audio.mp3", "rb")},
  )
  response.raise_for_status()
  result = response.json()

  print(f"Overall: {result['emotion']}")
  for window in result["time_series"]:
      end = window["start_ms"] + window["duration_ms"]
      print(f"  {window['start_ms']}-{end} ms: {window['emotion']}")
  ```

  ```javascript JavaScript theme={null}
  import fs from "fs";
  import FormData from "form-data";

  const form = new FormData();
  form.append("upload_file", fs.createReadStream("audio.mp3"), { filename: "audio.mp3" });

  const response = await fetch(
    "https://platform.modulate.ai/api/velma-2-emotion-batch",
    {
      method: "POST",
      headers: { "X-API-Key": process.env.MODULATE_API_KEY, ...form.getHeaders() },
      body: form,
    }
  );

  const result = await response.json();
  console.log(`Overall: ${result.emotion}`);
  for (const w of result.time_series) {
    console.log(`  ${w.start_ms}-${w.start_ms + w.duration_ms} ms: ${w.emotion}`);
  }
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "emotion": "Happy",
    "time_series": [
      { "start_ms": 0, "duration_ms": 15000, "emotion": "Happy" },
      { "start_ms": 15000, "duration_ms": 15000, "emotion": "Neutral" }
    ]
  }
  ```
</Accordion>

### What you can configure

| Form field           | Default    | Effect                                                                                    |
| -------------------- | ---------- | ----------------------------------------------------------------------------------------- |
| `upload_file`        | *required* | The audio to analyze.                                                                     |
| `use_ensemble`       | `false`    | Set `true` for a slower, more thorough analysis. Labels may change and latency increases. |
| `training_permitted` | `true`     | Set `false` to exclude this request's audio from model improvement.                       |

### Audio formats

Accepted extensions: `.aac`, `.aiff`, `.flac`, `.mov`, `.mp3`, `.mp4`, `.ogg`, `.opus`, `.wav`, `.webm`.

Maximum file size is 100 MB. Empty files are rejected with `400`.

### Labels

`Neutral`, `Calm`, `Happy`, `Amused`, `Excited`, `Proud`, `Affectionate`, `Interested`, `Hopeful`, `Frustrated`, `Angry`, `Contemptuous`, `Concerned`, `Afraid`, `Sad`, `Ashamed`, `Bored`, `Tired`, `Surprised`, `Anxious`, `Stressed`, `Disgusted`, `Disappointed`, `Confused`, `Relieved`, `Confident`.

The same set is used by the `emotion_signal` enrichment on Multilingual Transcription.

### Working with the time series

Windows are consecutive and cover the file from the start. A trailing remainder shorter than one full window is dropped. A file shorter than a single window returns an empty `time_series`, but `emotion` is still present, so the whole-file label is safe to rely on at any length.

```python theme={null}
result = response.json()

overall = result["emotion"]

if result["time_series"]:
    shifts = [w["emotion"] for w in result["time_series"]]
    print(f"{overall} overall; window by window: {shifts}")
else:
    print(f"{overall} (file shorter than one window)")
```

## Emotion alongside a transcript

Multilingual Transcription accepts `emotion_signal=true`, which attaches an `emotion` label to every utterance from the same label set, in the same call as the transcript. Use that when a transcript is also needed. This endpoint exists for whole-file classification with no transcript. See [Transcription](/get-started/stt).

## API reference

* [Emotion Detection Batch](/api-reference/emotion/batch)


# Language Detection
Source: https://docs.modulate.ai/get-started/language-detection

Identify the spoken language of an audio file, with a confidence score, across 100 languages in one synchronous call.

Language Detection identifies the language being spoken and scores its confidence in that answer. It produces no transcript, diarization, or enrichment data.

The usual use is routing: read the language code off the response and send the audio to the matching downstream pipeline.

## Language Detection (batch)

Returns `application/json`. Every field is always present.

| Field                     | Type    | Contents                                                                                              |
| ------------------------- | ------- | ----------------------------------------------------------------------------------------------------- |
| `predicted_language`      | string  | Human-readable language name, such as `"English"`. Suitable for display.                              |
| `predicted_language_code` | string  | Lowercase ISO 639-1 code, such as `"en"`. Suitable for routing, locale switching, or BCP 47 tags.     |
| `confidence`              | number  | Probability for the predicted language, 0.0 to 1.0.                                                   |
| `duration_ms`             | integer | Total duration of the decoded audio. Only the first 30 seconds are analyzed regardless of this value. |

### Try it

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://platform.modulate.ai/api/velma-2-language-detection-batch \
    -H "X-API-Key: $MODULATE_API_KEY" \
    -F "upload_file=@audio.mp3"
  ```

  ```python Python theme={null}
  import os, requests

  response = requests.post(
      "https://platform.modulate.ai/api/velma-2-language-detection-batch",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
      files={"upload_file": open("audio.mp3", "rb")},
  )
  response.raise_for_status()
  result = response.json()

  print(f"{result['predicted_language']} ({result['predicted_language_code']}) "
        f"at {result['confidence']:.4f}")
  ```

  ```javascript JavaScript theme={null}
  import fs from "fs";
  import FormData from "form-data";

  const form = new FormData();
  form.append("upload_file", fs.createReadStream("audio.mp3"), { filename: "audio.mp3" });

  const response = await fetch(
    "https://platform.modulate.ai/api/velma-2-language-detection-batch",
    {
      method: "POST",
      headers: { "X-API-Key": process.env.MODULATE_API_KEY, ...form.getHeaders() },
      body: form,
    }
  );

  const result = await response.json();
  console.log(`${result.predicted_language} (${result.predicted_language_code}) at ${result.confidence}`);
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "predicted_language": "English",
    "predicted_language_code": "en",
    "confidence": 0.9847,
    "duration_ms": 14253
  }
  ```
</Accordion>

### What you can configure

| Form field    | Default    | Effect                                                         |
| ------------- | ---------- | -------------------------------------------------------------- |
| `upload_file` | *required* | The audio to analyze. This endpoint takes no other parameters. |

### Audio formats

Accepted extensions include `.mp3`, `.wav`, `.flac`, `.m4a`, `.mp4`, `.ogg`, `.opus`, `.webm`, `.aac`, `.aiff`, and `.mov`, plus 86 others.

<Accordion title="All 97 accepted extensions">
  `.3g2`, `.3ga`, `.3gp`, `.3gpp`, `.8svx`, `.aa3`, `.aac`, `.ac3`, `.act`, `.adts`, `.aif`,
  `.aifc`, `.aiff`, `.alac`, `.amb`, `.amr`, `.ape`, `.asf`, `.at3`, `.au`, `.avi`, `.avr`,
  `.awb`, `.bwf`, `.c2`, `.caf`, `.dss`, `.dts`, `.dtshd`, `.eac3`, `.ec3`, `.f4a`, `.f4b`,
  `.flac`, `.flv`, `.gsm`, `.iff`, `.m2a`, `.m2ts`, `.m4a`, `.m4b`, `.m4r`, `.m4v`, `.mka`,
  `.mkv`, `.mlp`, `.mmf`, `.mov`, `.mp+`, `.mp1`, `.mp2`, `.mp3`, `.mp4`, `.mpa`, `.mpc`, `.mpeg`,
  `.mpg`, `.mpga`, `.mpp`, `.mts`, `.mxf`, `.nist`, `.oga`, `.ogg`, `.ogx`, `.oma`, `.omg`,
  `.opus`, `.paf`, `.pvf`, `.qcp`, `.ra`, `.rf64`, `.rka`, `.rm`, `.rmvb`, `.sf`, `.shn`, `.snd`,
  `.sph`, `.spx`, `.svx`, `.tak`, `.thd`, `.ts`, `.tta`, `.vob`, `.voc`, `.vqf`, `.w64`, `.wav`,
  `.wave`, `.weba`, `.webm`, `.wma`, `.wmv`, `.wv`
</Accordion>

Maximum file size is 100 MB. Empty files are rejected with `400`.

### Behavior and constraints

Only the first 30 seconds of audio are analyzed. Longer files are accepted and the remainder is ignored, so `duration_ms` can exceed the analyzed span. Aim for at least 3 to 5 seconds of clear speech at the start.

Audio in a language outside the supported set returns the closest supported match, usually with low `confidence`.

## Acting on the confidence score

`confidence` near `1.0` means the model committed to an answer. Near `0.0` means it could not. There is no universally correct threshold; set one against how much misclassification the application can absorb.

```python theme={null}
result = response.json()

if result["confidence"] < 0.5:
    handle_unknown_language()
else:
    route_to_language_pipeline(result["predicted_language_code"])
```

## Supported languages

100 spoken languages are recognized: Afrikaans, Albanian, Amharic, Arabic, Armenian, Assamese, Azerbaijani, Bashkir, Basque, Belarusian, Bengali, Bosnian, Breton, Bulgarian, Cantonese, Catalan, Chinese, Croatian, Czech, Danish, Dutch, English, Estonian, Faroese, Finnish, French, Galician, Georgian, German, Greek, Gujarati, Haitian Creole, Hausa, Hawaiian, Hebrew, Hindi, Hungarian, Icelandic, Indonesian, Italian, Japanese, Javanese, Kannada, Kazakh, Khmer, Korean, Lao, Latin, Latvian, Lingala, Lithuanian, Luxembourgish, Macedonian, Malagasy, Malay, Malayalam, Maltese, Maori, Marathi, Mongolian, Myanmar, Nepali, Norwegian, Nynorsk, Occitan, Pashto, Persian, Polish, Portuguese, Punjabi, Romanian, Russian, Sanskrit, Serbian, Shona, Sindhi, Sinhala, Slovak, Slovenian, Somali, Spanish, Sundanese, Swahili, Swedish, Tagalog, Tajik, Tamil, Tatar, Telugu, Thai, Tibetan, Turkish, Turkmen, Ukrainian, Urdu, Uzbek, Vietnamese, Welsh, Yiddish, Yoruba.

## API reference

* [Language Detection Batch](/api-reference/language-detection/batch)


# Music & Speech Detection
Source: https://docs.modulate.ai/get-started/music-detection

Classify audio as music, speech, or neither, with frame-level probabilities. Batch and streaming.

Music & Speech Detection classifies what kind of sound is present across a clip: music, speech, both, or neither. Common uses are skipping hold music, gating a pipeline so only speech is processed downstream, and locating where a broadcast switches between talk and track.

`music_prob` and `speech_prob` are independent, not a split of 100%. A singer over a backing track scores high on both at once.

This is not AI Music Detection, which answers whether music was model-generated. See [AI Music Detection](/get-started/ai-music-detection).

|          | Batch                                                   | Streaming                                             |
| -------- | ------------------------------------------------------- | ----------------------------------------------------- |
| Protocol | HTTP POST                                               | WebSocket                                             |
| Input    | A complete file                                         | Live audio frames                                     |
| Output   | All frames at once, plus clip totals                    | Frames as they are classified, then the totals        |
| Use case | Sorting an archive, trimming hold music from recordings | Routing a live stream, real-time scene classification |

## Music & Speech Detection (batch)

Returns `application/json`. Every field is always present.

| Field                    | Type    | Contents                                                               |
| ------------------------ | ------- | ---------------------------------------------------------------------- |
| `filename`               | string  | The submitted filename, or an empty string if none was sent.           |
| `duration_s`             | number  | Total duration in **seconds**, not milliseconds.                       |
| `primary_label`          | string  | `music`, `speech`, `neither`, or `unknown`.                            |
| `music_pct`              | number  | Percentage of the clip classified as containing music, 0 to 100.       |
| `speech_pct`             | number  | Percentage of the clip classified as containing speech, 0 to 100.      |
| `latency_ms`             | number  | Server-side inference time.                                            |
| `frames`                 | array   | Per-frame results, ordered, covering the full duration.                |
| `frames[].start_time_ms` | integer | Frame start.                                                           |
| `frames[].end_time_ms`   | integer | Frame end. Each frame spans 192 ms.                                    |
| `frames[].music_prob`    | number  | Probability the frame contains music, 0 to 1, to four decimal places.  |
| `frames[].speech_prob`   | number  | Probability the frame contains speech, 0 to 1, to four decimal places. |

| `primary_label` | Condition                                                                |
| --------------- | ------------------------------------------------------------------------ |
| `music`         | Music covers at least as much of the clip as speech, and more than zero. |
| `speech`        | Speech covers more of the clip than music, and more than zero.           |
| `neither`       | Neither reached the dominant threshold for any part of the clip.         |
| `unknown`       | No frames could be produced from the audio.                              |

Note that duration here is `duration_s` in seconds. Most other Modulate endpoints report `duration_ms`.

### Try it

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://platform.modulate.ai/api/velma-2-music-detection-batch \
    -H "X-API-Key: $MODULATE_API_KEY" \
    -F "upload_file=@audio.mp3"
  ```

  ```python Python theme={null}
  import os, requests

  response = requests.post(
      "https://platform.modulate.ai/api/velma-2-music-detection-batch",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
      files={"upload_file": open("audio.mp3", "rb")},
  )
  response.raise_for_status()
  result = response.json()

  print(f"{result['primary_label']}: music {result['music_pct']}%, speech {result['speech_pct']}%")
  for frame in result["frames"]:
      print(f"  {frame['start_time_ms']}ms - {frame['end_time_ms']}ms  "
            f"music={frame['music_prob']:.4f}  speech={frame['speech_prob']:.4f}")
  ```

  ```javascript JavaScript theme={null}
  import fs from "fs";
  import FormData from "form-data";

  const form = new FormData();
  form.append("upload_file", fs.createReadStream("audio.mp3"), { filename: "audio.mp3" });

  const response = await fetch(
    "https://platform.modulate.ai/api/velma-2-music-detection-batch",
    {
      method: "POST",
      headers: { "X-API-Key": process.env.MODULATE_API_KEY, ...form.getHeaders() },
      body: form,
    }
  );

  const result = await response.json();
  console.log(`${result.primary_label}: music ${result.music_pct}%, speech ${result.speech_pct}%`);
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "filename": "my_audio.wav",
    "duration_s": 5.76,
    "primary_label": "speech",
    "music_pct": 0.0,
    "speech_pct": 86.7,
    "latency_ms": 1243.5,
    "frames": [
      { "start_time_ms": 0,   "end_time_ms": 192, "music_prob": 0.0213, "speech_prob": 0.9888 },
      { "start_time_ms": 192, "end_time_ms": 384, "music_prob": 0.0204, "speech_prob": 0.9931 }
    ]
  }
  ```
</Accordion>

### What you can configure

| Form field    | Default    | Effect                                                         |
| ------------- | ---------- | -------------------------------------------------------------- |
| `upload_file` | *required* | The audio to analyze. This endpoint takes no other parameters. |

### Audio formats

Accepted extensions include `.mp3`, `.wav`, `.flac`, `.m4a`, `.mp4`, `.ogg`, `.opus`, `.webm`, `.aac`, `.aiff`, and `.mov`, plus 86 others.

<Accordion title="All 97 accepted extensions">
  `.3g2`, `.3ga`, `.3gp`, `.3gpp`, `.8svx`, `.aa3`, `.aac`, `.ac3`, `.act`, `.adts`, `.aif`,
  `.aifc`, `.aiff`, `.alac`, `.amb`, `.amr`, `.ape`, `.asf`, `.at3`, `.au`, `.avi`, `.avr`,
  `.awb`, `.bwf`, `.c2`, `.caf`, `.dss`, `.dts`, `.dtshd`, `.eac3`, `.ec3`, `.f4a`, `.f4b`,
  `.flac`, `.flv`, `.gsm`, `.iff`, `.m2a`, `.m2ts`, `.m4a`, `.m4b`, `.m4r`, `.m4v`, `.mka`,
  `.mkv`, `.mlp`, `.mmf`, `.mov`, `.mp+`, `.mp1`, `.mp2`, `.mp3`, `.mp4`, `.mpa`, `.mpc`, `.mpeg`,
  `.mpg`, `.mpga`, `.mpp`, `.mts`, `.mxf`, `.nist`, `.oga`, `.ogg`, `.ogx`, `.oma`, `.omg`,
  `.opus`, `.paf`, `.pvf`, `.qcp`, `.ra`, `.rf64`, `.rka`, `.rm`, `.rmvb`, `.sf`, `.shn`, `.snd`,
  `.sph`, `.spx`, `.svx`, `.tak`, `.thd`, `.ts`, `.tta`, `.vob`, `.voc`, `.vqf`, `.w64`, `.wav`,
  `.wave`, `.weba`, `.webm`, `.wma`, `.wmv`, `.wv`
</Accordion>

Maximum file size is 100 MB. Empty files are rejected with `400`.

## Music & Speech Detection (streaming)

One `frame` message per completed 192 ms window, then a `done` message carrying clip totals computed across everything received.

| Message | Payload                                                                                                       |
| ------- | ------------------------------------------------------------------------------------------------------------- |
| `frame` | `frame` object with `start_time_ms`, `end_time_ms`, `music_prob`, `speech_prob`. Same shape as a batch frame. |
| `done`  | `duration_ms`, `frame_count`, `music_pct`, `speech_pct`, `primary_label`. The server closes after this.       |
| `error` | `error` string. The server closes after this.                                                                 |

Note that `done` reports `duration_ms` in milliseconds, where the batch response reports `duration_s` in seconds.

Send audio as binary WebSocket frames of any size; the server buffers and windows internally. Send an empty text frame (`""`) to end the stream. The server then flushes remaining audio, delivers final frames, sends `done`, and closes.

### Try it

<CodeGroup>
  ```bash websocat theme={null}
  websocat "wss://platform.modulate.ai/api/velma-2-music-detection-streaming?api_key=$MODULATE_API_KEY&audio_format=mp3" \
    --binary - < audio.mp3
  ```

  ```python Python theme={null}
  import os, asyncio, json, websockets

  API_KEY = os.environ["MODULATE_API_KEY"]
  AUDIO_FILE = "audio.mp3"
  CHUNK_SIZE = 65536

  async def stream():
      url = (
          f"wss://platform.modulate.ai/api/velma-2-music-detection-streaming"
          f"?api_key={API_KEY}&audio_format=mp3"
      )
      async with websockets.connect(url) as ws:
          async def send():
              with open(AUDIO_FILE, "rb") as f:
                  while chunk := f.read(CHUNK_SIZE):
                      await ws.send(chunk)
              await ws.send("")  # end of stream

          async def receive():
              async for message in ws:
                  msg = json.loads(message)
                  if msg["type"] == "frame":
                      f = msg["frame"]
                      print(f"  {f['start_time_ms']}ms - {f['end_time_ms']}ms  "
                            f"music={f['music_prob']:.4f}  speech={f['speech_prob']:.4f}")
                  elif msg["type"] == "done":
                      print(f"Done: {msg['duration_ms']}ms, {msg['frame_count']} frames")
                      print(f"{msg['primary_label']}: music {msg['music_pct']}%, "
                            f"speech {msg['speech_pct']}%")
                      break
                  elif msg["type"] == "error":
                      raise RuntimeError(msg["error"])

          await asyncio.gather(send(), receive())

  asyncio.run(stream())
  ```

  ```javascript JavaScript theme={null}
  import { WebSocket } from "ws";
  import { createReadStream } from "fs";

  const url =
    `wss://platform.modulate.ai/api/velma-2-music-detection-streaming` +
    `?api_key=${process.env.MODULATE_API_KEY}&audio_format=mp3`;

  const ws = new WebSocket(url);

  ws.on("open", () => {
    const stream = createReadStream("audio.mp3", { highWaterMark: 65536 });
    stream.on("data", (chunk) => ws.send(chunk));
    stream.on("end", () => ws.send(""));
  });

  ws.on("message", (data) => {
    const msg = JSON.parse(data);
    if (msg.type === "frame") {
      const f = msg.frame;
      console.log(`${f.start_time_ms}ms - ${f.end_time_ms}ms  music=${f.music_prob}  speech=${f.speech_prob}`);
    } else if (msg.type === "done") {
      console.log(`${msg.primary_label}: music ${msg.music_pct}%, speech ${msg.speech_pct}%`);
      ws.close();
    } else if (msg.type === "error") {
      throw new Error(msg.error);
    }
  });
  ```
</CodeGroup>

<Accordion title="Messages received">
  ```json theme={null}
  { "type": "frame", "frame": { "start_time_ms": 0, "end_time_ms": 192, "music_prob": 0.0213, "speech_prob": 0.9888 } }
  { "type": "frame", "frame": { "start_time_ms": 192, "end_time_ms": 384, "music_prob": 0.0204, "speech_prob": 0.9931 } }
  { "type": "done", "duration_ms": 15360, "frame_count": 80, "music_pct": 42.5, "speech_pct": 57.5, "primary_label": "speech" }
  ```
</Accordion>

WebSocket endpoints cannot be exercised with cURL. For command-line testing use [websocat](https://github.com/vi/websocat).

### What you can configure

| Query parameter | Default            | Effect                                                                                            |
| --------------- | ------------------ | ------------------------------------------------------------------------------------------------- |
| `api_key`       | *required*         | The API key. WebSocket connections authenticate through the query string, not a header.           |
| `audio_format`  | *required*         | How the server decodes the bytes sent. Required on every connection, including container formats. |
| `sample_rate`   | *required for raw* | Sample rate in Hz. Required for raw formats. Must not be sent for container formats.              |
| `num_channels`  | *required for raw* | 1 to 8. Required for raw formats. Must not be sent for container formats.                         |

### Audio formats

<Warning>
  `audio_format` is required on every connection, including self-describing containers. Omitting it closes the connection with code `1003`.
</Warning>

**Container formats.** Accepted values include `mp3`, `wav`, `flac`, `m4a`, `mp4`, `ogg`, `opus`, `webm`, `aac`, `aiff`, `wma`, `amr`, and `au`, plus 67 others. The stream header carries sample rate and channel count, so `sample_rate` and `num_channels` are ignored if supplied.

<Accordion title="All 80 accepted container values">
  `3g2`, `3ga`, `3gp`, `3gpp`, `8svx`, `aa3`, `aac`, `ac3`, `act`, `adts`, `aif`, `aifc`, `aiff`,
  `amb`, `amr`, `asf`, `at3`, `au`, `avr`, `awb`, `bwf`, `c2`, `caf`, `dss`, `dts`, `dtshd`,
  `eac3`, `ec3`, `f4a`, `f4b`, `flac`, `gsm`, `iff`, `m2a`, `m2ts`, `m4a`, `m4b`, `m4r`, `m4v`,
  `mka`, `mkv`, `mlp`, `mp+`, `mp1`, `mp2`, `mp3`, `mp4`, `mpa`, `mpc`, `mpga`, `mpp`, `mts`,
  `oga`, `ogg`, `ogx`, `oma`, `omg`, `opus`, `paf`, `pvf`, `qcp`, `ra`, `rf64`, `rm`, `rmvb`,
  `snd`, `spx`, `svx`, `thd`, `ts`, `tta`, `voc`, `vqf`, `w64`, `wav`, `wave`, `weba`, `webm`,
  `wma`, `wmv`
</Accordion>

The MP4-family values (`mp4`, `m4a`, `m4b`, `m4r`, `m4v`, `3gp`, `3gpp`, `3ga`, `3g2`, `f4a`, `f4b`) must be sent in a streamable layout. Anything else ends the connection with an audio-processing error.

**Raw formats.** `s8`, `s16le`, `s16be`, `s24le`, `s24be`, `s32le`, `s32be`, `u8`, `u16le`, `u16be`, `u24le`, `u24be`, `u32le`, `u32be`, `f32le`, `f32be`, `f64le`, `f64be`, `mulaw`, `alaw`, `g722`, `vox`. These are headerless, so `sample_rate` and `num_channels` are both required. `g722` and `vox` are mono-only: `num_channels` must be `1`.

`sample_rate` accepts `8000`, `11025`, `16000`, `22050`, `32000`, `44100`, `48000`, `96000`.

Common raw configurations:

| Source                         | `audio_format` | `sample_rate` | `num_channels` |
| ------------------------------ | -------------- | ------------- | -------------- |
| Native app default             | `s16le`        | `16000`       | `1`            |
| Web Audio API (`AudioWorklet`) | `f32le`        | `48000`       | `1`            |
| Native stereo capture          | `s16le`        | `48000`       | `2`            |
| Telephony (mu-law)             | `mulaw`        | `8000`        | `1`            |
| Telephony (A-law)              | `alaw`         | `8000`        | `1`            |

To convert a file to raw PCM:

```bash theme={null}
ffmpeg -i audio.mp3 -ar 16000 -ac 1 -f s16le audio.raw
```

Format parameters are validated at connection time, closing with `1003`, and again while decoding, closing with `4002`. The second catches undecodable audio and a mismatch between the declared `audio_format` and the bytes sent.

## API reference

* [Music & Speech Detection Batch](/api-reference/music-detection/batch)
* [Music & Speech Detection Streaming](/api-reference/music-detection/streaming)


# PII/PHI Redaction
Source: https://docs.modulate.ai/get-started/pii

Transcribe audio with sensitive spans removed from the text and the matching ranges silenced in the audio. Batch and streaming.

PII/PHI Redaction transcribes audio and returns a version safe to retain: sensitive spans replaced in the text, and the corresponding ranges silenced in the audio.

Redaction transcribes as part of its work, so the transcript comes back either way. A separate transcription call on the same audio is redundant.

Each detected span is replaced with an empty marker tag, with the surrounding words preserved. `<pii:CATEGORY></pii:CATEGORY>` marks personal information, where `CATEGORY` identifies the entity type. `<phi></phi>` marks health information.

|            | Tagging on Transcription                                       | Redaction                                       |
| ---------- | -------------------------------------------------------------- | ----------------------------------------------- |
| Request    | `pii_phi_tagging=true` on a Transcription endpoint             | The endpoints on this page                      |
| Transcript | Sensitive spans wrapped in tags, content preserved             | Sensitive spans replaced with empty marker tags |
| Audio      | Unchanged                                                      | Sensitive ranges silenced                       |
| Best fit   | The transcript needs marking up but the original audio is kept | Both transcript and audio must be sanitized     |

## PII/PHI Redaction (batch)

Returns **`multipart/form-data`, not JSON**. Two parts:

| Part       | Content type       | Contents                                                      |
| ---------- | ------------------ | ------------------------------------------------------------- |
| `metadata` | `application/json` | The redacted transcript, utterances, and the ranges silenced. |
| `audio`    | `audio/mpeg`       | The redacted MP3.                                             |

The `metadata` part:

| Field                         | Type    | Contents                                                   |
| ----------------------------- | ------- | ---------------------------------------------------------- |
| `text`                        | string  | The complete redacted transcript.                          |
| `duration_ms`                 | integer | Total audio duration.                                      |
| `utterances`                  | array   | Speaker turns, ordered by start time.                      |
| `utterances[].utterance_uuid` | string  | Identifier for the utterance.                              |
| `utterances[].text`           | string  | Redacted text for this utterance.                          |
| `utterances[].start_ms`       | integer | Start relative to the beginning of the audio.              |
| `utterances[].duration_ms`    | integer | Utterance length.                                          |
| `utterances[].speaker`        | integer | Speaker number, 1-indexed.                                 |
| `utterances[].language`       | string  | Detected language code for this utterance.                 |
| `redaction_ranges`            | array   | `[start_ms, end_ms]` pairs silenced in the returned audio. |

`redaction_ranges` reflects the final merged and padded ranges actually applied, not the raw detections, so a range can be wider than the words it covers.

### Try it

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://platform.modulate.ai/api/velma-2-pii-phi-redaction-batch \
    -H "X-API-Key: $MODULATE_API_KEY" \
    -F "upload_file=@audio.mp3" \
    -F "speaker_diarization=true" \
    -o response.multipart \
    -D response_headers.txt
  ```

  ```python Python theme={null}
  import os, json, requests
  from requests_toolbelt.multipart.decoder import MultipartDecoder

  response = requests.post(
      "https://platform.modulate.ai/api/velma-2-pii-phi-redaction-batch",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
      data={"speaker_diarization": "true"},
      files={"upload_file": open("audio.mp3", "rb")},
  )
  response.raise_for_status()

  for part in MultipartDecoder.from_response(response).parts:
      disposition = part.headers.get(b"Content-Disposition", b"").decode()
      if 'name="metadata"' in disposition:
          metadata = json.loads(part.content)
          print(metadata["text"])
      elif 'name="audio"' in disposition:
          with open("redacted.mp3", "wb") as f:
              f.write(part.content)
  ```
</CodeGroup>

Decoding the multipart response in Python needs `requests-toolbelt`:

```bash theme={null}
pip install requests-toolbelt
```

<Accordion title="Response (the metadata part)">
  ```json theme={null}
  {
    "text": "My name is <pii:name></pii:name> and my SSN is <pii:ssn></pii:ssn>.",
    "duration_ms": 5600,
    "utterances": [
      {
        "utterance_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
        "text": "My name is <pii:name></pii:name> and my SSN is <pii:ssn></pii:ssn>.",
        "start_ms": 0,
        "duration_ms": 5600,
        "speaker": 1,
        "language": "en"
      }
    ],
    "redaction_ranges": [[1100, 1800], [3200, 4400]]
  }
  ```
</Accordion>

### What you can configure

| Form field                   | Default     | Effect                                                                                                                                        |
| ---------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `upload_file`                | *required*  | The audio to transcribe and redact.                                                                                                           |
| `speaker_diarization`        | `true`      | Adds a 1-indexed `speaker` to each utterance.                                                                                                 |
| `start_redaction_padding_ms` | `100`       | Silence prepended to each redacted range, as a buffer before the sensitive words.                                                             |
| `end_redaction_padding_ms`   | `0`         | Silence appended to each redacted range. Raise this when trailing syllables survive.                                                          |
| `language`                   | auto-detect | ISO 639-1 hint such as `en` or `fr`. BCP 47 subtags like `en-US` are accepted; only the primary subtag is used. Omit to detect per utterance. |

An invalid `language` is rejected with `400`.

### Audio formats

Accepted extensions: `.aac`, `.aiff`, `.flac`, `.mov`, `.mp3`, `.mp4`, `.ogg`, `.opus`, `.wav`, `.webm`.

Maximum file size is 100 MB. Empty files are rejected with `400`.

## PII/PHI Redaction (streaming)

Redacted results arrive utterance by utterance while the call is still running. The stream interleaves JSON text frames and binary MP3 frames, and the pairing rule is the part a client has to get right.

| Message     | Payload                                      | Binary frame                                    |
| ----------- | -------------------------------------------- | ----------------------------------------------- |
| `utterance` | `utterance` object plus `redacted_audio`     | Only when `redacted_audio` is not null          |
| `done`      | `duration_ms` plus `trailing_redacted_audio` | Only when `trailing_redacted_audio` is not null |
| `error`     | `error` string                               | Never                                           |

When `redacted_audio` is not null it carries `start_ms` and `duration_ms` describing the window the next binary frame covers. It is null for an out-of-order utterance whose audio was already emitted in a previous window, and in that case no binary frame follows.

`trailing_redacted_audio` works the same way for audio after the final utterance. It is null when the last utterance already reached the end.

The `utterance` object carries `utterance_uuid`, `text`, `start_ms`, `duration_ms`, `speaker`, and `language`.

### Try it

<CodeGroup>
  ```bash websocat theme={null}
  websocat "wss://platform.modulate.ai/api/velma-2-pii-phi-redaction-streaming?api_key=$MODULATE_API_KEY&speaker_diarization=true" \
    --binary - < audio.mp3
  ```

  ```python Python theme={null}
  import os, asyncio, json, websockets

  API_KEY = os.environ["MODULATE_API_KEY"]
  AUDIO_FILE = "audio.mp3"
  CHUNK_SIZE = 4096

  async def redact():
      url = (
          f"wss://platform.modulate.ai/api/velma-2-pii-phi-redaction-streaming"
          f"?api_key={API_KEY}&speaker_diarization=true"
      )
      audio_clips = []

      async with websockets.connect(url) as ws:
          async def send():
              with open(AUDIO_FILE, "rb") as f:
                  while chunk := f.read(CHUNK_SIZE):
                      await ws.send(chunk)
              await ws.send("")  # end of stream

          async def receive():
              is_done = False
              async for message in ws:
                  if isinstance(message, bytes):
                      audio_clips.append(message)
                      if is_done:          # the trailing clip after done
                          break
                      continue
                  msg = json.loads(message)
                  if msg["type"] == "utterance":
                      u = msg["utterance"]
                      print(f"[{u['start_ms']}ms] Speaker {u['speaker']}: {u['text']}")
                  elif msg["type"] == "done":
                      is_done = True
                      if not msg.get("trailing_redacted_audio"):
                          break            # no binary frame follows
                  elif msg["type"] == "error":
                      raise RuntimeError(msg["error"])

          await asyncio.gather(send(), receive())

      if audio_clips:
          with open("redacted.mp3", "wb") as f:
              for clip in audio_clips:
                  f.write(clip)

  asyncio.run(redact())
  ```
</CodeGroup>

<Accordion title="Messages received">
  ```json theme={null}
  { "type": "utterance", "utterance": { "utterance_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", "text": "My name is <pii:name></pii:name> and my SSN is <pii:ssn></pii:ssn>.", "start_ms": 0, "duration_ms": 5600, "speaker": 1, "language": "en" }, "redacted_audio": { "start_ms": 0, "duration_ms": 5600 } }
  <binary MP3 frame>
  { "type": "done", "duration_ms": 5600, "trailing_redacted_audio": null }
  ```
</Accordion>

WebSocket endpoints cannot be exercised with cURL. For command-line testing use [websocat](https://github.com/vi/websocat).

### What you can configure

| Query parameter              | Default            | Effect                                                                                           |
| ---------------------------- | ------------------ | ------------------------------------------------------------------------------------------------ |
| `api_key`                    | *required*         | The API key. WebSocket connections authenticate through the query string, not a header.          |
| `speaker_diarization`        | `true`             | Adds a 1-indexed `speaker` to each utterance.                                                    |
| `language`                   | auto-detect        | ISO 639-1 hint. Omit to detect per utterance. An invalid code closes the connection with `1003`. |
| `start_redaction_padding_ms` | `100`              | Silence prepended to each redacted range.                                                        |
| `end_redaction_padding_ms`   | `0`                | Silence appended to each redacted range.                                                         |
| `audio_format`               | auto-detect        | Optional for self-describing containers. Required for raw formats.                               |
| `sample_rate`                | *required for raw* | Sample rate in Hz. Must not be sent for self-describing formats.                                 |
| `num_channels`               | *required for raw* | 1 to 8. Must not be sent for self-describing formats.                                            |

Neither redaction endpoint accepts `custom_terms`. `language` is the only vocabulary control.

### Audio formats

**Self-describing formats.** `wav`, `mp3`, `ogg`, `flac`, `webm`, `aac`, `aiff`. These carry sample rate and channel count in the stream, so `audio_format` is optional and is auto-detected from the headers when omitted. `sample_rate` and `num_channels` must not be sent for these.

Unlike the detection streaming endpoints, this one auto-detects containers. `audio_format` is only required for raw audio.

**Raw formats.** `s8`, `s16le`, `s16be`, `s24le`, `s24be`, `s32le`, `s32be`, `u8`, `u16le`, `u16be`, `u24le`, `u24be`, `u32le`, `u32be`, `f32le`, `f32be`, `f64le`, `f64be`, `mulaw`, `alaw`. Headerless, so `audio_format`, `sample_rate`, and `num_channels` are all required. Omitting either of the latter two closes the connection with `1003`.

`sample_rate` accepts `8000`, `11025`, `16000`, `22050`, `32000`, `44100`, `48000`, `96000`.

Common raw configurations:

| Source                         | `audio_format` | `sample_rate` | `num_channels` |
| ------------------------------ | -------------- | ------------- | -------------- |
| Native app default             | `s16le`        | `16000`       | `1`            |
| Web Audio API (`AudioWorklet`) | `f32le`        | `48000`       | `1`            |
| Telephony (mu-law)             | `mulaw`        | `8000`        | `1`            |
| Telephony (A-law)              | `alaw`         | `8000`        | `1`            |

To convert a file to raw PCM:

```bash theme={null}
ffmpeg -i audio.mp3 -ar 16000 -ac 1 -f s16le audio.raw
```

## API reference

* [PII/PHI Redaction Batch](/api-reference/redaction/batch)
* [PII/PHI Redaction Streaming](/api-reference/redaction/streaming)


# Transcription
Source: https://docs.modulate.ai/get-started/stt

Three transcription models, batch and streaming, with speaker labels, emotion, accent, deepfake, and PII/PHI signals.

Modulate has three transcription models.

* **Multilingual Transcription** carries per-utterance timing, speaker labels, and optional emotion, accent, deepfake, and PII/PHI signals. Batch and streaming.
* **English Fast Transcription** is English-only and tuned for throughput and latency. Batch and streaming.
* **Multilingual Fast Transcription** returns a transcript in any supported language with no metadata. Batch only.

|                        | Multilingual (batch) | Multilingual (streaming)     | English Fast (batch) | English Fast (streaming) | Multilingual Fast (batch) |
| ---------------------- | -------------------- | ---------------------------- | -------------------- | ------------------------ | ------------------------- |
| Protocol               | HTTP POST            | WebSocket                    | HTTP POST            | WebSocket                | HTTP POST                 |
| Languages              | Multilingual         | Multilingual                 | English only         | English only             | Multilingual              |
| Utterance-level output | ✓                    | ✓                            | ✓ (opt-in)           | ✓ (opt-in `endpointing`) | —                         |
| Speaker diarization    | ✓                    | ✓                            | ✓ (opt-in)           | —                        | —                         |
| Emotion signal         | ✓                    | ✓                            | —                    | —                        | —                         |
| Accent signal          | ✓                    | ✓                            | —                    | —                        | —                         |
| Deepfake signal        | ✓                    | ✓                            | —                    | —                        | —                         |
| PII/PHI tagging        | ✓                    | ✓                            | —                    | —                        | —                         |
| Language hint          | ✓                    | ✓                            | —                    | —                        | ✓                         |
| Custom vocabulary      | ✓                    | ✓                            | —                    | —                        | —                         |
| Partial transcripts    | —                    | ✓ (opt-in `partial_results`) | —                    | ✓ (every \~1.5 s)        | —                         |

Every signal is off by default except speaker diarization, which defaults to `true` on Multilingual Transcription and to `false` on English Fast Transcription.

## Multilingual Transcription (batch)

Returns `application/json`. `text`, `duration_ms`, and `utterances` are always present. `text` may be an empty string and `utterances` an empty array when no speech was recognized.

| Field                         | Type           | Contents                                                                         |
| ----------------------------- | -------------- | -------------------------------------------------------------------------------- |
| `text`                        | string         | The full transcript, all utterances concatenated.                                |
| `duration_ms`                 | integer        | Total audio duration.                                                            |
| `utterances`                  | array          | Utterances ordered by start time.                                                |
| `utterances[].utterance_uuid` | string         | Identifier for the utterance.                                                    |
| `utterances[].text`           | string         | Transcribed text for this utterance.                                             |
| `utterances[].start_ms`       | integer        | Start relative to the beginning of the file.                                     |
| `utterances[].duration_ms`    | integer        | Utterance length.                                                                |
| `utterances[].speaker`        | integer        | Speaker number, 1-indexed.                                                       |
| `utterances[].language`       | string         | Language detected for this utterance.                                            |
| `utterances[].emotion`        | string or null | Null unless `emotion_signal` is enabled.                                         |
| `utterances[].accent`         | string or null | Null unless `accent_signal` is enabled.                                          |
| `utterances[].deepfake_score` | number or null | Null unless `deepfake_signal` is enabled, or the utterance is under 0.5 seconds. |

Language is detected per utterance, so a file where speakers switch languages reports each utterance in the language actually spoken.

### Try it

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://platform.modulate.ai/api/velma-2-stt-batch \
    -H "X-API-Key: $MODULATE_API_KEY" \
    -F "upload_file=@audio.mp3" \
    -F "speaker_diarization=true" \
    -F "emotion_signal=true" \
    -F "accent_signal=true"
  ```

  ```python Python theme={null}
  import os, requests

  response = requests.post(
      "https://platform.modulate.ai/api/velma-2-stt-batch",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
      data={
          "speaker_diarization": "true",
          "emotion_signal": "true",
          "accent_signal": "true",
      },
      files={"upload_file": open("audio.mp3", "rb")},
  )
  response.raise_for_status()
  result = response.json()

  print(result["text"])
  for u in result["utterances"]:
      print(f"[{u['start_ms']}ms] Speaker {u['speaker']} ({u['language']}): {u['text']}")
  ```

  ```javascript JavaScript theme={null}
  import fs from "fs";
  import FormData from "form-data";

  const form = new FormData();
  form.append("upload_file", fs.createReadStream("audio.mp3"), { filename: "audio.mp3" });
  form.append("speaker_diarization", "true");
  form.append("emotion_signal", "true");

  const response = await fetch("https://platform.modulate.ai/api/velma-2-stt-batch", {
    method: "POST",
    headers: { "X-API-Key": process.env.MODULATE_API_KEY, ...form.getHeaders() },
    body: form,
  });

  const result = await response.json();
  for (const u of result.utterances) {
    console.log(`[${u.start_ms}ms] Speaker ${u.speaker} (${u.language}): ${u.text}`);
  }
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "text": "Hello, how are you? Bonjour, ça va?",
    "duration_ms": 5000,
    "utterances": [
      {
        "utterance_uuid": "e5f6a7b8-c9d0-1234-efab-345678901234",
        "text": "Hello, how are you?",
        "start_ms": 0,
        "duration_ms": 2000,
        "speaker": 1,
        "language": "en",
        "emotion": "Neutral",
        "accent": "American",
        "deepfake_score": null
      },
      {
        "utterance_uuid": "f6a7b8c9-d0e1-2345-fabc-456789012345",
        "text": "Bonjour, ça va?",
        "start_ms": 2500,
        "duration_ms": 2500,
        "speaker": 2,
        "language": "fr",
        "emotion": "Happy",
        "accent": "British",
        "deepfake_score": null
      }
    ]
  }
  ```
</Accordion>

The example above enables emotion and accent. With every signal off, the response shape is identical and `emotion`, `accent`, and `deepfake_score` are `null`.

### What you can configure

| Form field            | Default     | Effect                                                                                                                                                 |
| --------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `upload_file`         | *required*  | The audio to transcribe.                                                                                                                               |
| `speaker_diarization` | `true`      | Adds a 1-indexed `speaker` to each utterance.                                                                                                          |
| `emotion_signal`      | `false`     | Adds an `emotion` label per utterance.                                                                                                                 |
| `accent_signal`       | `false`     | Adds an `accent` label per utterance.                                                                                                                  |
| `deepfake_signal`     | `false`     | Adds a `deepfake_score` per utterance, 0.0 natural to 1.0 synthetic.                                                                                   |
| `pii_phi_tagging`     | `false`     | Wraps sensitive spans in the transcript text with entity tags.                                                                                         |
| `language`            | auto-detect | ISO 639-1 hint such as `en` or `fr`. BCP 47 subtags like `en-US` are accepted; only the primary subtag is used. Invalid codes are rejected with `400`. |
| `config`              | —           | A JSON-encoded object carrying the same flags plus `custom_terms`. A field set here overrides the matching top-level form field.                       |

`custom_terms` travels only inside the JSON `config` field. There is no top-level form field for it.

### Audio formats

Accepted extensions: `.aac`, `.aiff`, `.flac`, `.mov`, `.mp3`, `.mp4`, `.ogg`, `.opus`, `.wav`, `.webm`.

Maximum file size is 100 MB. Empty files are rejected with `400`.

## Multilingual Transcription (streaming)

Utterances arrive as JSON text messages while audio streams in.

| Message             | Payload                                                     |
| ------------------- | ----------------------------------------------------------- |
| `utterance`         | `utterance` object, same fields as the batch response.      |
| `partial_utterance` | In-progress text. Emitted only with `partial_results=true`. |
| `done`              | `duration_ms`. The server closes after this.                |
| `error`             | `error` string. The server closes after this.               |

Each `partial_utterance` carries `text`, `start_ms`, `speaker`, `emotion`, `accent`, and `deepfake_score`. Any of those except `text` may be null before a value is available. Each partial replaces the previous one for the active utterance, and the finalized `utterance` supersedes all partials before it.

Send audio as binary WebSocket frames in any chunk size. Send an empty text frame (`""`) to end the stream.

### Try it

<CodeGroup>
  ```bash websocat theme={null}
  websocat "wss://platform.modulate.ai/api/velma-2-stt-streaming?api_key=$MODULATE_API_KEY&speaker_diarization=true" \
    --binary - < audio.mp3
  ```

  ```python Python theme={null}
  import os, asyncio, json, websockets

  API_KEY = os.environ["MODULATE_API_KEY"]
  AUDIO_FILE = "audio.mp3"
  CHUNK_SIZE = 4096

  async def stream():
      url = (
          f"wss://platform.modulate.ai/api/velma-2-stt-streaming"
          f"?api_key={API_KEY}&speaker_diarization=true"
      )
      async with websockets.connect(url) as ws:
          async def send():
              with open(AUDIO_FILE, "rb") as f:
                  while chunk := f.read(CHUNK_SIZE):
                      await ws.send(chunk)
              await ws.send("")  # end of stream

          async def receive():
              async for message in ws:
                  msg = json.loads(message)
                  if msg["type"] == "utterance":
                      u = msg["utterance"]
                      print(f"[{u['start_ms']}ms] Speaker {u['speaker']}: {u['text']}")
                  elif msg["type"] == "done":
                      print(f"Done: {msg['duration_ms']}ms")
                      break
                  elif msg["type"] == "error":
                      raise RuntimeError(msg["error"])

          await asyncio.gather(send(), receive())

  asyncio.run(stream())
  ```

  ```javascript JavaScript theme={null}
  import { WebSocket } from "ws";
  import { createReadStream } from "fs";

  const url =
    `wss://platform.modulate.ai/api/velma-2-stt-streaming` +
    `?api_key=${process.env.MODULATE_API_KEY}&speaker_diarization=true`;

  const ws = new WebSocket(url);

  ws.on("open", () => {
    const stream = createReadStream("audio.mp3", { highWaterMark: 4096 });
    stream.on("data", (chunk) => ws.send(chunk));
    stream.on("end", () => ws.send(""));
  });

  ws.on("message", (data) => {
    const msg = JSON.parse(data);
    if (msg.type === "utterance") {
      const u = msg.utterance;
      console.log(`[${u.start_ms}ms] Speaker ${u.speaker}: ${u.text}`);
    } else if (msg.type === "done") {
      ws.close();
    } else if (msg.type === "error") {
      throw new Error(msg.error);
    }
  });
  ```
</CodeGroup>

<Accordion title="Messages received">
  ```json theme={null}
  { "type": "utterance", "utterance": { "utterance_uuid": "b2c3d4e5-f6a7-8901-bcde-f12345678901", "text": "Bonjour, comment allez-vous?", "start_ms": 0, "duration_ms": 2800, "speaker": 1, "language": "fr", "emotion": null, "accent": null, "deepfake_score": null } }
  { "type": "done", "duration_ms": 45000 }
  ```
</Accordion>

WebSocket endpoints cannot be exercised with cURL. For command-line testing use [websocat](https://github.com/vi/websocat).

### What you can configure

| Query parameter       | Default            | Effect                                                                                  |
| --------------------- | ------------------ | --------------------------------------------------------------------------------------- |
| `api_key`             | *required*         | The API key. WebSocket connections authenticate through the query string, not a header. |
| `speaker_diarization` | `true`             | Adds a 1-indexed `speaker` to each utterance.                                           |
| `emotion_signal`      | `false`            | Adds an `emotion` label per utterance.                                                  |
| `accent_signal`       | `false`            | Adds an `accent` label per utterance.                                                   |
| `deepfake_signal`     | `false`            | Adds a `deepfake_score` per utterance.                                                  |
| `pii_phi_tagging`     | `false`            | Wraps sensitive spans in the transcribed text with entity tags.                         |
| `partial_results`     | `false`            | Emits `partial_utterance` messages with in-progress text.                               |
| `language`            | auto-detect        | ISO 639-1 hint. An invalid code closes the connection with `1003`.                      |
| `audio_format`        | auto-detect        | Optional for self-describing containers. Required for raw formats.                      |
| `sample_rate`         | *required for raw* | Sample rate in Hz. Must not be sent for self-describing formats.                        |
| `num_channels`        | *required for raw* | 1 to 8. Must not be sent for self-describing formats.                                   |

An optional JSON configuration can be sent as the **first text frame**, before any audio. Any field there overrides the matching query parameter, and it is the only way to pass `custom_terms` on this endpoint. A binary first frame is treated as audio and the query-parameter defaults apply, so binary-first clients keep working unchanged.

### Audio formats

**Self-describing formats.** `wav`, `mp3`, `ogg`, `flac`, `webm`, `aac`, `aiff`. Auto-detected from headers when `audio_format` is omitted. `sample_rate` and `num_channels` must not be sent for these.

**Raw formats.** `s8`, `s16le`, `s16be`, `s24le`, `s24be`, `s32le`, `s32be`, `u8`, `u16le`, `u16be`, `u24le`, `u24be`, `u32le`, `u32be`, `f32le`, `f32be`, `f64le`, `f64be`, `mulaw`, `alaw`. Headerless, so `audio_format`, `sample_rate`, and `num_channels` are all required.

`sample_rate` accepts `8000`, `11025`, `16000`, `22050`, `32000`, `44100`, `48000`, `96000`.

## English Fast Transcription (batch)

Returns `application/json`. `text` and `duration_ms` are always present. Two independent opt-in flags each add one key: `time_stamps` adds `words`, and `speaker_diarization` adds `utterances`. An optional key is absent rather than `null` when its flag is off, so the response schema does not change shape as flags are toggled.

| Field                         | Type    | Contents                                                                                                                             |
| ----------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `text`                        | string  | The full transcript, auto-capitalized and auto-punctuated.                                                                           |
| `duration_ms`                 | integer | Total audio duration.                                                                                                                |
| `words`                       | array   | Word timings covering `text`, in document order. Present only with `time_stamps=true`.                                               |
| `words[].word`                | string  | The word as it appears in `text`, including attached punctuation and capitalization.                                                 |
| `words[].start`               | number  | Start time in seconds from the start of the file, to three decimals.                                                                 |
| `words[].end`                 | number  | End time in seconds from the start of the file, to three decimals.                                                                   |
| `words[].score`               | number  | Alignment confidence, to four decimals. Scores how well the word was located in the audio, not whether it was transcribed correctly. |
| `utterances`                  | array   | Speaker turns. Present only with `speaker_diarization=true`. May be empty when no transcribable speech is found.                     |
| `utterances[].utterance_uuid` | string  | Identifier for the utterance.                                                                                                        |
| `utterances[].text`           | string  | Transcribed text for this turn.                                                                                                      |
| `utterances[].start_ms`       | integer | Start within the audio, in milliseconds.                                                                                             |
| `utterances[].duration_ms`    | integer | Turn length, in milliseconds.                                                                                                        |
| `utterances[].speaker`        | integer | Speaker number, 1-indexed.                                                                                                           |
| `utterances[].words`          | array   | Word timings for this turn only. Present only when both flags are enabled.                                                           |

<Warning>
  The two arrays carry different units. `words` times are fractional **seconds**; `utterances` times are integer **milliseconds**. Both are measured from the start of the file. That includes `utterances[].words`, whose times are file-relative rather than utterance-relative, so they compare directly against the top-level `words` array with no offset arithmetic.
</Warning>

With diarization enabled, `text` becomes the time-ordered concatenation of the utterance texts.

Word timings come from aligning the finished transcript against the audio, so every entry in `words` matches a word in `text`. Words that cannot be aligned are omitted, which means `words` can be shorter than the number of whitespace-separated tokens in `text`. Indexing one against the other drifts.

This model produces no emotion, accent, deepfake, PII/PHI, or custom vocabulary output.

### Try it

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://platform.modulate.ai/api/velma-2-stt-batch-english-vfast \
    -H "X-API-Key: $MODULATE_API_KEY" \
    -F "upload_file=@audio.mp3" \
    -F "time_stamps=true" \
    -F "speaker_diarization=true"
  ```

  ```python Python theme={null}
  import os, requests

  response = requests.post(
      "https://platform.modulate.ai/api/velma-2-stt-batch-english-vfast",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
      data={"time_stamps": "true", "speaker_diarization": "true"},
      files={"upload_file": open("audio.mp3", "rb")},
  )
  response.raise_for_status()
  result = response.json()

  print(result["text"])

  # words[] times are seconds; utterances[] times are milliseconds
  for w in result.get("words", []):
      print(f"{w['start']:6.2f}s - {w['end']:6.2f}s  {w['word']}")

  for u in result.get("utterances", []):
      print(f"[{u['start_ms']}ms] Speaker {u['speaker']}: {u['text']}")
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "text": "Thanks for calling. My account got locked.",
    "duration_ms": 9000,
    "words": [
      { "word": "Thanks", "start": 0.2, "end": 0.61, "score": 0.9881 },
      { "word": "for", "start": 0.64, "end": 0.79, "score": 0.9799 },
      { "word": "calling.", "start": 0.82, "end": 1.35, "score": 0.9744 },
      { "word": "My", "start": 4.51, "end": 4.68, "score": 0.9702 },
      { "word": "account", "start": 4.7, "end": 5.18, "score": 0.9835 },
      { "word": "got", "start": 5.2, "end": 5.39, "score": 0.9781 },
      { "word": "locked.", "start": 5.42, "end": 5.94, "score": 0.9668 }
    ],
    "utterances": [
      {
        "utterance_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
        "text": "Thanks for calling.",
        "start_ms": 0,
        "duration_ms": 1500,
        "speaker": 1,
        "words": [
          { "word": "Thanks", "start": 0.2, "end": 0.61, "score": 0.9881 },
          { "word": "for", "start": 0.64, "end": 0.79, "score": 0.9799 },
          { "word": "calling.", "start": 0.82, "end": 1.35, "score": 0.9744 }
        ]
      },
      {
        "utterance_uuid": "b2c3d4e5-f6a7-8901-bcde-f23456789012",
        "text": "My account got locked.",
        "start_ms": 4400,
        "duration_ms": 1700,
        "speaker": 2,
        "words": [
          { "word": "My", "start": 4.51, "end": 4.68, "score": 0.9702 },
          { "word": "account", "start": 4.7, "end": 5.18, "score": 0.9835 },
          { "word": "got", "start": 5.2, "end": 5.39, "score": 0.9781 },
          { "word": "locked.", "start": 5.42, "end": 5.94, "score": 0.9668 }
        ]
      }
    ]
  }
  ```
</Accordion>

The second speaker's word times are file-relative: `4.51` seconds into the file, not `0.11` seconds into their turn.

With both flags off, the response is `text` and `duration_ms` only. The `words` and `utterances` keys are absent rather than empty.

### What you can configure

| Form field            | Default    | Effect                                                                                                                                   |
| --------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `upload_file`         | *required* | The audio to transcribe.                                                                                                                 |
| `time_stamps`         | `false`    | Adds the `words` array, and `words` inside each utterance when diarization is also on. Costs latency on long audio (see Behavior below). |
| `speaker_diarization` | `false`    | Adds the `utterances` array. Note this defaults off here, unlike Multilingual Transcription.                                             |

The two flags are independent: enable either, both, or neither.

### Audio formats

Accepted extensions include `.mp3`, `.wav`, `.flac`, `.m4a`, `.mp4`, `.ogg`, `.opus`, `.webm`, `.aac`, `.aiff`, and `.mov`, plus 86 others.

<Accordion title="All 97 accepted extensions">
  `.3g2`, `.3ga`, `.3gp`, `.3gpp`, `.8svx`, `.aa3`, `.aac`, `.ac3`, `.act`, `.adts`, `.aif`,
  `.aifc`, `.aiff`, `.alac`, `.amb`, `.amr`, `.ape`, `.asf`, `.at3`, `.au`, `.avi`, `.avr`,
  `.awb`, `.bwf`, `.c2`, `.caf`, `.dss`, `.dts`, `.dtshd`, `.eac3`, `.ec3`, `.f4a`, `.f4b`,
  `.flac`, `.flv`, `.gsm`, `.iff`, `.m2a`, `.m2ts`, `.m4a`, `.m4b`, `.m4r`, `.m4v`, `.mka`,
  `.mkv`, `.mlp`, `.mmf`, `.mov`, `.mp+`, `.mp1`, `.mp2`, `.mp3`, `.mp4`, `.mpa`, `.mpc`, `.mpeg`,
  `.mpg`, `.mpga`, `.mpp`, `.mts`, `.mxf`, `.nist`, `.oga`, `.ogg`, `.ogx`, `.oma`, `.omg`,
  `.opus`, `.paf`, `.pvf`, `.qcp`, `.ra`, `.rf64`, `.rka`, `.rm`, `.rmvb`, `.sf`, `.shn`, `.snd`,
  `.sph`, `.spx`, `.svx`, `.tak`, `.thd`, `.ts`, `.tta`, `.vob`, `.voc`, `.vqf`, `.w64`, `.wav`,
  `.wave`, `.weba`, `.webm`, `.wma`, `.wmv`, `.wv`
</Accordion>

Maximum file size is 100 MB. Files above it are rejected with `413`. Empty files are rejected with `400`.

### Behavior

`time_stamps` adds a forced-alignment pass over the audio. On short audio that pass overlaps transcription and is effectively free. Its cost grows faster than linearly with audio length, so on long recordings it can add substantial latency. Benchmark with the flag set to its production value, not with it off.

`speaker_diarization` transcribes each speaker turn separately rather than the file as a whole. It is slower than the default path, and the transcript it returns can differ from the same audio submitted without the flag. Do not treat a diarized `text` as byte-identical to an undiarized one.

With diarization enabled, quiet background speech on far-field or multi-party audio can surface as additional utterances. Overlapping speech is attributed to a single speaker per time span.

<Note>
  `503` carries a second meaning on this endpoint: besides a temporary outage, it is returned when `speaker_diarization` is requested while diarization is disabled on the deployment serving the request. Retrying will not clear that case. Drop the flag, or ask about having diarization enabled.
</Note>

## English Fast Transcription (streaming)

The lowest-latency option, and the one for cases where a person is waiting on the text.

| Message             | Payload                                                          |
| ------------------- | ---------------------------------------------------------------- |
| `partial_utterance` | `text` and `is_final: false`. Emitted roughly every 1.5 seconds. |
| `utterance`         | `text`, `is_final: true`, `start_ms`, `duration_ms`.             |
| `done`              | `duration_ms`. The server closes after this.                     |
| `error`             | `error` string. The server closes after this.                    |

Text in both message types is already auto-capitalized and auto-punctuated. Render `text` directly.

<Warning>
  Each `partial_utterance` contains the complete transcript of its scope so far, not a delta. Replace the displayed text on every partial. Never append. The last few words, including their capitalization and punctuation, can be revised in the next partial as the model receives more context.
</Warning>

Every connection ends with at least one final `utterance`. A stream containing no speech yields a single final with empty `text`.

### Try it

<CodeGroup>
  ```bash websocat theme={null}
  websocat "wss://platform.modulate.ai/api/velma-2-stt-streaming-english-v2?api_key=$MODULATE_API_KEY&audio_format=ogg&endpointing=true" \
    --binary - < audio.ogg
  ```

  ```python Python theme={null}
  import asyncio, json, os, websockets

  API_KEY = os.environ["MODULATE_API_KEY"]
  AUDIO_FILE = "audio.ogg"
  CHUNK_SIZE = 8192

  async def transcribe():
      url = (
          f"wss://platform.modulate.ai/api/velma-2-stt-streaming-english-v2"
          f"?api_key={API_KEY}&audio_format=ogg&endpointing=true"
      )
      async with websockets.connect(url, max_size=None) as ws:
          async def send():
              with open(AUDIO_FILE, "rb") as f:
                  while chunk := f.read(CHUNK_SIZE):
                      await ws.send(chunk)
              await ws.send("")  # end of stream

          send_task = asyncio.create_task(send())
          try:
              async for message in ws:
                  msg = json.loads(message)
                  if msg["type"] == "partial_utterance":
                      # Replace the displayed partial, do not append.
                      print(f"\r[partial] {msg['partial_utterance']['text']}", end="", flush=True)
                  elif msg["type"] == "utterance":
                      print(f"\n[final]   {msg['utterance']['text']}")
                  elif msg["type"] == "done":
                      print(f"\nDone: {msg['duration_ms']}ms")
                      break
                  elif msg["type"] == "error":
                      raise RuntimeError(msg["error"])
          finally:
              if not send_task.done():
                  send_task.cancel()

  asyncio.run(transcribe())
  ```
</CodeGroup>

<Accordion title="Messages received">
  ```json theme={null}
  {"type": "partial_utterance", "partial_utterance": {"text": "Hello, how are you", "is_final": false}}
  {"type": "partial_utterance", "partial_utterance": {"text": "Hello, how are you doing", "is_final": false}}
  {"type": "utterance", "utterance": {"text": "Hello, how are you doing today?", "is_final": true, "start_ms": 0, "duration_ms": 2360}}
  {"type": "done", "duration_ms": 14253}
  ```
</Accordion>

WebSocket endpoints cannot be exercised with cURL. For command-line testing use [websocat](https://github.com/vi/websocat).

### What you can configure

| Query parameter | Default            | Effect                                                               |
| --------------- | ------------------ | -------------------------------------------------------------------- |
| `api_key`       | *required*         | The API key.                                                         |
| `audio_format`  | *required*         | How the server decodes the bytes sent. Required on every connection. |
| `sample_rate`   | *required for raw* | Sample rate in Hz. Ignored for container formats.                    |
| `num_channels`  | *required for raw* | 1 to 8. Ignored for container formats.                               |
| `endpointing`   | `false`            | Segments speech at pauses.                                           |

With `endpointing=false`, the connection produces a single final `utterance` at end of stream covering the whole stream with `start_ms=0`, and each partial reflects the whole stream so far.

With `endpointing=true`, trailing silence closes the current utterance and starts a new one. The server emits one final `utterance` per speech segment, each with its own `start_ms` and `duration_ms`, and each partial reflects the current segment. The full transcript is the concatenation of the final texts. This is what makes the endpoint usable for live conversation.

### Audio formats

`audio_format` is required on every connection, including containers. Omitting it closes the connection with `1003`.

**Container formats.** Accepted values include `mp3`, `wav`, `flac`, `m4a`, `mp4`, `ogg`, `opus`, `webm`, `aac`, `aiff`, `wma`, `amr`, and `au`, plus 67 others. The header determines sample rate and channel count, and `sample_rate` and `num_channels` are ignored.

<Accordion title="All 80 accepted container values">
  `3g2`, `3ga`, `3gp`, `3gpp`, `8svx`, `aa3`, `aac`, `ac3`, `act`, `adts`, `aif`, `aifc`, `aiff`,
  `amb`, `amr`, `asf`, `at3`, `au`, `avr`, `awb`, `bwf`, `c2`, `caf`, `dss`, `dts`, `dtshd`,
  `eac3`, `ec3`, `f4a`, `f4b`, `flac`, `gsm`, `iff`, `m2a`, `m2ts`, `m4a`, `m4b`, `m4r`, `m4v`,
  `mka`, `mkv`, `mlp`, `mp+`, `mp1`, `mp2`, `mp3`, `mp4`, `mpa`, `mpc`, `mpga`, `mpp`, `mts`,
  `oga`, `ogg`, `ogx`, `oma`, `omg`, `opus`, `paf`, `pvf`, `qcp`, `ra`, `rf64`, `rm`, `rmvb`,
  `snd`, `spx`, `svx`, `thd`, `ts`, `tta`, `voc`, `vqf`, `w64`, `wav`, `wave`, `weba`, `webm`,
  `wma`, `wmv`
</Accordion>

The MP4-family values (`mp4`, `m4a`, `m4b`, `m4r`, `m4v`, `3gp`, `3gpp`, `3ga`, `3g2`, `f4a`, `f4b`) must be sent in a streamable layout. Anything else ends the connection with an audio-processing error.

**Raw formats.** `s8`, `s16le`, `s16be`, `s24le`, `s24be`, `s32le`, `s32be`, `u8`, `u16le`, `u16be`, `u24le`, `u24be`, `u32le`, `u32be`, `f32le`, `f32be`, `f64le`, `f64be`, `mulaw`, `alaw`, `g722`, `vox`. `sample_rate` and `num_channels` are both required. `g722` and `vox` are mono-only: `num_channels` must be `1`.

`sample_rate` accepts `8000`, `11025`, `16000`, `22050`, `32000`, `44100`, `48000`, `96000`.

For the lowest end-to-end latency use `audio_format=s16le&sample_rate=16000&num_channels=1`, which bypasses the server's audio decoder.

Sending container bytes while declaring a raw PCM format closes the connection with `4002`.

## Multilingual Fast Transcription (batch)

Returns `application/json`. All three fields are always present.

| Field         | Type    | Contents                                                                                                              |
| ------------- | ------- | --------------------------------------------------------------------------------------------------------------------- |
| `text`        | string  | The full transcript, auto-capitalized and auto-punctuated.                                                            |
| `duration_ms` | integer | Total audio duration.                                                                                                 |
| `language`    | string  | The language the transcript is in, as a short code. Echoes a declared `language`, otherwise reports the detected one. |

For audio containing more than one language, `language` reports the predominant one. The transcript still reflects each part in the language spoken there.

This model produces no diarization, utterance breakdown, or enrichment signals.

### Try it

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://platform.modulate.ai/api/velma-2-stt-batch-multilingual-vfast \
    -H "X-API-Key: $MODULATE_API_KEY" \
    -F "upload_file=@audio.mp3" \
    -F "language=es"
  ```

  ```python Python theme={null}
  import os, requests

  response = requests.post(
      "https://platform.modulate.ai/api/velma-2-stt-batch-multilingual-vfast",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
      data={"language": "es"},
      files={"upload_file": open("audio.mp3", "rb")},
  )
  response.raise_for_status()
  result = response.json()
  print(f"[{result['language']}] {result['text']}")
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "text": "Hola a todos. Bienvenidos a la junta semanal.",
    "duration_ms": 4200,
    "language": "es"
  }
  ```
</Accordion>

### What you can configure

| Form field    | Default     | Effect                                                                                                                                              |
| ------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| `upload_file` | *required*  | The audio to transcribe.                                                                                                                            |
| `language`    | auto-detect | A short code such as `en`, `es`, `fr`, `ja`. Declaring it takes the fastest direct path and the value is echoed back. Omit to detect automatically. |

### Audio formats

Accepted extensions: `.aac`, `.aiff`, `.flac`, `.mov`, `.mp3`, `.mp4`, `.ogg`, `.opus`, `.wav`, `.webm`.

Maximum file size is 100 MB. Files above it are rejected with `413`. Empty files are rejected with `400`.

### Error codes

This endpoint returns `401` for a missing or invalid key, where the other transcription endpoints return `403`. It also returns `502` when the request cannot be completed and `504` on timeout.

## Signals

### Speaker diarization

`speaker_diarization`, boolean. Identifies distinct speakers and assigns each utterance a 1-indexed `speaker`. Numbers are consistent within one request: speaker 1 in one utterance is the same person as speaker 1 in another from the same audio. They are not consistent across separate requests or files.

Diarization is independent of language detection. In a multilingual conversation a speaker keeps one label across a language switch.

### Emotion

`emotion_signal`, boolean. Classifies the emotional tone of each utterance from the voice signal, into `emotion`. Classification is acoustic, so two utterances with identical text can receive different labels if the delivery differs.

`Neutral`, `Calm`, `Happy`, `Amused`, `Excited`, `Proud`, `Affectionate`, `Interested`, `Hopeful`, `Frustrated`, `Angry`, `Contemptuous`, `Concerned`, `Afraid`, `Sad`, `Ashamed`, `Bored`, `Tired`, `Surprised`, `Anxious`, `Stressed`, `Disgusted`, `Disappointed`, `Confused`, `Relieved`, `Confident`.

For a whole-file label with no transcript, see [Emotion Detection](/get-started/emotion).

### Accent

`accent_signal`, boolean. Classifies the regional or national accent of each utterance's speaker, into `accent`. Results are typically stable for a speaker with a consistent accent, and vary more on short or acoustically difficult segments.

`American`, `British`, `Australian`, `Southern`, `Indian`, `Irish`, `Scottish`, `Eastern_European`, `African`, `Asian`, `Latin_American`, `Middle_Eastern`, `Unknown`.

For a whole-file label with no transcript, see [Accent Detection](/get-started/accent).

### Deepfake score

`deepfake_signal`, boolean. Scores each utterance for the likelihood it contains AI-generated speech, into `deepfake_score`.

| Value  | Meaning                                                        |
| ------ | -------------------------------------------------------------- |
| `0.0`  | Likely natural human speech.                                   |
| `1.0`  | Likely synthetic speech.                                       |
| `null` | The signal is disabled, or the utterance is under 0.5 seconds. |

This is one score per utterance. For frame-level verdicts across a whole file, explicit `no-content` handling for silence, or verdicts on live audio without a transcript, see [Deepfake Detection](/get-started/deepfake).

### PII/PHI tagging

`pii_phi_tagging`, boolean. Wraps personally identifiable and personal health information in the transcript text with entity tags. Transcript content is preserved and only markup is added.

To also silence the corresponding audio ranges, use [PII/PHI Redaction](/get-started/pii) instead.

### Language hint

`language`, ISO 639-1 code. By default the language is detected per utterance. Passing a code hints the expected language. BCP 47 region and script subtags such as `en-US` are accepted, but only the primary subtag is used. Case-insensitive.

Transport differs by endpoint, and getting it wrong produces a `400` or a silently ignored value:

| Endpoint                                | Where `language` goes                                    |
| --------------------------------------- | -------------------------------------------------------- |
| Multilingual Transcription (batch)      | Top-level form field, or inside the JSON `config` field  |
| Multilingual Transcription (streaming)  | Query parameter, or inside the first-frame configuration |
| Multilingual Fast Transcription (batch) | Top-level form field                                     |
| PII/PHI Redaction (batch)               | Top-level form field                                     |
| PII/PHI Redaction (streaming)           | Query parameter                                          |
| Velma Triage (batch and streaming)      | The `stt` block of the config                            |

An invalid code is rejected with `400` on batch endpoints, and closes streaming connections with `1003`. English Fast Transcription accepts no language parameter in either mode.

### Custom vocabulary

`custom_terms`, array. Biases transcription toward domain terms and names that would otherwise be mistranscribed. Each entry is a plain string or an object:

| Field            | Type             | Contents                                                                                                                                                                            |
| ---------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `term`           | string           | Required. The term to bias toward.                                                                                                                                                  |
| `definition`     | string           | Optional, 256 characters maximum.                                                                                                                                                   |
| `pronunciations` | array of strings | Optional, 16 maximum, 128 characters each. X-SAMPA notation, or hyphen-separated ASCII respelling with the stressed syllable capitalized, such as `GOO-guhl`. Printable ASCII only. |

At most 1000 entries, and the term strings serialized together as JSON must total under 8000 characters. Blank terms and blank pronunciations are dropped.

```json theme={null}
{
  "custom_terms": [
    "Modulate",
    { "term": "Velma", "definition": "Modulate's conversation intelligence model", "pronunciations": ["VEL-muh"] }
  ]
}
```

| Endpoint                               | Where `custom_terms` goes            |
| -------------------------------------- | ------------------------------------ |
| Multilingual Transcription (batch)     | Inside the JSON `config` form field  |
| Multilingual Transcription (streaming) | Inside the first-frame configuration |
| Velma Triage (batch and streaming)     | The `stt` block of the config        |

The other transcription endpoints and both redaction endpoints do not accept custom vocabulary.

## API reference

* [Multilingual Transcription Batch](/api-reference/stt/batch)
* [Multilingual Transcription Streaming](/api-reference/stt/streaming)
* [English Fast Transcription Batch](/api-reference/stt/batch-english-vfast)
* [English Fast Transcription Streaming](/api-reference/stt/streaming-vfast)
* [Multilingual Fast Transcription Batch](/api-reference/stt/batch-multilingual-vfast)


# Velma Triage
Source: https://docs.modulate.ai/get-started/velma

Analyze a whole conversation for behaviors, topics, sentiment, speaker roles, and a summary, over a diarized transcript. Batch and streaming.

Velma Triage analyzes conversations rather than audio characteristics. It evaluates a configured set of behaviors against the transcript and reports which ones were detected, with the clips that triggered each detection, alongside a conversation type, per-speaker roles, topics, per-topic sentiment, and a summary.

Behaviors come from a catalog of presets, from definitions written for the deployment, or from both.

Velma transcribes as part of its analysis. The diarized transcript is returned as `clips`, so a separate transcription call on the same audio is redundant.

|          | Batch                                               | Streaming                                             |
| -------- | --------------------------------------------------- | ----------------------------------------------------- |
| Protocol | HTTP POST                                           | WebSocket                                             |
| Input    | A complete recording                                | A live or in-progress conversation                    |
| Output   | One JSON response                                   | Typed events as analysis develops                     |
| Use case | Post-call QA, compliance review, backlog processing | Live monitoring, real-time alerting, in-call coaching |

## Velma Triage (batch)

Returns `application/json`. `duration_ms`, `clips`, and `behaviors` are always present. The rest depend on configuration.

| Field                    | Type             | Contents                                                 |
| ------------------------ | ---------------- | -------------------------------------------------------- |
| `duration_ms`            | integer          | Total audio duration.                                    |
| `clips`                  | array            | The diarized transcript, one entry per speaker turn.     |
| `behaviors`              | array            | One entry per behavior evaluated, detected or not.       |
| `conversation_type_pick` | object or null   | The conversation type chosen for the session.            |
| `participant_role_picks` | array            | One role assignment per speaker.                         |
| `topics`                 | array of strings | Topics extracted from the conversation.                  |
| `topic_sentiments`       | array            | Per-speaker sentiment for each topic.                    |
| `summary`                | string or null   | Narrative summary. Null when `produce_summary` is false. |

Each entry in `clips` carries `clip_uuid`, `text`, `start_ms`, `duration_ms`, `speaker_label`, `language`, and the optional `emotion`, `accent`, and `deepfake_score` signals, which are null unless enabled in the `stt` block.

Each entry in `behaviors` carries `behavior_uuid`, `behavior_name`, `speaker_label`, `detected`, `confidence`, `evidence_clip_uuids`, `definitive_clip_uuid`, and `reasoning`. Entries appear for every behavior evaluated, so `detected: false` means the behavior was checked and not found.

`conversation_type_pick` and each entry in `participant_role_picks` carry a `confidence` and a `selection_source` of `inferred`, `auto_selected_single_option`, or `default`, which distinguishes a real inference from a fallback.

### Try it

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://platform.modulate.ai/api/velma-2-batch \
    -H "X-API-Key: $MODULATE_API_KEY" \
    -F "upload_file=@recording.mp3" \
    -F 'config={"behaviors":["preset:harassment","preset:service_churn"]}'
  ```

  ```python Python theme={null}
  import os, json, requests

  config = {
      "behaviors": ["preset:harassment", "preset:service_churn"],
      "stt": {"speaker_diarization": True, "emotion_signal": True},
      "produce_summary": True,
  }

  response = requests.post(
      "https://platform.modulate.ai/api/velma-2-batch",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
      data={"config": json.dumps(config)},
      files={"upload_file": open("recording.mp3", "rb")},
  )
  response.raise_for_status()
  result = response.json()

  for b in result["behaviors"]:
      if b["detected"]:
          print(f"{b['behavior_name']} - {b['speaker_label']} ({b.get('confidence')})")
  print(result.get("summary"))
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "duration_ms": 45000,
    "clips": [
      {
        "clip_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
        "text": "Thanks for calling, how can I help?",
        "start_ms": 0,
        "duration_ms": 2800,
        "speaker_label": "Speaker 1",
        "language": "en",
        "emotion": "Calm",
        "accent": null,
        "deepfake_score": null
      }
    ],
    "conversation_type_pick": {
      "conversation_type_uuid": "c1d2e3f4-a5b6-7890-cdef-123456789012",
      "name": "Customer support call",
      "confidence": 0.93,
      "selection_source": "inferred",
      "detail": "Agent greeting followed by an account issue."
    },
    "participant_role_picks": [
      {
        "speaker_label": "Speaker 1",
        "participant_role_uuid": "d2e3f4a5-b6c7-8901-def0-234567890123",
        "name": "Support agent",
        "confidence": 0.95,
        "selection_source": "inferred",
        "detail": "Opens the call and handles the request."
      }
    ],
    "behaviors": [
      {
        "behavior_uuid": "e3f4a5b6-c7d8-9012-ef01-345678901234",
        "behavior_name": "Service churn",
        "speaker_label": "Speaker 2",
        "detected": true,
        "confidence": 0.81,
        "evidence_clip_uuids": ["a1b2c3d4-e5f6-7890-abcd-ef1234567890"],
        "definitive_clip_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
        "reasoning": "The caller states they are considering cancelling."
      }
    ],
    "topics": ["billing", "account access"],
    "topic_sentiments": [
      { "topic": "billing", "speaker_label": "Speaker 2", "sentiment_score": -0.6, "sentiment_label": "negative" }
    ],
    "summary": "A customer called about a locked account and raised a billing concern."
  }
  ```
</Accordion>

### What you can configure

Configuration travels in one `config` form field, holding either the literal string `default` or a JSON-encoded object.

| `config` key                | Default               | Effect                                                                                                                                                                                             |
| --------------------------- | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `behaviors`                 | **nothing evaluated** | What to look for. Each entry is a `preset:<identifier>` reference or a full behavior definition.                                                                                                   |
| `conversation_types`        | built-in set          | Candidate conversation types. Pass `[]` for no candidates.                                                                                                                                         |
| `participant_roles`         | built-in set          | Candidate speaker roles. Pass `[]` for no candidates.                                                                                                                                              |
| `stt`                       | —                     | Transcription options: `speaker_diarization`, `emotion_signal`, `accent_signal`, `deepfake_signal`, `pii_phi_tagging`, `language`, `custom_terms`. See [Transcription](/get-started/stt) for each. |
| `produce_topics`            | `true`                | Set `false` to skip topic extraction.                                                                                                                                                              |
| `produce_topic_sentiments`  | `true`                | Set `false` to skip per-topic sentiment.                                                                                                                                                           |
| `produce_summary`           | `true`                | Set `false` to skip the narrative summary.                                                                                                                                                         |
| `default_conversation_type` | built-in fallback     | Used when no candidate applies. Set to `null` to disable the fallback.                                                                                                                             |
| `default_participant_role`  | built-in fallback     | Used when no candidate applies. Set to `null` to disable the fallback.                                                                                                                             |

<Warning>
  Passing a custom `config` without a `behaviors` key evaluates **no behaviors**. There is no implicit run-everything. This differs from `conversation_types` and `participant_roles`, which do fall back to built-in defaults when omitted.

  Omitting `config` entirely, or sending the literal `default`, loads a curated built-in configuration that does include behaviors. To evaluate against a broad set, list the `preset:<identifier>` references explicitly.
</Warning>

<Accordion title="A config with every key set">
  ```json theme={null}
  {
    "behaviors": [
      "preset:harassment",
      "preset:service_churn",
      {
        "behavior_uuid": "8f14e45f-ceea-467a-9c2b-7f1e4a1b2c3d",
        "name": "Unverified account change",
        "short_description": "Caller requests an account change without completing verification.",
        "detailed_description": "Detected when a participant asks to change account details and the agent proceeds without confirming identity through the documented verification steps.",
        "applies_to_conversation_type_uuids": null,
        "applies_to_participant_role_uuids": null
      }
    ],
    "conversation_types": [
      {
        "conversation_type_uuid": "c1d2e3f4-a5b6-7890-cdef-123456789012",
        "name": "Customer support call",
        "short_description": "Inbound support contact.",
        "detailed_description": "A customer contacts support about an existing product or account issue."
      }
    ],
    "participant_roles": [
      {
        "participant_role_uuid": "d2e3f4a5-b6c7-8901-def0-234567890123",
        "name": "Support agent",
        "short_description": "The representative handling the call.",
        "detailed_description": "Employee who greets the caller, diagnoses the issue, and takes action.",
        "applies_to_conversation_type_uuids": null
      }
    ],
    "stt": {
      "speaker_diarization": true,
      "emotion_signal": true,
      "accent_signal": false,
      "deepfake_signal": true,
      "pii_phi_tagging": true,
      "language": "en",
      "custom_terms": [
        "Modulate",
        { "term": "Velma", "definition": "Conversation intelligence model", "pronunciations": ["VEL-muh"] }
      ]
    },
    "produce_topics": true,
    "produce_topic_sentiments": true,
    "produce_summary": true,
    "default_conversation_type": {
      "conversation_type_uuid": "f4a5b6c7-d8e9-0123-4567-89abcdef0123",
      "name": "Unclassified conversation",
      "short_description": "Fallback when no candidate applies.",
      "detailed_description": "Used when none of the candidate conversation types match the audio."
    },
    "default_participant_role": {
      "participant_role_uuid": "a5b6c7d8-e9f0-1234-5678-9abcdef01234",
      "name": "Unidentified participant",
      "short_description": "Fallback when no candidate role applies.",
      "detailed_description": "Used when none of the candidate participant roles match a speaker.",
      "applies_to_conversation_type_uuids": null
    }
  }
  ```
</Accordion>

Preset identifiers come from `GET /api/velma-2-batch/list-presets`, which returns each preset's `identifier`, `name`, `short_description`, and `detailed_description`:

```bash theme={null}
curl https://platform.modulate.ai/api/velma-2-batch/list-presets \
  -H "X-API-Key: $MODULATE_API_KEY"
```

A malformed `config`, an unknown preset identifier, or a definition missing a required field is rejected with `422`.

### Audio formats

Accepted extensions: `.aac`, `.aiff`, `.flac`, `.mov`, `.mp3`, `.mp4`, `.ogg`, `.opus`, `.wav`, `.webm`.

Maximum file size is 100 MB. Empty files are rejected with `400`.

## Velma Triage (streaming)

The same analysis, emitted as typed JSON events while the conversation is still running. Connect, send the config as the first text frame, then stream audio.

| Event                | Payload                                                                                                                                    |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `partial_clip`       | An in-progress utterance with interim `emotion`, `accent`, and `deepfake_score`. Reports `end_ms`, which grows, rather than `duration_ms`. |
| `clip`               | The finalized utterance, reusing the `clip_uuid` of its partials.                                                                          |
| `clip_update`        | Refined `emotion` and `accent` for a clip that already finalized.                                                                          |
| `behavior_detection` | One behavior's verdict, with evidence clip UUIDs and reasoning.                                                                            |
| `conversation_type`  | The session's conversation type pick.                                                                                                      |
| `participant_role`   | A role pick for one speaker.                                                                                                               |
| `topics`             | The current topic list.                                                                                                                    |
| `topic_sentiment`    | Sentiment for one topic and speaker.                                                                                                       |
| `summary`            | The current summary text.                                                                                                                  |
| `done`               | `duration_ms`. The server closes after this.                                                                                               |
| `error`              | `error` string. The server closes after this.                                                                                              |

Three events have update semantics worth building for:

* `partial_clip` is a transient preview. Multiple partials share one `clip_uuid`, and the eventual `clip` reuses it, so a run of partials correlates with its final clip.
* `clip_update` can arrive any number of times for a finalized clip, always before `done`. The latest value for each field supersedes the value on the `clip` event and on any earlier update.
* `topics` and `summary` fully replace the previous event of that type. Never merge them. `topic_sentiment` supersedes an earlier event for the same topic and speaker.

### Try it

```python Python theme={null}
import os, asyncio, json, websockets

API_KEY = os.environ["MODULATE_API_KEY"]
AUDIO_FILE = "recording.mp3"
CHUNK_SIZE = 4096

config = {"behaviors": ["preset:harassment", "preset:service_churn"]}

async def analyze():
    url = f"wss://platform.modulate.ai/api/velma-2-streaming?api_key={API_KEY}"
    clips = {}

    async with websockets.connect(url) as ws:
        # The config must be the first text frame, before any audio.
        await ws.send(json.dumps(config))

        async def send():
            with open(AUDIO_FILE, "rb") as f:
                while chunk := f.read(CHUNK_SIZE):
                    await ws.send(chunk)
            await ws.send("")  # end of stream

        async def receive():
            async for message in ws:
                event = json.loads(message)
                kind = event["type"]
                if kind == "partial_clip":
                    print(f"  ... {event['partial_clip']['text']}", end="\r")
                elif kind == "clip":
                    clip = event["clip"]
                    clips[clip["clip_uuid"]] = clip
                    print(f"[{clip['start_ms']}ms] {clip['speaker_label']}: {clip['text']}")
                elif kind == "clip_update":
                    update = event["clip_update"]
                    clips.get(update["clip_uuid"], {}).update(
                        {k: v for k, v in update.items() if k != "clip_uuid"}
                    )
                elif kind == "behavior_detection":
                    d = event["detection"]
                    if d["detected"]:
                        print(f"{d['behavior_name']} - {d['speaker_label']}")
                elif kind == "summary":
                    print(f"Summary: {event['text']}")
                elif kind == "done":
                    print(f"Done: {event['duration_ms']}ms")
                    break
                elif kind == "error":
                    raise RuntimeError(event["error"])

        await asyncio.gather(send(), receive())

asyncio.run(analyze())
```

<Accordion title="Events received">
  ```json theme={null}
  { "type": "partial_clip", "partial_clip": { "clip_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", "text": "Thanks for calling, how", "start_ms": 0, "end_ms": 1400, "speaker_label": null, "emotion": null, "accent": null, "deepfake_score": null } }
  { "type": "clip", "clip": { "clip_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", "text": "Thanks for calling, how can I help?", "start_ms": 0, "duration_ms": 2800, "speaker_label": "Speaker 1", "language": "en", "emotion": null, "accent": null, "deepfake_score": null } }
  { "type": "clip_update", "clip_update": { "clip_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", "emotion": "Calm", "accent": "American" } }
  { "type": "behavior_detection", "detection": { "behavior_uuid": "e3f4a5b6-c7d8-9012-ef01-345678901234", "behavior_name": "Service churn", "speaker_label": "Speaker 2", "detected": true, "confidence": 0.81, "evidence_clip_uuids": ["a1b2c3d4-e5f6-7890-abcd-ef1234567890"], "reasoning": "The caller states they are considering cancelling." } }
  { "type": "topics", "topics": ["billing", "account access"] }
  { "type": "done", "duration_ms": 45000 }
  ```
</Accordion>

WebSocket endpoints cannot be exercised with cURL. For command-line testing use [websocat](https://github.com/vi/websocat).

### What you can configure

The config object is the same one used by batch, sent as the first text frame instead of a form field. Because it rides on that frame, `language` and `custom_terms` in the `stt` block travel with it and need no separate transport.

| Query parameter | Default            | Effect                                                                                  |
| --------------- | ------------------ | --------------------------------------------------------------------------------------- |
| `api_key`       | *required*         | The API key. WebSocket connections authenticate through the query string, not a header. |
| `audio_format`  | auto-detect        | Optional for self-describing containers. Required for raw formats.                      |
| `sample_rate`   | *required for raw* | Sample rate in Hz. Cannot be sent without `audio_format`.                               |
| `num_channels`  | *required for raw* | 1 to 8. Cannot be sent without `audio_format`.                                          |

<Warning>
  The config frame must arrive before any audio. Sending audio first closes the connection with `1003`.

  This endpoint reports authentication failure as close code `4003`. It does not use `4001`.
</Warning>

### Audio formats

**Self-describing formats.** `aac`, `aiff`, `flac`, `mp3`, `ogg`, `wav`, `webm`. Auto-detected when `audio_format` is omitted.

**Raw formats.** `s8`, `s16le`, `s16be`, `s24le`, `s24be`, `s32le`, `s32be`, `u8`, `u16le`, `u16be`, `u24le`, `u24be`, `u32le`, `u32be`, `f32le`, `f32be`, `f64le`, `f64be`, `mulaw`, `alaw`. Headerless, so `audio_format`, `sample_rate`, and `num_channels` are all required.

`sample_rate` accepts `8000`, `11025`, `16000`, `22050`, `32000`, `44100`, `48000`, `96000`.

## Behaviors

* [What are behaviors?](/velma/behaviors/what-are-behaviors)
* [Using behaviors](/velma/behaviors/using-behaviors)
* [Custom behaviors](/velma/behaviors/custom-behaviors)
* [Best practices](/velma/behaviors/best-practices)
* [Detection packages](/velma/detection-packages), ready-made behavior sets for fraud, trust and safety, compliance, and retention

## Combining with Deepfake Detection

Behavior detections say what a caller did, not whether the voice was synthetic. [Screen a call for voice fraud](/get-started/voice-fraud-screening) runs Velma Triage alongside [Deepfake Detection](/get-started/deepfake) on one recording and combines both outputs into a single decision.

## API reference

* [Velma Triage Batch](/api-reference/velma/batch)
* [Velma Triage Streaming](/api-reference/velma/streaming)
* [Behavior presets](/api-reference/velma/presets)


# Screen a call for voice fraud
Source: https://docs.modulate.ai/get-started/voice-fraud-screening

Combine Deepfake Detection and Velma Triage on one recording to decide whether a call needs review, then evaluate the result against a test matrix.

Deepfake Detection reports whether a voice is synthetic. Velma Triage reports what the caller did. Fraud screening needs both, because neither answer predicts the other.

An AI scheduling assistant calling on a customer's behalf is synthetic and legitimate. A human social engineer running an account-takeover script is natural speech and fraudulent. A cloned voice running that same script is both.

| Call                                 | Deepfake Detection verdict | Velma Triage detections                  | Screening outcome |
| ------------------------------------ | -------------------------- | ---------------------------------------- | ----------------- |
| Ordinary customer                    | `non-synthetic`            | none                                     | Proceed           |
| AI assistant acting for a customer   | `synthetic`                | none                                     | Proceed           |
| Human social engineer                | `non-synthetic`            | vishing, coercion, account impersonation | Review            |
| Cloned voice running the same script | `synthetic`                | vishing, coercion, account impersonation | Review            |

A synthetic verdict on its own separates rows one and two, which need the same handling. Behavior detections on their own separate rows three and four from the rest, but lose the escalation signal that a cloned voice carries. This page runs both models on one recording and combines the outputs into a single decision.

## Before starting

<Steps>
  <Step title="Set your API key">
    ```bash theme={null}
    export MODULATE_API_KEY=your_api_key_here
    ```

    See [Authentication](/guides/authentication) for key handling and limits.
  </Step>

  <Step title="Get a call recording in a shared format">
    Velma Triage batch accepts `.aac`, `.aiff`, `.flac`, `.mov`, `.mp3`, `.mp4`, `.ogg`, `.opus`, `.wav`, and `.webm`. Deepfake Detection batch accepts those plus `.3gp`, `.3gpp`, `.amr`, `.au`, `.m4a`, and `.wma`. Velma's list is the binding constraint for a file sent to both.

    Phone and voicemail recordings are often `.m4a`, which Velma Triage rejects. Convert first:

    ```bash theme={null}
    ffmpeg -i call.m4a -ac 1 -ar 16000 -c:a pcm_s16le call.wav
    ```

    Maximum file size is 100 MB on both endpoints.
  </Step>

  <Step title="Install the Python dependency">
    The `curl` examples need nothing else.

    ```bash theme={null}
    pip install requests
    ```
  </Step>
</Steps>

## Step 1: Classify the voice

Which endpoint classifies the voice depends on how the call was recorded.

| Recording                                                    | Endpoint                                                                                 | Attribution                                                        |
| ------------------------------------------------------------ | ---------------------------------------------------------------------------------------- | ------------------------------------------------------------------ |
| Caller-only audio, or one channel of a two-channel recording | [Deepfake Detection (batch)](/get-started/deepfake#deepfake-detection-batch)             | A verdict per 4-second frame across the clip                       |
| Mixed mono, both parties in one track                        | [Velma Triage (batch)](/get-started/velma#velma-triage-batch) with `stt.deepfake_signal` | A `deepfake_score` per transcript clip, carrying a `speaker_label` |

<Warning>
  **Worth knowing:** Deepfake Detection batch classifies single-speaker audio. On a mixed recording of a two-party call, frames cover whichever voice is speaking and no field says which. Split the channels before calling it, or take the second path and read `deepfake_score` off the clips, where Velma's diarization has already attributed each clip to a speaker.
</Warning>

The rest of this step covers the dedicated endpoint. For the transcription signal instead, skip to [Step 2](#step-2-analyze-the-behaviors) and enable `stt.deepfake_signal` there.

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://platform.modulate.ai/api/velma-2-synthetic-voice-detection-batch \
    -H "X-API-Key: $MODULATE_API_KEY" \
    -F "upload_file=@caller.wav"
  ```

  ```python Python theme={null}
  import os, requests

  response = requests.post(
      "https://platform.modulate.ai/api/velma-2-synthetic-voice-detection-batch",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
      files={"upload_file": open("caller.wav", "rb")},
  )
  response.raise_for_status()
  frames = response.json()["frames"]
  ```
</CodeGroup>

See the [Deepfake Detection](/get-started/deepfake#deepfake-detection-batch) page for the full response shape.

### Aggregate the frames into one verdict

The response is a timeline, not a clip-level verdict. Screening needs one value, so compute it:

* Drop `no-content` frames. They are silence, classified before inference rather than by the model, and counting them dilutes the result on a call with hold time.
* Weight by frame duration rather than counting frames, so a trimmed final frame does not carry the same weight as a full 4-second one.
* Divide synthetic speech time by total speech time.
* Compare against a threshold.

```python theme={null}
def clip_verdict(frames, threshold=0.5):
    """Collapse per-frame verdicts into one clip-level verdict and a share."""
    speech_ms = synthetic_ms = 0
    for frame in frames:
        if frame["verdict"] == "no-content":
            continue
        duration = frame["end_time_ms"] - frame["start_time_ms"]
        speech_ms += duration
        if frame["verdict"] == "synthetic":
            synthetic_ms += duration

    if speech_ms == 0:
        return "inconclusive", 0.0

    share = synthetic_ms / speech_ms
    return ("synthetic" if share >= threshold else "non-synthetic"), share


verdict, share = clip_verdict(frames)
print(f"{verdict}  ({share:.0%} of speech)")
```

<Note>
  **Worth knowing:** `0.5` is a starting point, not a value the API defines. Real recordings carry IVR prompts, hold music, and transfers, so a partial share is common and a clip is rarely 0% or 100%. Set the threshold against your own labelled audio and your tolerance for false positives.
</Note>

Recommended clip length for this endpoint is 4 to 60 seconds. On a long call, `frames` still covers the full duration, and the per-frame timestamps locate a synthetic segment rather than only reporting that one exists.

## Step 2: Analyze the behaviors

Velma Triage evaluates a configured set of behaviors against the transcript. Nothing is evaluated unless the `behaviors` array names it, so fraud screening starts by choosing which signals to turn on.

These seven presets from the [Fraud Detection and Prevention](/velma/detection-packages/fraud-detection-and-prevention) package are the fraud tactics themselves:

| Preset                           | Tactic                                            |
| -------------------------------- | ------------------------------------------------- |
| `preset:vishing`                 | Eliciting sensitive information through deception |
| `preset:account_impersonation`   | Claiming to be another account holder             |
| `preset:coercion_manipulation`   | Pressure through intimidation or threats          |
| `preset:bargaining_manipulation` | Pressure through cajoling and persuasion          |
| `preset:feigned_ignorance`       | Feigned confusion to draw sympathy                |
| `preset:return_fraud_attempt`    | Fraudulent product return                         |
| `preset:ai_agent_manipulation`   | Pushing an AI agent into unintended behavior      |

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://platform.modulate.ai/api/velma-2-batch \
    -H "X-API-Key: $MODULATE_API_KEY" \
    -F "upload_file=@call.wav" \
    -F 'config={"behaviors":["preset:vishing","preset:account_impersonation","preset:coercion_manipulation","preset:bargaining_manipulation","preset:feigned_ignorance","preset:return_fraud_attempt","preset:ai_agent_manipulation"],"stt":{"deepfake_signal":true}}'
  ```

  ```python Python theme={null}
  import os, json, requests

  FRAUD_TACTICS = [
      "preset:vishing",
      "preset:account_impersonation",
      "preset:coercion_manipulation",
      "preset:bargaining_manipulation",
      "preset:feigned_ignorance",
      "preset:return_fraud_attempt",
      "preset:ai_agent_manipulation",
  ]

  config = {
      "behaviors": FRAUD_TACTICS,
      "stt": {"speaker_diarization": True, "deepfake_signal": True},
  }

  response = requests.post(
      "https://platform.modulate.ai/api/velma-2-batch",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
      data={"config": json.dumps(config)},
      files={"upload_file": open("call.wav", "rb")},
  )
  response.raise_for_status()
  result = response.json()

  clips = {c["clip_uuid"]: c for c in result["clips"]}

  for behavior in result["behaviors"]:
      if not behavior["detected"]:
          continue
      print(f"{behavior['behavior_name']}  {behavior['speaker_label']}  {behavior.get('confidence')}")
      print(f"  {behavior.get('reasoning')}")
      for clip_uuid in behavior.get("evidence_clip_uuids", []):
          print(f"  [{clips[clip_uuid]['start_ms']}ms] {clips[clip_uuid]['text']}")
  ```
</CodeGroup>

Every configured behavior comes back whether or not it fired, so `detected: false` means Velma checked and found nothing, not that the check was skipped. `evidence_clip_uuids` and `definitive_clip_uuid` index into `clips`, which is how the snippet above prints the words that triggered each detection. See [Velma Triage](/get-started/velma#velma-triage-batch) for the full response shape.

<Note>
  **Worth knowing:** the fraud package holds 18 presets, and 11 of them are context and outcome signals rather than fraud tactics. `preset:complaints`, `preset:issue_resolved`, and `preset:refund_or_credit_issued` describe how a call went, and firing on them is not evidence of fraud. Split the list by what each signal means before wiring detections to an action.
</Note>

### Move to the full package

The seven presets above run without any other configuration. The full package adds 25 conversation types and 14 participant roles, which constrain what Velma infers about the call and improve role attribution on support calls. Download `fraud-detection-and-prevention.json` from the [package page](/velma/detection-packages/fraud-detection-and-prevention) and send it as the `config` value:

```bash theme={null}
curl -X POST https://platform.modulate.ai/api/velma-2-batch \
  -H "X-API-Key: $MODULATE_API_KEY" \
  -F "upload_file=@call.wav" \
  -F "config=<fraud-detection-and-prevention.json"
```

Preset identifiers resolve server-side against the catalog your organization has access to. An unknown identifier is rejected with `422`, so confirm the set available to you before pinning a list:

```bash theme={null}
curl https://platform.modulate.ai/api/velma-2-batch/list-presets \
  -H "X-API-Key: $MODULATE_API_KEY"
```

## Step 3: Combine the two outputs

Each model contributes one axis of the decision.

| Deepfake Detection verdict | Fraud-tactic detections | Screening outcome                                  |
| -------------------------- | ----------------------- | -------------------------------------------------- |
| `non-synthetic`            | none                    | Proceed                                            |
| `synthetic`                | none                    | Proceed, and record the synthetic verdict          |
| `non-synthetic`            | one or more             | Route to review                                    |
| `synthetic`                | one or more             | Route to review, ahead of the natural-speech cases |

```python theme={null}
def screen(frames, velma_result):
    """Combine both outputs. Assumes the Velma config enabled fraud tactics only."""
    voice, share = clip_verdict(frames)
    tactics = [b["behavior_name"] for b in velma_result["behaviors"] if b["detected"]]

    return {
        "voice": voice,
        "synthetic_share": round(share, 3),
        "tactics": tactics,
        "action": "review" if tactics else "proceed",
        "priority": "high" if tactics and voice == "synthetic" else "normal",
        "summary": velma_result.get("summary"),
    }
```

Every detection counts as a tactic here because the config enabled nothing else. Running the full package mixes tactics with context and outcome signals, so keep the subset you treat as a tactic and filter on `behavior_uuid`, which is stable across catalog updates. Filtering on `behavior_name` breaks when a display name is reworded.

<Accordion title="Screening output for a cloned voice running an account-takeover script">
  ```json theme={null}
  {
    "voice": "synthetic",
    "synthetic_share": 0.94,
    "tactics": ["Vishing", "Account Impersonation", "Coercion Manipulation"],
    "action": "review",
    "priority": "high",
    "summary": "A caller reported being locked out, declined to complete verification, asked the agent to read back a one-time code, and threatened a complaint when refused."
  }
  ```
</Accordion>

Behavior detections are signals, not verdicts. A single detection at moderate confidence is weaker evidence than three tactics firing together with the evidence clips agreeing. Weigh several detections against your own rules before acting on any of them.

## Build a test matrix

Two synthetic recordings are enough to see that behavior analysis separates them, and not enough to set a threshold. Cover all four quadrants, because each one fails differently.

|                      | Benign intent                      | Fraudulent intent             |
| -------------------- | ---------------------------------- | ----------------------------- |
| **Natural speech**   | Ordinary customer                  | Human social engineer         |
| **Synthetic speech** | AI assistant acting for a customer | Cloned voice running a script |

<Steps>
  <Step title="Record a natural-speech control">
    Ten seconds of your own voice, converted with the `ffmpeg` command above. Deepfake Detection should return `non-synthetic` across the speech frames. A control that comes back `synthetic` means the format or the pipeline needs checking before any other result is worth reading.
  </Step>

  <Step title="Generate synthetic samples matching your threat model">
    Text-to-speech output exercises the detection path but is not what a motivated attacker uses. Test against the voice-cloning tools relevant to the accounts you protect.
  </Step>

  <Step title="Use real call recordings">
    Consented, compliant recordings from your own environment carry the codecs, channel counts, hold music, and transfers that synthetic test clips do not. Thresholds tuned on clean studio audio move once real calls arrive.
  </Step>

  <Step title="Include benign fraud-adjacent calls">
    A frustrated customer who cannot remember which email they signed up with produces some of the same acoustic markers as feigned ignorance. These calls set your false-positive rate.
  </Step>
</Steps>

## Before trusting the result

Concurrency is capped per model, and the default is 3 in flight against one endpoint. Screening a backlog runs two endpoints, so bound each with its own semaphore rather than retrying into a full queue. A rejected request returns `429` on both, and the `detail` field distinguishes a full queue from exhausted credits. Match on `detail` rather than on the status alone. See [Limits](/guides/authentication#limits).

Deepfake Detection classifies acoustically and reports no reason for its verdict. Velma Triage returns `reasoning` and evidence clips for every detection, which is what an analyst reviewing a flagged call can act on. Route synthetic-voice verdicts to a queue where a human sees the transcript, not to an automatic block.

## Related

* [Deepfake Detection](/get-started/deepfake), both endpoints and how the frame verdicts compare with the transcription signal
* [Velma Triage](/get-started/velma), the full configuration reference for batch and streaming
* [Fraud Detection and Prevention](/velma/detection-packages/fraud-detection-and-prevention), all 18 presets with detection criteria
* [Using behaviors](/velma/behaviors/using-behaviors), listing the catalog and writing custom definitions
* [Which API should I use?](/guides/which-api), the choice between endpoints that return overlapping signals


# Authentication and rate limits
Source: https://docs.modulate.ai/guides/authentication

How to pass a Modulate API key, which limits apply per model, and what each auth and limit error means.

Modulate exposes two API surfaces with different authentication schemes. The **Models API** covers every model endpoint documented on this site. The **Modulate Platform API** is a separate early-access orchestration surface, covered at the end of this page.

| Surface               | Host                               | Auth                                                             |
| --------------------- | ---------------------------------- | ---------------------------------------------------------------- |
| Models API            | `platform.modulate.ai`             | `X-API-Key` header (HTTP), `api_key` query parameter (WebSocket) |
| Modulate Platform API | `cloud-processing-api.modulate.ai` | `accountuuid` + `apikey` headers                                 |

## API keys

Generate keys from the [API Keys page](https://platform.modulate.ai/dashboard/api-keys) in the dashboard.

A key belongs to an organization. Model access and usage limits are properties of the organization, not of the individual key, and they are fixed once the key is issued.

<Frame>
  <img alt="API key creation screen" />
</Frame>

## Passing the key

HTTP endpoints take the key as a request header:

```http theme={null}
X-API-Key: YOUR_API_KEY
```

WebSocket endpoints take it as a query parameter on the connection URL. A header cannot be used, because the key must be present during the handshake:

```text theme={null}
wss://platform.modulate.ai/api/velma-2-stt-streaming?api_key=YOUR_API_KEY
```

This applies to every Models API endpoint. HTTP endpoints are the `/api/velma-2-*-batch` paths plus `/api/velma-2-batch`; WebSocket endpoints are the `/api/velma-2-*-streaming` paths plus `/api/velma-2-streaming`.

<Warning>
  API keys in WebSocket URLs can appear in server access logs and proxy logs. Avoid logging or persisting the full connection URL.
</Warning>

## Limits

Concurrency is capped **per model**: the number of requests or connections in flight at the same time against one endpoint. The default is 3. Reaching it on one model does not affect any other model.

Credits are separate. A request is rejected for insufficient credits whatever the concurrency situation.

Concurrency ceilings are set by Modulate per organization. To raise one, [contact us](/support#contact-us) with the model and the traffic you expect.

## Error responses

### Authentication

| Status | Meaning                                                                                             |
| ------ | --------------------------------------------------------------------------------------------------- |
| `401`  | The key is missing or invalid.                                                                      |
| `403`  | The key is valid but the request is not permitted, or the organization has no access to this model. |

Which of the two an endpoint returns is not uniform. Several endpoints, including Multilingual Transcription (batch), Deepfake Detection (batch), and PII/PHI Redaction (batch), return `403` for an invalid key rather than `401`. Read the `detail` field rather than branching on the status alone. The per-endpoint reference page lists the exact set each endpoint returns.

On WebSocket endpoints the handshake closes instead of returning a status:

| Close code | Meaning                                                                                                                                     |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `4001`     | The `api_key` parameter is missing or invalid. Sent by English Fast Transcription (streaming) and AI Music Detection (streaming).           |
| `4003`     | The request is not permitted. On the other streaming endpoints this also covers authentication failure, which they do not report as `4001`. |
| `4004`     | The key has no access to this model. Sent by English Fast Transcription (streaming) and AI Music Detection (streaming).                     |

### Limits

Every rejected-for-limits request returns `429`. The `detail` field names the cause:

| `detail`                                                                                 | Cause                                | Retry                        |
| ---------------------------------------------------------------------------------------- | ------------------------------------ | ---------------------------- |
| `Concurrent request limit reached. Please retry after your in-flight requests complete.` | The model's concurrency cap is full. | Yes, after a short delay.    |
| `Insufficient credits.`                                                                  | The organization is out of credits.  | Not until credits are added. |

On WebSocket endpoints these arrive as close codes `4030` and `4029`.

Most endpoints report both conditions as `Insufficient credits.` and close code `4029`. The split above is documented on English Fast Transcription (batch and streaming), Multilingual Fast Transcription (batch), and AI Music Detection (batch and streaming). Match on the `detail` string rather than on `429` alone.

For a backlog of files, bound the work with a semaphore rather than retrying into a full queue:

```python theme={null}
import asyncio

MAX_CONCURRENT = 3  # the model's concurrency cap
semaphore = asyncio.Semaphore(MAX_CONCURRENT)

async def transcribe_file(session, filepath):
    async with semaphore:
        ...
```

## Modulate Platform API

The Modulate Platform API (`cloud-processing-api.modulate.ai`) submits jobs that combine transcription with optional analysis features such as emotion, demographics, deepfake detection, and behavioral insights. It authenticates differently from the Models API.

<Warning>
  The Platform API is in **early access**. Any part of the contract can change before 1.0.0.
</Warning>

Every request carries two headers:

```http theme={null}
accountuuid: YOUR_ACCOUNT_UUID
apikey: YOUR_API_KEY
```

`accountuuid` comes from an account administrator or the Platform dashboard. `apikey` is a Platform key, issued separately from Models API keys.

A single `POST /api_service` endpoint accepts three submission patterns:

* **Real-time WebSocket**: `submission_type: "realtime_websocket"` with no files. The response carries a `realtime_url` for streaming through the [Pipecat Client SDK](https://docs.pipecat.ai/client/js/introduction).
* **Single-file batch**: one audio file of 5 MB or less in a single POST.
* **Multi-file batch**: one POST per file sharing a `job_id`, with `finalize_job: true` on the last.

All three then poll `GET /api_service/job_status/{job_id}` until `status='completed'`.

## Related

* [Troubleshooting](/guides/troubleshooting)


# Troubleshooting
Source: https://docs.modulate.ai/guides/troubleshooting

Common errors by category, with causes and fixes for auth, rate limits, audio validation, timeouts, and server errors.

Common errors organized by category, with causes and fixes.

## Authentication errors

### `401 Unauthorized`

**Cause:** No `X-API-Key` header was sent, or the header name is wrong.

**Fix:** Add the header to your request:

```bash theme={null}
curl -H "X-API-Key: your_api_key_here" ...
```

Check the exact casing: `X-API-Key`, not `x-api-key` or `Authorization`.

### `403 Forbidden` (REST)

Three different problems return `403`:

| Scenario                                      | Response detail                                      | Fix                                                                                                                                                           |
| --------------------------------------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| The request is not permitted                  | `"This request is not permitted."`                   | Confirm the key is valid and active on the [API Keys page](https://platform.modulate.ai/dashboard/api-keys)                                                   |
| The model is not enabled for the organization | `"This API key does not have access to this model."` | Check the access on the [API Keys page](https://platform.modulate.ai/dashboard/api-keys), otherwise contact [support@modulate.ai](mailto:support@modulate.ai) |

Several endpoints, including Multilingual Transcription (batch), Deepfake Detection (batch), and PII/PHI Redaction (batch), also return `403` for an invalid key where others return `401`. Read `detail` rather than branching on the status.

### WebSocket close code `4001` (English Fast Transcription streaming, AI Music Detection streaming)

**Cause:** The `api_key` query parameter is missing or contains an invalid key.

**Fix:** Pass the key at connection time:

```text theme={null}
wss://platform.modulate.ai/api/velma-2-stt-streaming-english-v2?api_key=your_api_key_here&audio_format=ogg
```

The other streaming endpoints do not use `4001`. They report an authentication failure as `4003`.

### WebSocket close code `4003`

**Cause:** Authentication succeeded but the model is not enabled for your organization.

**Fix:** Check [the limits specified for your API key](https://platform.modulate.ai/dashboard/api-keys); otherwise contact [support@modulate.ai](mailto:support@modulate.ai).

## Rate limit errors

### `429 Too Many Requests` (REST)

**Cause:** You've exceeded the concurrent request limit for that model.

**Fix:** Implement exponential backoff with jitter and retry:

```python theme={null}
import time, random, httpx

def post_with_retry(url, files, headers, max_retries=5):
    for attempt in range(max_retries):
        response = httpx.post(url, files=files, headers=headers)
        if response.status_code != 429:
            return response
        wait = (2 ** attempt) + random.uniform(0, 1)
        time.sleep(wait)
    return response
```

Persistent `429`s mean the traffic pattern exceeds the model's concurrency cap rather than spiking past it. Bound the work with a semaphore, or [contact us](/support#contact-us) to raise the cap. See [Limits](/guides/authentication#limits).

### WebSocket close codes `4029` and `4030`

**Cause:** `4030` means the model's concurrency cap is full. `4029` means insufficient credits. Most streaming endpoints report both as `4029`.

**Fix:** For a concurrency cap, reduce simultaneous connections and reconnect after a delay. For credits, check the balance in the dashboard. Neither resolves by retrying immediately.

## Audio validation errors

### `400 Bad Request`, unsupported format

**Cause:** The audio file format isn't supported by the endpoint you're calling.

| Endpoint                                                                                                                                                         | Common mistake                                                                  | Fix                                                                                                  |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| Velma Triage, Multilingual Transcription, Multilingual Fast Transcription, PII/PHI Redaction, Emotion Detection, Accent Detection, Audio Event Detection (batch) | Sending a file outside the 10 extensions these accept                           | Convert to one of `.aac`, `.aiff`, `.flac`, `.mov`, `.mp3`, `.mp4`, `.ogg`, `.opus`, `.wav`, `.webm` |
| Velma Triage, Multilingual Transcription, PII/PHI Redaction (streaming)                                                                                          | Sending `m4a`, `mp4`, `opus`, or another container outside the 7 these accept   | Convert to `wav`, `mp3`, `ogg`, `flac`, `webm`, `aac`, `aiff`, or to raw PCM                         |
| Deepfake Detection, Music & Speech Detection, AI Music Detection, English Fast Transcription (streaming)                                                         | Sending an MP4-family value (`mp4`, `m4a`, `3gp`, …) in a non-streamable layout | Re-mux with a streamable (faststart) layout, or send raw PCM                                         |
| Any streaming endpoint                                                                                                                                           | Declaring a raw PCM `audio_format` while sending container bytes                | Match the declared format to the bytes, or drop `audio_format` where the endpoint auto-detects       |

The accepted set differs per endpoint. Each capability page lists its own in full.

### `422 Unprocessable Entity`

**Cause:** On Deepfake Detection, the audio is shorter than 0.5 seconds. On other endpoints, `422` means a required request field is missing or malformed, most often the `X-API-Key` header or the `upload_file` part. Velma Triage also returns `422` for a malformed `config`, an unknown preset identifier, or a definition missing a required field.

Empty files, and files whose content is only metadata with no audio samples, also trigger this.

**Fix:** Verify the actual audio duration not just the file size. Silent files or files where the audio track was stripped can report a non-zero duration but contain no usable samples. Use ffprobe to inspect:

```bash theme={null}
ffprobe -v quiet -show_entries format=duration -of csv=p=0 yourfile.wav
```

### WebSocket close code `1003`, invalid query parameters

**Cause:** `audio_format`, `sample_rate`, or `num_channels` is missing, misspelled, or unsupported. On Deepfake Detection, Music & Speech Detection, AI Music Detection, and English Fast Transcription streaming, omitting `audio_format` alone triggers this, because those four require it on every connection.

**Fix:** Check that:

* `audio_format` is in the endpoint's accepted list. Each capability page states it.
* `sample_rate` is one of `8000`, `11025`, `16000`, `22050`, `32000`, `44100`, `48000`, `96000`, and is sent only with a raw format.
* `num_channels` is between 1 and 8, and is sent only with a raw format.

Sending `sample_rate` or `num_channels` for a container format is itself an error on the endpoints that reject it.

### WebSocket close code `4002`, audio could not be decoded

**Cause:** The audio could not be decoded, or the raw PCM bytes you're sending don't match the `audio_format`, `sample_rate`, or `num_channels` you declared at connection time.

**Fix:** If sending a container format (e.g. WebM, Ogg), make sure the stream is valid and not truncated. If sending raw PCM, confirm that your encoding pipeline produces exactly the format you declared. If you resampled to 16000 Hz but declared `sample_rate=44100`, the model will receive malformed frames.

## Timeout errors

### `504 Gateway Timeout`

**Cause:** Batch processing exceeded 60 seconds. This is uncommon for typical audio files but can happen with very long recordings or during periods of high server load. It can also happen if the file is above the maximum recommended file size of 100 MB.

**Fix:**

* Verify your file is within the recommended length range.
* Verify your file is under 100 MB.
* If the issue is persistent on files that should process quickly, email [support@modulate.ai](mailto:support@modulate.ai) with the file (if possible), the file type, file duration, file size, and endpoint.
* For long recordings, consider splitting into smaller chunks.

### English Fast Transcription timeouts

If `504` recurs on this endpoint, check that the file is not corrupted or padded with long stretches of silence, and that it is within the 100 MB limit.

## Server errors

### `502 Bad Gateway` (Multilingual Fast Transcription batch)

**Cause:** Transcription is temporarily unavailable.

**Fix:** Retry with exponential backoff. The response `detail` field is intentionally generic. If the error persists, email [support@modulate.ai](mailto:support@modulate.ai).

### `503 Service Unavailable`

**Cause:** The inference server is temporarily overloaded.

**Fix:** Retry with exponential backoff. Do not hammer the endpoint with immediate retries as this worsens the overload. See the retry pattern under [Rate limit errors](#429-too-many-requests-rest) above.

## Still stuck?

If none of the above matches your situation, email [support@modulate.ai](mailto:support@modulate.ai) with:

* Endpoint URL
* Full request headers (**redact your API key**)
* Response body and status code
* Audio format, duration, and file size


# Which API should I use?
Source: https://docs.modulate.ai/guides/which-api

Modulate's model families, what each model returns, and how to choose between endpoints that overlap.

Modulate's models are grouped into families by the kind of output they produce. Within a family, endpoints differ by language coverage, latency, and whether they take a file or a live stream.

Each capability page below carries the full parameter, response, and audio format detail for its endpoints.

## Model families

| Family            | Model                                                       | Output                                                                                                                                    |
| ----------------- | ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| **Transcription** | [Multilingual Transcription](/get-started/stt)              | A transcript with per-utterance timing, speaker labels, and optional emotion, accent, deepfake, and PII/PHI signals. Batch and streaming. |
|                   | [English Fast Transcription](/get-started/stt)              | An English transcript, with speaker turns when diarization is enabled. Batch and streaming.                                               |
|                   | [Multilingual Fast Transcription](/get-started/stt)         | A transcript, its duration, and its language. No metadata. Batch.                                                                         |
| **Detection**     | [Deepfake Detection](/get-started/deepfake)                 | A `synthetic`, `non-synthetic`, or `no-content` verdict per frame, with confidence. Batch and streaming.                                  |
|                   | [Emotion Detection](/get-started/emotion)                   | A whole-file emotion label plus a per-window time series. Batch.                                                                          |
|                   | [Accent Detection](/get-started/accent)                     | A whole-file accent label plus a per-window time series. Batch.                                                                           |
|                   | [Music & Speech Detection](/get-started/music-detection)    | Music and speech probabilities per 192 ms frame, plus clip totals. Batch and streaming.                                                   |
|                   | [AI Music Detection](/get-started/ai-music-detection)       | A clip verdict on AI-generated music, plus per-window vocal and instrumental scores. Batch and streaming.                                 |
|                   | [Language Detection](/get-started/language-detection)       | The spoken language with a confidence score, across 100 languages. Batch.                                                                 |
|                   | [Audio Event Detection](/get-started/audio-event-detection) | A probability for each of 42 non-speech sound events. Batch.                                                                              |
| **Redaction**     | [PII/PHI Redaction](/get-started/pii)                       | A redacted transcript plus audio with the sensitive ranges silenced. Batch and streaming.                                                 |
| **Triage**        | [Velma Triage](/get-started/velma)                          | Behaviors, conversation type, participant roles, topics, sentiment, a summary, and a diarized transcript. Batch and streaming.            |

No Detection model returns a transcript.

## Endpoints that overlap

### Three models return a transcript

Multilingual Transcription, PII/PHI Redaction, and Velma Triage all return a transcript. Pairing any of the last two with a transcription call returns the same text twice.

| Requirement                                                        | Endpoint that covers it                                                 |
| ------------------------------------------------------------------ | ----------------------------------------------------------------------- |
| A redacted transcript and audio with the sensitive ranges silenced | PII/PHI Redaction. A separate transcription call returns the same text. |
| Behaviors, topics, and sentiment across a conversation             | Velma Triage. The diarized transcript comes back as `clips`.            |

### Four signals have two paths

Each is available as a flag on Multilingual Transcription, attached per utterance, or as a dedicated endpoint that produces no transcript. English Fast and Multilingual Fast Transcription carry none of them.

| Signal   | Multilingual Transcription flag                             | Dedicated endpoint | What the dedicated endpoint adds                                                                         |
| -------- | ----------------------------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------- |
| Deepfake | `deepfake_signal=true`, one score per utterance             | Deepfake Detection | Frame-level verdicts, explicit `no-content` for silence, and verdicts on live audio without a transcript |
| Emotion  | `emotion_signal=true`, one label per utterance              | Emotion Detection  | A whole-file label and a per-window time series, without a transcript                                    |
| Accent   | `accent_signal=true`, one label per utterance               | Accent Detection   | A whole-file label and a per-window time series, without a transcript                                    |
| PII/PHI  | `pii_phi_tagging=true`, spans tagged in the transcript text | PII/PHI Redaction  | Silenced audio, not only tagged text                                                                     |

## Common scenarios

| Goal                                                              | Endpoint                                                                                                                                                              |
| ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Recorded English calls, high volume                               | English Fast Transcription (batch). Bound parallel requests with a semaphore to stay inside the concurrency cap.                                                      |
| Recorded calls in mixed or non-English languages, transcript only | Multilingual Fast Transcription (batch). Pass `language` when it is known.                                                                                            |
| Captions, or search that seeks into the audio                     | English Fast Transcription (batch) with `time_stamps=true`. Returns a start time, end time, and alignment confidence per word, in seconds from the start of the file. |
| Meeting transcription with speakers and emotion                   | Multilingual Transcription (batch) with `speaker_diarization=true` and `emotion_signal=true`.                                                                         |
| Live captions with speaker labels or non-English audio            | Multilingual Transcription (streaming).                                                                                                                               |
| Voice agent input, minimum latency                                | English Fast Transcription (streaming) with `endpointing=true`. Replace the displayed partial on each message rather than appending.                                  |
| Screening a submitted clip for AI-generated voice                 | Deepfake Detection (batch).                                                                                                                                           |
| Anti-spoofing during a live voice-authentication flow             | Deepfake Detection (streaming). Frame verdicts arrive during the call.                                                                                                |
| A compliance recording that must be shareable                     | PII/PHI Redaction (batch). Returns the silenced MP3 and the tagged transcript together.                                                                               |
| Routing audio by spoken language                                  | Language Detection (batch). Read `predicted_language_code`; low `confidence` means no commitment.                                                                     |
| Processing only speech from a live stream                         | Music & Speech Detection (streaming). Route on the frame classification.                                                                                              |
| Screening uploaded tracks for AI-generated music                  | AI Music Detection (batch). The clip `primary_verdict` is more accurate than the per-window scores.                                                                   |
| Identifying non-speech sounds in a clip                           | Audio Event Detection (batch). Rank the 41 shared keys to find the most prominent event; threshold `cry` on its own.                                                  |
| Fraud or compliance review on recorded calls                      | Velma Triage (batch) with a [detection package](/velma/detection-packages).                                                                                           |
| Live monitoring for escalation or churn                           | Velma Triage (streaming). `behavior_detection` events arrive during the call.                                                                                         |

## Related

* [Authentication and rate limits](/guides/authentication)
* [Transcription signals](/get-started/stt#signals)


# Modulate developer docs
Source: https://docs.modulate.ai/index

Build with Modulate's voice AI platform. Start with Velma Triage for whole-conversation analysis, or call individual models for transcription, detection, and redaction.

Every Modulate model is audio-native. Each one analyzes the acoustic signal rather than working from transcribed words alone, which is what lets them report tone, accent, synthetic speech, and music alongside what was said.

## Velma Triage

<Card title="Velma Triage: conversation analysis" href="/velma/overview">
  Detect behaviors, classify conversations, identify participant roles, and extract topics with sentiment. Available as a single API call or a real-time stream.
</Card>

## Models

Models are grouped by the kind of output they produce. [Which API should I use?](/guides/which-api) covers the choice between endpoints in detail, including the cases where one call covers two needs.

### Transcription

<CardGroup>
  <Card title="Transcription" icon="microphone" href="/get-started/stt">
    Three models across batch and streaming. Speaker diarization, per-utterance timing, and optional emotion, accent, deepfake, and PII/PHI signals.
  </Card>
</CardGroup>

### Detection

<CardGroup>
  <Card title="Deepfake Detection" icon="shield-halved" href="/get-started/deepfake">
    Per-frame synthetic-voice verdicts on files or live audio.
  </Card>

  <Card title="Emotion Detection" icon="face-smile" href="/get-started/emotion">
    A whole-file emotion label plus a per-window time series.
  </Card>

  <Card title="Accent Detection" icon="globe" href="/get-started/accent">
    A whole-file accent label plus a per-window time series.
  </Card>

  <Card title="Music & Speech Detection" icon="music" href="/get-started/music-detection">
    Frame-level music and speech probabilities.
  </Card>

  <Card title="AI Music Detection" icon="robot" href="/get-started/ai-music-detection">
    Whether a track contains AI-generated vocals or instrumentals.
  </Card>

  <Card title="Language Detection" icon="language" href="/get-started/language-detection">
    The spoken language of a clip, with a confidence score, across 100 languages.
  </Card>

  <Card title="Audio Event Detection" icon="waveform-lines" href="/get-started/audio-event-detection">
    A probability for each of 42 non-speech sound events, from instruments to gunshots.
  </Card>
</CardGroup>

### Redaction

<CardGroup>
  <Card title="PII/PHI Redaction" icon="eye-slash" href="/get-started/pii">
    A redacted transcript plus audio with the sensitive ranges silenced. Batch and streaming.
  </Card>
</CardGroup>

## New here?

<Card title="Quick start" icon="rocket" href="/quickstart">
  Make your first API call in under five minutes. No SDK required.
</Card>

## What you can build

* **Meeting transcription.** Multilingual transcripts with speaker labels, timestamps, and optional emotion or accent signals.
* **Live captions.** Stream audio over WebSocket and render utterances as they are spoken.
* **Voice agents.** Sub-two-second partial transcripts with utterance segmentation at pauses.
* **Anti-spoofing.** Real-time deepfake verdicts during a voice authentication flow.
* **Compliance archives.** Shareable recordings with PII/PHI removed from the transcript and silenced in the audio.
* **Call QA and coaching.** Behavior detections, topics, sentiment, and a summary across a whole conversation.
* **Content moderation.** Frame-by-frame music and speech classification at scale.
* **Catalog screening.** Batch checks for AI-generated music or AI-generated voice in uploaded audio.
* **Language routing.** Identify the spoken language and send audio to the matching pipeline.


# Quick start
Source: https://docs.modulate.ai/quickstart

Send an audio file to Multilingual Transcription and read the response, in about five minutes.

This page goes from nothing to a transcript. The same shape applies to every other Modulate model: one key, one request, one response.

## Before starting

<Steps>
  <Step title="Get an API key">
    [Create a free account](https://platform.modulate.ai/signup-request), then create a key from the **API Keys** tab in the dashboard.
  </Step>

  <Step title="Store the key">
    Set it as an environment variable rather than hard-coding it.

    ```bash theme={null}
    export MODULATE_API_KEY=your_api_key_here
    ```
  </Step>

  <Step title="Get a sample audio file">
    Any speech clip of 5 to 30 seconds works. The examples below assume `audio.mp3` in the working directory.
  </Step>
</Steps>

The `curl` example needs nothing else. The Python example needs `requests`:

```bash theme={null}
pip install requests
```

## Make the first call

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://platform.modulate.ai/api/velma-2-stt-batch \
    -H "X-API-Key: $MODULATE_API_KEY" \
    -F "upload_file=@audio.mp3"
  ```

  ```python Python theme={null}
  import os, requests

  response = requests.post(
      "https://platform.modulate.ai/api/velma-2-stt-batch",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
      files={"upload_file": open("audio.mp3", "rb")},
  )
  response.raise_for_status()
  print(response.json()["text"])
  ```

  ```javascript JavaScript theme={null}
  import fs from "fs";
  import FormData from "form-data";

  const form = new FormData();
  form.append("upload_file", fs.createReadStream("audio.mp3"), { filename: "audio.mp3" });

  const response = await fetch("https://platform.modulate.ai/api/velma-2-stt-batch", {
    method: "POST",
    headers: { "X-API-Key": process.env.MODULATE_API_KEY, ...form.getHeaders() },
    body: form,
  });

  const result = await response.json();
  console.log(result.text);
  ```
</CodeGroup>

The response is `application/json`.

<Accordion title="Response">
  ```json theme={null}
  {
    "text": "Hello everyone. Welcome to the meeting. We'll be discussing results today.",
    "duration_ms": 8400,
    "utterances": [
      {
        "utterance_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
        "text": "Hello everyone. Welcome to the meeting.",
        "start_ms": 0,
        "duration_ms": 4200,
        "speaker": 1,
        "language": "en",
        "emotion": null,
        "accent": null,
        "deepfake_score": null
      },
      {
        "utterance_uuid": "b2c3d4e5-f6a7-8901-bcde-f23456789012",
        "text": "We'll be discussing results today.",
        "start_ms": 4200,
        "duration_ms": 4200,
        "speaker": 1,
        "language": "en",
        "emotion": null,
        "accent": null,
        "deepfake_score": null
      }
    ]
  }
  ```
</Accordion>

`text` is the full transcript. `utterances` breaks it into speaker turns with millisecond timing and a language detected per utterance. `emotion`, `accent`, and `deepfake_score` are `null` until those signals are enabled. See [Transcription](/get-started/stt).

Response formats differ across the API. Most endpoints return JSON, [PII/PHI Redaction](/get-started/pii) batch returns `multipart/form-data`, and the streaming endpoints return sequences of JSON messages, some interleaved with binary audio frames. Each capability page states the format for its endpoints.

## Go deeper by capability

Each page states what the endpoint returns, what can be configured, the accepted audio formats, and a runnable call for every endpoint in that family.

<CardGroup>
  <Card title="Velma Triage" icon="comments" href="/get-started/velma">
    Behaviors, topics, sentiment, and a summary across a whole conversation.
  </Card>

  <Card title="Transcription" icon="microphone" href="/get-started/stt">
    Multilingual, English Fast, and Multilingual Fast. Batch and streaming.
  </Card>

  <Card title="Deepfake Detection" icon="shield-halved" href="/get-started/deepfake">
    Per-frame synthetic-voice verdicts on files or live audio.
  </Card>

  <Card title="PII/PHI Redaction" icon="eye-slash" href="/get-started/pii">
    A redacted transcript plus audio with the sensitive ranges silenced.
  </Card>

  <Card title="Emotion Detection" icon="face-smile" href="/get-started/emotion">
    A whole-file emotion label plus a per-window time series.
  </Card>

  <Card title="Accent Detection" icon="globe" href="/get-started/accent">
    A whole-file accent label plus a per-window time series.
  </Card>

  <Card title="Music & Speech Detection" icon="music" href="/get-started/music-detection">
    Frame-level music and speech probabilities.
  </Card>

  <Card title="AI Music Detection" icon="robot" href="/get-started/ai-music-detection">
    Whether a track contains AI-generated vocals or instrumentals.
  </Card>

  <Card title="Audio Event Detection" icon="waveform-lines" href="/get-started/audio-event-detection">
    A probability for each of 42 non-speech sound events, from instruments to gunshots.
  </Card>

  <Card title="Language Detection" icon="language" href="/get-started/language-detection">
    The spoken language of a clip, with a confidence score.
  </Card>
</CardGroup>

[Which API should I use?](/guides/which-api) works from the output backwards to the endpoint that produces it.

## Combine models

One endpoint answers one question. Most production decisions read two or more outputs together.

<Card title="Screen a call for voice fraud" icon="user-shield" href="/get-started/voice-fraud-screening">
  Run Deepfake Detection and Velma Triage on one recording, aggregate the frame verdicts, and combine both outputs into a single screening decision.
</Card>


# Support
Source: https://docs.modulate.ai/support

How to reach the Modulate team for technical questions, bug reports, feature requests, or limit increases.

<Warning>
  2026-09-03 16:16 EDT -- We are aware of an issue impacting our velma-2-stt-batch and velma-2-stt-streaming endpoints. We're working on resolving this as soon as possible.
</Warning>

## Contact us

<script />

<div />

Use the support form above to get in touch with Modulate Support for technical questions, feature requests, bug reports, or account requests. We respond within 2 business days.

### How to get the fastest response

Where applicable, please include:

* The endpoint you were calling (e.g. Multilingual Transcription batch, Deepfake Detection streaming)
* The HTTP status code or WebSocket close code you received
* The error message or response body
* The audio format and approximate file duration
* Any request IDs included in the response

The more detail you provide, the faster we can diagnose the issue!

### Increasing concurrency

Concurrency caps are set per model and default to 3 simultaneous requests. [Fill out the form above](#contact-us) with the model and the traffic you expect, and we will raise it. See [Limits](/guides/authentication#limits).

## Privacy and legal

Before contacting us please be sure to review our [privacy policy](https://www.modulate.ai/privacy-policy) and our [FAQ](/faq).

For data processing agreements, GDPR questions, or audio deletion requests: [legal@modulate.ai](mailto:legal@modulate.ai)

For privacy rights requests (access, correction, deletion): [privacy@modulate.ai](mailto:privacy@modulate.ai)

## Other resources

| Resource                 | Link                                                                                 |
| ------------------------ | ------------------------------------------------------------------------------------ |
| Usage dashboard          | [platform.modulate.ai/dashboard/usage](https://platform.modulate.ai/dashboard/usage) |
| Sign up / get an API key | [platform.modulate.ai/signup-request](https://platform.modulate.ai/signup-request)   |
| Try Velma in the browser | [preview.modulate.ai](https://preview.modulate.ai)                                   |
| Modulate website         | [modulate.ai](https://modulate.ai)                                                   |
| Full privacy policy      | [modulate.ai/privacy-policy](https://www.modulate.ai/privacy-policy)                 |
| Troubleshooting guide    | [Common errors and fixes](/guides/troubleshooting)                                   |
| FAQ                      | [Frequently asked questions](/faq)                                                   |


# Best practices & what to avoid
Source: https://docs.modulate.ai/velma/behaviors/best-practices

How to write behavior descriptions that produce consistent, accurate detection results — and the common patterns that cause false positives and missed detections.

The quality of your behavior detection is directly tied to the quality of your `detailed_description`. Velma reads your description as instructions — the more precisely you define the signal, the more reliably it identifies it.

## Start with the title and short description

Before writing your `detailed_description`, nail the `name` and `short_description` first. These aren't labels — they set the frame for everything that follows.

* **`name`** — keep it under five words. It should be specific enough that someone unfamiliar with your setup understands immediately what the behavior catches. "Aggressive Language" is okay. "Customer Threat Toward Agent" is better.
* **`short_description`** — one sentence, the dictionary definition of the behavior. If you can't summarize it in one sentence, the behavior is probably trying to do too much. Consider splitting it.

Once these are locked, use them as the definition to check your criteria against as you write the `detailed_description`.

## Write criteria, not examples

Your `detailed_description` should be a list of explicit conditions that must be met for the behavior to fire. This is not a list of examples.

Examples narrow down what the system looks for and produce unreliable results when the audio doesn't match them closely. Criteria define the underlying structure the examples would satisfy.

**Do this:**

```
This behavior is present if the speech meets ALL of the following criteria:
- The speech features a conditional statement where a negative action is tied to compliance with a demand.
- The demand is made in the first or second person voice.
- The speech is not a mutual negotiation where the other party is invited to propose an alternative.
```

**Not this:**

```
Examples of this behavior include: "If you don't fix this I'll leave a one-star review", 
"I'll tell all my friends to cancel", "You'll be hearing from my lawyer."
```

## Ground criteria in concrete language

Vague terms like "insults," "hateful language," or "inappropriate content" require Velma to make its own interpretive judgment — and that judgment may not match what you intended. Ground your criteria in concrete linguistic elements: sentence structure, subject-verb relationships, tense, person, and specific word types.

**Vague:**

```
The speech must contain hateful or insulting language.
```

**Concrete:**

```
The speech must contain at least one sentence where the subject is described using 
adjectives with clearly negative connotations, or where the subject is the target 
of a threatened harmful action expressed in future tense.
```

The more you can translate the behavior into something closer to a grammar or structure check rather than a social context judgment, the more consistent your results will be.

## Don't restate the title in the description

If your behavior is named "Coercion Manipulation" and your description says "identify patterns of harassment and intimidation," you haven't told Velma anything it didn't already infer from the title. The `detailed_description` must go deeper — into specific structural and linguistic markers — not sideways into synonyms.

Whenever you find yourself using words that mean the same thing as the behavior name, that is a signal the criteria need more work.

## Specify who can trigger a behavior

Many behaviors are only meaningful when a specific participant exhibits them. A question like "Can you confirm the account number?" is routine from an agent, but a potential fraud signal from a customer. Without scoping, you will generate false positives.

Use `applies_to_participant_role_uuids` to restrict detection to specific roles:

```json theme={null}
{
  "behavior_uuid": "your-uuid-here",
  "name": "Account Verification Reversal",
  "short_description": "Customer attempts to solicit account verification from the agent.",
  "detailed_description": "...",
  "applies_to_participant_role_uuids": ["22222222-2222-4222-8222-222222222017"]
}
```

If role-based scoping alone isn't sufficient, you can also encode the role context directly in the `detailed_description`:

```
The speech must feature a request for personal account information (such as account number, 
date of birth, or security question answers) made by the customer — not the agent. 
Do not flag if the request originates from an agent following a verification procedure.
```

## Include negation criteria

Once you have defined what the behavior catches, spend equal time on what it should not catch. Negation criteria prevent false positives from content that superficially resembles the target signal but isn't what you're after.

A useful technique: think of the most common thing that looks like the behavior but clearly isn't, then write a rule that excludes it.

```
This behavior should NOT be flagged if:
- The statement is in the past tense and references a third party not present in the conversation.
- The statement is a scripted disclosure read by the agent at the start of the call.
- The speaker explicitly acknowledges the refusal and disengages.
```

## Define your umbrella terms

Broad category terms like "personal information," "sensitive topics," or "inappropriate content" will be interpreted by Velma using its own judgment. That judgment may not match what you have in mind for your specific industry or context. Always follow a broad term with an explicit list of what it includes.

**Without definition:**

```
The speech must contain references to personal information.
```

**With definition:**

```
Personal information, for the purposes of this behavior, includes: full name, 
date of birth, account number, home address, and social security number.
```

## Scope to context, not universality

A behavior that works well in one context rarely translates cleanly to another. A "return fraud" behavior scoped to an online clothing retailer will have different signals than return fraud at a food delivery service. Trying to write a single behavior that handles every edge case produces a definition that is harder to tune and less precise.

Write a tighter version for the context you are actually working in. You can always create multiple variants for different conversation types using `applies_to_conversation_type_uuids`.

## Checklist before you ship

<Steps>
  <Step title="Title and short description are locked">
    Name is under five words and unambiguous. Short description summarizes the behavior in one sentence.
  </Step>

  <Step title="Criteria are explicit conditions, not examples">
    The detailed description is a list of testable rules, not illustrative cases.
  </Step>

  <Step title="Criteria are grounded in language structure">
    Terms like "subject," "adjective," "future tense," "conditional framing" rather than "insults," "hostility," "inappropriate content."
  </Step>

  <Step title="Negation criteria are present">
    At least one "do not flag if" rule covering the most common false positive case.
  </Step>

  <Step title="Role scope is set if relevant">
    Behaviors that only apply to one participant type are scoped via applies\_to\_participant\_role\_uuids or have role context in the description.
  </Step>

  <Step title="Umbrella terms are defined">
    Any broad category term is followed by a concrete list of what it means in this context.
  </Step>

  <Step title="Configuration is stored externally">
    Velma does not persist behavior definitions between sessions. Your BatchConfig — including all UUIDs and description text — is stored in your own system so you can reproduce it reliably.
  </Step>
</Steps>

## What to avoid

<Warning>
  If your `detailed_description` contains the words "any," "all," or "every" without a bounded list, that is a signal the scope is too broad. These words tend to expand coverage in ways that are hard to predict and produce inconsistent results.
</Warning>

| Pattern                            | Problem                                              | Fix                                      |
| ---------------------------------- | ---------------------------------------------------- | ---------------------------------------- |
| Description restates the title     | Adds no new information for Velma                    | Go deeper into linguistic structure      |
| Criteria are examples              | Misses cases that don't match the examples closely   | Rewrite as explicit conditions           |
| No negation criteria               | High false positive rate for similar-looking content | Add "do not flag if" rules               |
| Umbrella terms without definitions | Velma applies its own interpretation                 | List what the term means in your context |
| One behavior catches too much      | Inconsistent results, harder to tune                 | Split into multiple specific behaviors   |

## Related

* [Custom behaviors](/velma/behaviors/custom-behaviors) — how to define your own behaviors with all required fields
* [Using behaviors](/velma/behaviors/using-behaviors) — how to include behaviors in requests


# Custom behaviors
Source: https://docs.modulate.ai/velma/behaviors/custom-behaviors

Define your own behavior from scratch, or adapt a pre-built behavior's language for your specific context.

A custom behavior is any `BehaviorDef` where you supply the UUID and write the descriptions yourself. The schema is identical to a pre-built behavior — the difference is that you control everything in it.

## Required fields

Every custom behavior needs all four required fields:

| Field                  | What to provide                                                                                   |
| ---------------------- | ------------------------------------------------------------------------------------------------- |
| `behavior_uuid`        | A UUID v4 you generate. This is how Velma identifies the behavior in `behavior_detection` events. |
| `name`                 | A short label, ideally under five words.                                                          |
| `short_description`    | One sentence — the dictionary definition of what you're detecting.                                |
| `detailed_description` | The detection criteria. This is what Velma actually reads to evaluate the behavior.               |

**Generating a UUID:**

<CodeGroup>
  ```python Python theme={null}
  import uuid
  print(str(uuid.uuid4()))
  # e.g. "f47ac10b-58cc-4372-a567-0e02b2c3d479"
  ```

  ```bash shell theme={null}
  uuidgen
  # e.g. "F47AC10B-58CC-4372-A567-0E02B2C3D479"
  ```
</CodeGroup>

## A minimal custom behavior

```json theme={null}
{
  "behavior_uuid": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
  "name": "Agent Script Deviation",
  "short_description": "Agent departs from the required call opening script.",
  "detailed_description": "This behavior is present if the agent fails to state the required disclosure within the first 60 seconds of the call. The disclosure must include the company name, the agent's name, and an acknowledgment that the call may be recorded. Absence of any one of these three elements qualifies. Do not flag if the agent states all three elements, even if phrased differently from the standard script."
}
```

## Adapting a pre-built behavior

If a pre-built behavior is close to what you need but the detection language doesn't fit your context, start from its definition and modify it. The critical step is to **use a new UUID** — this tells Velma to apply your description, not the original.

<Steps>
  <Step title="Get the pre-built definition">
    Call `GET /api/velma-2-streaming/list-presets` and find the behavior you want to adapt. Copy the full object.
  </Step>

  <Step title="Generate a new UUID">
    Replace `behavior_uuid` with a UUID you generate. This decouples your custom version from the original.
  </Step>

  <Step title="Rewrite the descriptions">
    Update `name`, `short_description`, and `detailed_description` to match your context. See [Best practices](/velma/behaviors/best-practices) for guidance on writing effective criteria.
  </Step>

  <Step title="Include it in your BatchConfig">
    Add the modified object to `BatchConfig.behaviors`. If you also include the original preset slug in `presets`, your custom version takes precedence because user-supplied entries override preset entries.
  </Step>
</Steps>

<Note>
  If you include the same behavior UUID as a pre-built behavior, Velma will use the pre-built language — not your modified description. Always use a different UUID for a custom version.
</Note>

**Example — adapting "Coercion Manipulation" for a specific context:**

<Tabs>
  <Tab title="Pre-built (from list-presets)">
    ```json theme={null}
    {
      "behavior_uuid": "33333333-3333-4333-8333-033333333008",
      "name": "Coercion Manipulation",
      "short_description": "Social engineering through intimidation or threats.",
      "detailed_description": "To qualify as coercion-based manipulation, the speech must contain a threatened or implied negative consequence tied to compliance and at least one additional qualifying signal..."
    }
    ```
  </Tab>

  <Tab title="Custom version (scoped to returns context)">
    ```json theme={null}
    {
      "behavior_uuid": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
      "name": "Return Threat",
      "short_description": "Customer uses implied consequences to pressure a return approval.",
      "detailed_description": "This behavior is present if the speech meets all of the following criteria: the speaker is the customer role; the speech features a conditional statement tying a negative action (such as a negative review, escalation to a supervisor, or public complaint) to whether the return is approved; the speech does not feature a mutual negotiation or question directed at the agent. Do not flag if the speaker is an agent, or if the statement is a factual description of a past event without conditional framing."
    }
    ```
  </Tab>
</Tabs>

## Optional fields

| Field                                | Type       | What it does                                                                              |
| ------------------------------------ | ---------- | ----------------------------------------------------------------------------------------- |
| `applies_to_conversation_type_uuids` | UUID array | Restricts detection to the specified conversation types. Null or omitted means all types. |
| `applies_to_participant_role_uuids`  | UUID array | Restricts detection to speakers assigned those roles. Null or omitted means all roles.    |

Scoping behaviors to specific roles is one of the most powerful ways to reduce false positives. If a behavior only makes sense when a customer does it — not when an agent does — restrict it accordingly:

```json theme={null}
{
  "behavior_uuid": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
  "name": "Return Threat",
  "short_description": "Customer uses implied consequences to pressure a return approval.",
  "detailed_description": "...",
  "applies_to_participant_role_uuids": ["22222222-2222-4222-8222-222222222017"]
}
```

## Storing your configurations

Velma does not offer a server-side endpoint for persisting behavior definitions. Your full `BatchConfig` — including all behavior UUIDs and descriptions — must be sent in the first text frame of every new connection. Keep your configurations in your own codebase or config system so you can reproduce them reliably.

## Related

* [Using behaviors](/velma/behaviors/using-behaviors) — how to include behaviors in a request
* [Best practices](/velma/behaviors/best-practices) — how to write `detailed_description` effectively


# Using behaviors
Source: https://docs.modulate.ai/velma/behaviors/using-behaviors

How to retrieve the preset catalog, apply preset slugs, and include BehaviorDef objects in your BatchConfig.

The `behaviors` array in `BatchConfig` accepts two types of entries — preset reference strings and full `BehaviorDef` objects — and you can mix both freely.

## Listing available presets

Call `GET /api/velma-2-batch/list-presets` or `GET /api/velma-2-streaming/list-presets` to retrieve the behavior preset catalog. Both endpoints require the `X-API-Key` header and return the same response format.

<CodeGroup>
  ```bash curl theme={null}
  curl https://platform.modulate.ai/api/velma-2-batch/list-presets \
    -H "X-API-Key: $MODULATE_API_KEY"
  ```

  ```python Python theme={null}
  import os, requests

  response = requests.get(
      "https://platform.modulate.ai/api/velma-2-batch/list-presets",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
  )
  response.raise_for_status()
  for preset in response.json()["presets"]:
      print(f"preset:{preset['identifier']}  —  {preset['name']}")
      print(f"  {preset['short_description']}")
  ```
</CodeGroup>

<Accordion title="Example response (truncated)">
  ```json theme={null}
  {
    "presets": [
      {
        "identifier": "service-churn",
        "name": "Service Churn",
        "short_description": "Customer decides to cancel an ongoing service.",
        "detailed_description": "To qualify, the speech must fit the following criteria: The speech must feature assertions that the speaker would like to cancel a service or subscription. The speech must not be phrased as a threat in the first or second person voice."
      },
      {
        "identifier": "harassment",
        "name": "Harassment",
        "short_description": "Targeted, repeated hostile behavior toward a participant.",
        "detailed_description": "..."
      }
    ]
  }
  ```
</Accordion>

The `identifier` is what you use in a preset reference string. The `detailed_description` is the exact detection language Velma applies when you reference a preset — use it as a starting point when adapting a preset into a custom behavior.

## Using preset references

Include a preset by adding a `"preset:<identifier>"` string to the `behaviors` array. Velma expands it into the full behavior definition before processing.

<CodeGroup>
  ```bash websocat theme={null}
  websocat "wss://platform.modulate.ai/api/velma-2-streaming?api_key=$MODULATE_API_KEY" \
    --text - <<'EOF'
  {"behaviors":["preset:harassment","preset:service_churn","preset:account_impersonation"]}
  EOF
  ```

  ```python Python theme={null}
  import os, json, asyncio, websockets

  config = {
      "behaviors": [
          "preset:harassment",
          "preset:service_churn",
          "preset:account_impersonation",
      ]
  }

  async def stream():
      url = f"wss://platform.modulate.ai/api/velma-2-streaming?api_key={os.environ['MODULATE_API_KEY']}"
      async with websockets.connect(url) as ws:
          await ws.send(json.dumps(config))
          with open("audio.mp3", "rb") as f:
              while chunk := f.read(4096):
                  await ws.send(chunk)
          await ws.send("")
          async for message in ws:
              event = json.loads(message)
              if event["type"] == "done":
                  break

  asyncio.run(stream())
  ```
</CodeGroup>

## Including BehaviorDef objects directly

Put full `BehaviorDef` objects in the `behaviors` array for precise control. All four required fields must be present.

**To define a custom behavior:** supply all four fields with a UUID you generate. See [Custom behaviors](/velma/behaviors/custom-behaviors).

<CodeGroup>
  ```bash websocat theme={null}
  websocat "wss://platform.modulate.ai/api/velma-2-streaming?api_key=$MODULATE_API_KEY" \
    --text - <<'EOF'
  {
    "behaviors": [
      {
        "behavior_uuid": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
        "name": "Service Churn",
        "short_description": "Customer decides to cancel an ongoing service.",
        "detailed_description": "To qualify, the speech must fit the following criteria: The speech must feature assertions that the speaker would like to cancel a service or subscription. The speech must not be phrased as a threat in the first or second person voice."
      }
    ]
  }
  EOF
  ```

  ```python Python theme={null}
  import os, json, asyncio, websockets

  config = {
      "behaviors": [
          {
              "behavior_uuid": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
              "name": "Service Churn",
              "short_description": "Customer decides to cancel an ongoing service.",
              "detailed_description": (
                  "To qualify, the speech must fit the following criteria: "
                  "The speech must feature assertions that the speaker would like to cancel "
                  "a service or subscription. The speech must not be phrased as a threat "
                  "in the first or second person voice."
              ),
          }
      ]
  }

  async def stream():
      url = f"wss://platform.modulate.ai/api/velma-2-streaming?api_key={os.environ['MODULATE_API_KEY']}"
      async with websockets.connect(url) as ws:
          await ws.send(json.dumps(config))
          with open("audio.mp3", "rb") as f:
              while chunk := f.read(4096):
                  await ws.send(chunk)
          await ws.send("")
          async for message in ws:
              event = json.loads(message)
              if event["type"] == "behavior_detection":
                  d = event["detection"]
                  print(f"{d['behavior_name']}: detected={d['detected']} confidence={d['confidence']}")
              elif event["type"] == "done":
                  break

  asyncio.run(stream())
  ```
</CodeGroup>

## Mixing presets and custom behaviors

Preset references and `BehaviorDef` objects can coexist in the same `behaviors` array. If a custom `BehaviorDef` shares a UUID with a preset entry, the custom definition takes precedence.

```json theme={null}
{
  "behaviors": [
    "preset:harassment",
    {
      "behavior_uuid": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
      "name": "Service Churn",
      "short_description": "Customer decides to cancel an ongoing service.",
      "detailed_description": "To qualify, the speech must feature assertions that the speaker would like to cancel a service or subscription. The speech must not be phrased as a threat in the first or second person voice."
    }
  ]
}
```

## Behavior fields reference

| Field                                | Type                | Required |
| ------------------------------------ | ------------------- | -------- |
| `behavior_uuid`                      | UUID string         | Yes      |
| `name`                               | string (min 1 char) | Yes      |
| `short_description`                  | string              | Yes      |
| `detailed_description`               | string              | Yes      |
| `applies_to_conversation_type_uuids` | UUID array or null  | No       |
| `applies_to_participant_role_uuids`  | UUID array or null  | No       |

## Related

* [Custom behaviors](/velma/behaviors/custom-behaviors) — define your own from scratch or adapt pre-built ones
* [Best practices](/velma/behaviors/best-practices) — write descriptions that produce accurate results
* [Capabilities](/velma/capabilities) — the full BatchConfig reference


# What are behaviors?
Source: https://docs.modulate.ai/velma/behaviors/what-are-behaviors

Behaviors are the signals you tell Velma to listen for. Each one is a named, described detection target that Velma evaluates against the conversation audio.

A behavior is a detection target — a named signal you want Velma to look for in a conversation. You include behaviors in your `BatchConfig` before streaming audio, and Velma emits a `behavior_detection` event for each one with a verdict, a confidence score, and the clips that served as evidence.

Behaviors are the primary mechanism for customizing what Velma pays attention to. The same underlying audio can be evaluated against a compliance checklist, a safety policy, a sales coaching rubric, or any other set of criteria — it all depends on the behaviors you define.

## Two ways to use behaviors

**Use a pre-built behavior.** Modulate maintains a catalog of behaviors covering common use cases across customer support, sales, content moderation, media monitoring, and more. Call `GET /api/velma-2-batch/list-presets` or `GET /api/velma-2-streaming/list-presets` to retrieve the catalog. Each entry includes an `identifier` you reference with the `"preset:<identifier>"` syntax in your `behaviors` array.

**Define a custom behavior.** Supply your own `BehaviorDef` with a name and description you write. Velma evaluates whatever criteria you specify. You can start from scratch or adapt a pre-built definition as a template.

Both paths use the same `BehaviorDef` object in your `BatchConfig`. The difference is whether the UUID and descriptions come from the preset catalog or from you.

## The BehaviorDef schema

Every behavior in your `BatchConfig.behaviors` array must include all four required fields:

| Field                                | Type        | Required | Description                                                                                                                                                     |
| ------------------------------------ | ----------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `behavior_uuid`                      | UUID string | Yes      | Uniquely identifies this behavior. For pre-built behaviors, use the UUID from the list-presets response. For custom behaviors, supply a UUID v4 you generate.   |
| `name`                               | string      | Yes      | A short label. Used in `behavior_detection` event responses to identify which behavior fired.                                                                   |
| `short_description`                  | string      | Yes      | One sentence summarizing what the behavior detects.                                                                                                             |
| `detailed_description`               | string      | Yes      | The criteria Velma uses to determine whether the behavior is present. This is the most important field — see [Best practices](/velma/behaviors/best-practices). |
| `applies_to_conversation_type_uuids` | UUID array  | No       | Restrict this behavior to specific conversation types. Null or omitted means all types.                                                                         |
| `applies_to_participant_role_uuids`  | UUID array  | No       | Restrict this behavior to specific participant roles. Null or omitted means all roles.                                                                          |

## What Velma returns

Each behavior produces a `behavior_detection` event:

```json theme={null}
{
  "type": "behavior_detection",
  "detection": {
    "behavior_uuid": "33333333-3333-4333-8333-033333333006",
    "behavior_name": "Service Churn",
    "speaker_label": "Speaker_1",
    "detected": true,
    "confidence": 0.91,
    "evidence_clip_uuids": ["a1b2c3d4-...", "e5f6a7b8-..."],
    "definitive_clip_uuid": "a1b2c3d4-...",
    "reasoning": "Speaker explicitly stated they want to cancel their subscription."
  }
}
```

`detected` is the primary verdict. `evidence_clip_uuids` links the detection back to the specific clips in the session.

## A minimal example

Using a preset reference — the simplest path:

```json theme={null}
{
  "behaviors": ["preset:service_churn"]
}
```

Or with a full `BehaviorDef` for custom or adapted language:

```json theme={null}
{
  "behaviors": [
    {
      "behavior_uuid": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
      "name": "Service Churn",
      "short_description": "Customer decides to cancel an ongoing service.",
      "detailed_description": "To qualify, the speech must feature assertions that the speaker would like to cancel a service or subscription. The speech must not be phrased as a threat in the first or second person voice."
    }
  ]
}
```

## Next steps

<CardGroup>
  <Card title="Using behaviors" icon="play" href="/velma/behaviors/using-behaviors">
    How to retrieve the preset catalog and include behaviors in your requests.
  </Card>

  <Card title="Custom behaviors" icon="pen-to-square" href="/velma/behaviors/custom-behaviors">
    Define your own behaviors from scratch or adapted from pre-built ones.
  </Card>

  <Card title="Best practices" icon="circle-check" href="/velma/behaviors/best-practices">
    Write descriptions that produce consistent, accurate detections.
  </Card>
</CardGroup>


# Capabilities
Source: https://docs.modulate.ai/velma/capabilities

A complete reference for the Velma Triage batch and streaming endpoints — configuration, all outputs, and how the analysis fits together.

Velma Triage offers two endpoints with identical analysis capabilities. Both accept the same `BatchConfig` schema and return the same set of outputs — the difference is protocol and response shape.

|              | Batch                                            | Streaming                                          |
| ------------ | ------------------------------------------------ | -------------------------------------------------- |
| **Endpoint** | `POST /api/velma-2-batch`                        | `wss://platform.modulate.ai/api/velma-2-streaming` |
| **Auth**     | `X-API-Key` header                               | `api_key` query parameter                          |
| **Config**   | `config` form field (JSON string or `"default"`) | First text frame (JSON or `"default"`)             |
| **Response** | Single JSON object                               | Stream of typed events                             |

## Configuration

Both endpoints use the same `BatchConfig` schema. You can send the literal string `"default"` instead of a full config to use Velma's built-in default behavior set without specifying anything.

### Conversation types

A conversation type tells Velma what kind of interaction it is analyzing. Velma uses this to contextualize behavior detection and role assignment. You can define multiple types — Velma will infer which one best matches, or use `default_conversation_type` as the fallback.

```json theme={null}
{
  "conversation_types": [
    {
      "conversation_type_uuid": "11111111-1111-4111-8111-111111111008",
      "name": "Customer Support Call",
      "short_description": "A customer contacting a business for help with a product or service.",
      "detailed_description": "The conversation involves a customer with a service issue and a representative resolving it. Tone is professional. The representative is expected to follow company procedure."
    }
  ]
}
```

### Participant roles

Roles describe the speakers Velma expects. Scope roles to specific conversation types via `applies_to_conversation_type_uuids`. If omitted, the role applies to all types.

```json theme={null}
{
  "participant_roles": [
    {
      "participant_role_uuid": "22222222-2222-4222-8222-222222222017",
      "name": "Customer",
      "short_description": "The recipient of a service or good.",
      "detailed_description": "A customer calling in with a service issue.",
      "applies_to_conversation_type_uuids": ["11111111-1111-4111-8111-111111111008"]
    }
  ]
}
```

### Behaviors

The `behaviors` array accepts two types of entries — full `BehaviorDef` objects and preset reference strings — and you can mix both in the same array.

**Preset reference:** a string in the form `"preset:<identifier>"`. Velma expands it into the full behavior definition before processing. Use `GET /api/velma-2-batch/list-presets` or `GET /api/velma-2-streaming/list-presets` to discover available identifiers.

**Full `BehaviorDef`:** supply all four required fields yourself. Takes precedence over any preset entry with the same UUID.

```json theme={null}
{
  "behaviors": [
    "preset:harassment",
    "preset:service_churn",
    {
      "behavior_uuid": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
      "name": "Agent Script Deviation",
      "short_description": "Agent departs from the required call opening script.",
      "detailed_description": "..."
    }
  ]
}
```

See [Behaviors](/velma/behaviors/what-are-behaviors) for the full guide.

### Transcription options

Control what transcription data appears in clip outputs:

| Option                | Type    | Default | What it adds                                                                                                                           |
| --------------------- | ------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `speaker_diarization` | boolean | `true`  | Per-speaker clip attribution                                                                                                           |
| `emotion_signal`      | boolean | `false` | Per-clip `emotion` label                                                                                                               |
| `accent_signal`       | boolean | `false` | Per-clip `accent` label                                                                                                                |
| `deepfake_signal`     | boolean | `false` | Per-clip `deepfake_score` (0–1)                                                                                                        |
| `pii_phi_tagging`     | boolean | `false` | Sensitive spans wrapped in entity tags                                                                                                 |
| `language`            | string  | auto    | Optional language hint (case-insensitive ISO 639-1 code, e.g. `en`); the language is detected automatically for each clip when omitted |
| `custom_terms`        | array   | none    | Custom vocabulary to bias transcription toward domain terms and names — see [Custom vocabulary](/get-started/stt#custom-vocabulary)    |

### Aggregate outputs

| Option                     | Type    | Default |
| -------------------------- | ------- | ------- |
| `produce_topics`           | boolean | `true`  |
| `produce_topic_sentiments` | boolean | `true`  |
| `produce_summary`          | boolean | `true`  |

Set any of these to `false` to suppress the corresponding output.

***

## Batch endpoint

`POST /api/velma-2-batch` — submit a complete audio file, receive a single JSON response.

**Request** — `multipart/form-data`:

| Field         | Type   | Required | Description                                                                                        |
| ------------- | ------ | -------- | -------------------------------------------------------------------------------------------------- |
| `upload_file` | binary | Yes      | Audio file. Max 100 MB. Supported formats: AAC, AIFF, FLAC, MP3, MP4, MOV, OGG, Opus, WAV, WebM.   |
| `config`      | string | No       | JSON-encoded `BatchConfig`, or the literal string `"default"`. Defaults to `"default"` if omitted. |

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://platform.modulate.ai/api/velma-2-batch \
    -H "X-API-Key: $MODULATE_API_KEY" \
    -F "upload_file=@recording.mp3" \
    -F 'config={"behaviors":["preset:harassment","preset:service_churn"]}'
  ```

  ```python Python theme={null}
  import os, json, requests

  config = {
      "behaviors": ["preset:harassment", "preset:service_churn"],
      "stt": {"speaker_diarization": True},
      "produce_summary": True,
  }

  response = requests.post(
      "https://platform.modulate.ai/api/velma-2-batch",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
      data={"config": json.dumps(config)},
      files={"upload_file": open("recording.mp3", "rb")},
  )
  response.raise_for_status()
  result = response.json()
  ```
</CodeGroup>

**Response** — `BatchResponse`:

| Field                    | Type           | Description                                                                   |
| ------------------------ | -------------- | ----------------------------------------------------------------------------- |
| `duration_ms`            | integer        | Total audio duration                                                          |
| `clips`                  | array          | Transcribed segments — see [Clip](#clip)                                      |
| `conversation_type_pick` | object or null | Inferred conversation type                                                    |
| `participant_role_picks` | array          | Per-speaker role assignments                                                  |
| `behaviors`              | array          | Per-behavior detection results — see [BehaviorDetection](#behavior_detection) |
| `topics`                 | array          | Extracted topic strings                                                       |
| `topic_sentiments`       | array          | Per-speaker sentiment per topic                                               |
| `summary`                | string or null | Narrative summary                                                             |

**Error responses:**

| Status | Meaning                                                                                                                                                                                                                 |
| ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `400`  | Unsupported file format, empty file, or malformed request                                                                                                                                                               |
| `403`  | Request not permitted                                                                                                                                                                                                   |
| `422`  | Invalid `config` value (malformed JSON, unknown preset identifier, or a definition missing a required field), or a required request field missing or the wrong type (e.g. the `X-API-Key` header or `upload_file` part) |
| `429`  | Insufficient credits                                                                                                                                                                                                    |
| `500`  | Internal server error                                                                                                                                                                                                   |
| `502`  | Request could not be validated or completed — retry                                                                                                                                                                     |

***

## Streaming events

Velma emits JSON events throughout a streaming session. Every event has a `type` field.

### `clip`

A transcribed segment of speech. Emitted in near real time.

```json theme={null}
{
  "type": "clip",
  "clip": {
    "clip_uuid": "a1b2c3d4-...",
    "text": "I'd like to cancel my subscription.",
    "start_ms": 4800,
    "duration_ms": 2100,
    "speaker_label": "Speaker_1",
    "language": "en",
    "emotion": null,
    "accent": null,
    "deepfake_score": null
  }
}
```

`emotion`, `accent`, and `deepfake_score` are non-null only when their corresponding Transcription options are enabled.

### `partial_clip`

An in-progress clip streamed while an utterance is still being spoken. Multiple partials may arrive for the same `clip_uuid` as the utterance grows; the eventual `clip` event reuses that `clip_uuid` and supersedes all of its partials.

```json theme={null}
{
  "type": "partial_clip",
  "partial_clip": {
    "clip_uuid": "a1b2c3d4-...",
    "text": "I'd like to cancel",
    "start_ms": 4800,
    "end_ms": 5900,
    "speaker_label": "Speaker_1",
    "emotion": null,
    "accent": null,
    "deepfake_score": null
  }
}
```

### `clip_update`

Refined `emotion` / `accent` values for a previously finalized clip; `clip_uuid` matches an earlier `clip` event. A clip may receive any number of updates (including none), always before the `done` event — for each field present, the latest received value wins.

```json theme={null}
{
  "type": "clip_update",
  "clip_update": {
    "clip_uuid": "a1b2c3d4-...",
    "emotion": "Frustrated",
    "accent": "American"
  }
}
```

### `conversation_type`

Velma's pick for the conversation type, emitted once enough context is available.

```json theme={null}
{
  "type": "conversation_type",
  "pick": {
    "conversation_type_uuid": "11111111-1111-4111-8111-111111111008",
    "name": "Customer Support Call",
    "confidence": 0.94,
    "selection_source": "inferred",
    "detail": "...",
    "reasoning": "..."
  }
}
```

`selection_source` is one of `inferred`, `auto_selected_single_option`, or `default`.

### `participant_role`

A per-speaker role assignment. One event per speaker label.

```json theme={null}
{
  "type": "participant_role",
  "pick": {
    "speaker_label": "Speaker_1",
    "participant_role_uuid": "22222222-2222-4222-8222-222222222017",
    "name": "Customer",
    "confidence": 0.88,
    "selection_source": "inferred",
    "detail": "...",
    "reasoning": "..."
  }
}
```

### `behavior_detection`

A per-behavior verdict. Emitted for each behavior once Velma has enough audio to decide.

```json theme={null}
{
  "type": "behavior_detection",
  "detection": {
    "behavior_uuid": "33333333-3333-4333-8333-033333333006",
    "behavior_name": "Service Churn",
    "speaker_label": "Speaker_1",
    "detected": true,
    "confidence": 0.91,
    "evidence_clip_uuids": ["a1b2c3d4-...", "e5f6a7b8-..."],
    "definitive_clip_uuid": "a1b2c3d4-...",
    "reasoning": "Speaker explicitly stated they want to cancel their subscription."
  }
}
```

### `topics`

Aggregated list of subjects discussed. May be emitted more than once as the conversation progresses; each event fully replaces the previous list, so always treat the latest as authoritative.

```json theme={null}
{
  "type": "topics",
  "topics": ["subscription cancellation", "billing dispute", "refund policy"]
}
```

### `topic_sentiment`

Per-speaker sentiment for each topic. May be emitted more than once as the conversation progresses; a later event supersedes an earlier one for the same topic and speaker.

```json theme={null}
{
  "type": "topic_sentiment",
  "topic_sentiment": {
    "topic": "subscription cancellation",
    "speaker_label": "Speaker_1",
    "sentiment_score": -0.72,
    "sentiment_label": "negative"
  }
}
```

`sentiment_score` ranges from −1 (strongly negative) to +1 (strongly positive).

### `summary`

A free-form narrative summary. May be emitted more than once as the conversation progresses; each event fully replaces the previous summary.

```json theme={null}
{
  "type": "summary",
  "text": "The customer called to cancel their subscription due to billing concerns. The representative offered a partial credit, which the customer declined. The call ended without resolution."
}
```

### `done`

Signals streaming is complete. Always the final event.

```json theme={null}
{
  "type": "done",
  "duration_ms": 183400
}
```

### `error`

Emitted if a processing error occurs. The connection closes after this event.

```json theme={null}
{
  "type": "error",
  "error": "Invalid input audio"
}
```

***

## WebSocket close codes

| Code   | Meaning                                                                                                          |
| ------ | ---------------------------------------------------------------------------------------------------------------- |
| `1000` | Normal closure after the `done` event                                                                            |
| `1003` | Protocol error — invalid or incomplete config, audio sent before config, unsupported audio format or sample rate |
| `4003` | Request could not be validated, or not permitted                                                                 |
| `4029` | Insufficient credits                                                                                             |

***

## Related

* [Behaviors](/velma/behaviors/what-are-behaviors) — define what Velma listens for
* [Velma Triage](/get-started/velma) — supported formats and raw PCM parameters
* [Authentication](/guides/authentication) — API key setup


# Detection packages
Source: https://docs.modulate.ai/velma/detection-packages

Ready-made Velma Triage configurations that group behaviors into clear, use-case buckets you can explore and download.

Detection packages are ready-made Velma Triage configurations that group related behaviors into clear, use-case buckets. Each package bundles the conversation types, participant roles, and behaviors for a common goal — so you can point at the bucket that matches your use case, see exactly what it detects, and drop the config straight into your own scripts.

Every package is a complete `BatchConfig`: explore it below, then download or copy the JSON to use it as the `config` payload in a Velma Triage request.

<CardGroup>
  <Card title="Fraud Detection and Prevention" icon="shield-halved" href="/velma/detection-packages/fraud-detection-and-prevention">
    Catch social engineering, account-takeover attempts, and impersonation in customer conversations.
  </Card>

  <Card title="Agentic AI Guardrails" icon="robot" href="/velma/detection-packages/agentic-ai-guardrails">
    Keep AI voice agents on-policy: catch off-script, unsafe, or out-of-scope behavior.
  </Card>

  <Card title="Trust and Safety" icon="shield-check" href="/velma/detection-packages/trust-and-safety">
    Flag harassment, hate, and safety violations in live social and voice-chat audio.
  </Card>

  <Card title="Customer Retention" icon="heart" href="/velma/detection-packages/customer-retention">
    Surface churn signals, dissatisfaction, and save opportunities on every call.
  </Card>

  <Card title="Human Agent Welfare" icon="hand-holding-heart" href="/velma/detection-packages/human-agent-welfare">
    Protect agents by detecting abuse, distress, and burnout indicators.
  </Card>

  <Card title="Compliance and Risk Monitoring" icon="scale-balanced" href="/velma/detection-packages/compliance-and-risk-monitoring">
    Catch disclosure gaps, compliance breaches, and risk events across regulated calls.
  </Card>
</CardGroup>


# Agentic AI Guardrails
Source: https://docs.modulate.ai/velma/detection-packages/agentic-ai-guardrails

Keep AI voice agents on-policy: catch off-script, unsafe, or out-of-scope behavior.

Monitor AI voice agents for off-policy, unsafe, or out-of-scope behavior so you can keep automated conversations inside the guardrails you define.

Voice agents are just one application. The same signals apply wherever automated or scripted conversations need to stay within defined bounds — IVR flows, AI-assisted human agents, and beyond. Use this package as a starting point — add or swap in any [preset from the catalog](/velma/behaviors/using-behaviors#listing-available-presets), or [define your own behaviors](/velma/behaviors/custom-behaviors) to fit your use case.

**This package includes:** 2 conversation types · 6 participant roles · 8 behaviors.

## Use this package

The config below is ready to use as-is — download or copy it and pass it as the `config` payload in a Velma Triage request. Its `behaviors` are [preset references](/velma/behaviors/using-behaviors): the API expands each `preset:<identifier>` into its full definition at request time, so you don't need the criteria inline to run the package.

<a href="#">Download agentic-ai-guardrails.json</a>

<Accordion title="View / copy full config">
  <div>
    ```json theme={null}
    {
      "conversation_types": [
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111017",
          "name": "AI Agent Customer Support Call",
          "short_description": "Any Customer Support Call with an AI agent",
          "detailed_description": "customer-supportprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111018",
          "name": "AI Agent Interview Monitoring",
          "short_description": "Any interview call with an AI interviewer",
          "detailed_description": "business-operationsprofessional"
        }
      ],
      "participant_roles": [
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222004",
          "name": "Support Specialist",
          "short_description": "Someone who works with employees or contractors to solve problems with things like IT, logistics, or communication",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222012",
          "name": "Customer Service Representative",
          "short_description": "A representative from a business who is assisting a customer with an issue they are having",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222014",
          "name": "Insurance Agent",
          "short_description": "A customer support representative from an insurance company",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222015",
          "name": "Interview Candidate",
          "short_description": "Someone who is being interview for a job",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222016",
          "name": "Interviewer",
          "short_description": "A person giving an interview",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222017",
          "name": "Customer",
          "short_description": "The recipient of a service or good",
          "detailed_description": ""
        }
      ],
      "behaviors": [
        "preset:vishing",
        "preset:service_churn",
        "preset:off_topic_discussion",
        "preset:inapropriate_ai_agent_content",
        "preset:ai_agent_manipulation",
        "preset:inappropriate_speech",
        "preset:issue_resolved",
        "preset:sexual_harassment"
      ]
    }
    ```
  </div>
</Accordion>

### Expand the full criteria

To produce a self-contained config with every behavior's full criteria inlined — for review, customization, or pinning a snapshot — fetch the live [preset catalog](/velma/behaviors/using-behaviors#listing-available-presets) and merge it into the downloaded config. The catalog is the source of truth for detection criteria.

<CodeGroup>
  ```bash curl + jq theme={null}
  curl -s https://platform.modulate.ai/api/velma-2-batch/list-presets \
    -H "X-API-Key: $MODULATE_API_KEY" \
  | jq --slurpfile cfg agentic-ai-guardrails.json '
      [ $cfg[0].behaviors[] | ltrimstr("preset:") ] as $ids
      | { conversation_types: $cfg[0].conversation_types,
          participant_roles:  $cfg[0].participant_roles,
          behaviors: [ .presets[] | select(.identifier as $i | $ids | index($i)) ] }
    ' > agentic-ai-guardrails.full.json
  ```

  ```python Python theme={null}
  import os, json, requests

  cfg = json.load(open("agentic-ai-guardrails.json"))
  ids = {b.removeprefix("preset:") for b in cfg["behaviors"]}

  catalog = requests.get(
      "https://platform.modulate.ai/api/velma-2-batch/list-presets",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
  ).json()["presets"]

  cfg["behaviors"] = [p for p in catalog if p["identifier"] in ids]
  json.dump(cfg, open("agentic-ai-guardrails.full.json", "w"), indent=2)
  ```
</CodeGroup>

`agentic-ai-guardrails.full.json` keeps the same `conversation_types` and `participant_roles` and replaces each preset reference with its full behavior definition — drop it into the `config` payload exactly like the preset version.

## Conversation types

The interaction contexts this package expects to see.

| Name                           | What it is                                 |
| ------------------------------ | ------------------------------------------ |
| AI Agent Customer Support Call | Any Customer Support Call with an AI agent |
| AI Agent Interview Monitoring  | Any interview call with an AI interviewer  |

## Participant roles

The speaker roles the package distinguishes.

| Name                            | What it is                                                                                                         |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| Support Specialist              | Someone who works with employees or contractors to solve problems with things like IT, logistics, or communication |
| Customer Service Representative | A representative from a business who is assisting a customer with an issue they are having                         |
| Insurance Agent                 | A customer support representative from an insurance company                                                        |
| Interview Candidate             | Someone who is being interview for a job                                                                           |
| Interviewer                     | A person giving an interview                                                                                       |
| Customer                        | The recipient of a service or good                                                                                 |

## Behaviors

The 8 signals this package detects. Each maps to a reusable `preset:<identifier>` you can drop into the `behaviors` array of any [BatchConfig](/velma/behaviors/using-behaviors) — the config above already references them.

<Note>
  Full detection criteria are not duplicated here. The live preset catalog is the source of truth — retrieve the exact criteria for any behavior by name from the [`list-presets` endpoint](/velma/behaviors/using-behaviors#listing-available-presets).
</Note>

| Behavior                      | What it detects                                                                                                                                                                                                                                               | Preset                                 |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- |
| Vishing                       | Attempts to elicit sensitive information through deceptive voice interactions. We detect vishing based on abnormal call pacing, probing question patterns, stress-induced pitch shifts, and background noise suggesting call centers or spoofed environments. | `preset:vishing`                       |
| Service Churn                 | Customer decides to cancel an ongoing service. We detect this through resignation tone, conclusive phrasing, disengaging cadence, and emotional withdrawal.                                                                                                   | `preset:service_churn`                 |
| Off-topic Discussion          | Conversation largely unrelated to call purpose. We detect this using semantic drift paired with relaxed pacing, reduced task-oriented urgency, and tonal divergence from initial intent.                                                                      | `preset:off_topic_discussion`          |
| Inapropriate AI Agent Content | AI-generated speech unsuitable for professional contexts. We detect this by tonal incongruence with task intent, inappropriate emotional expression, unnatural phrasing patterns, and drift into unrelated or personal topics.                                | `preset:inapropriate_ai_agent_content` |
| AI Agent Manipulation         | Attempts to coerce an AI into unintended behavior. We detect this using repetitive prompt pressure, altered speaking cadence, strategic pauses, adversarial tone, and escalating emotional manipulation signals across the interaction.                       | `preset:ai_agent_manipulation`         |
| Inappropriate Speech          | Unprofessional or unsuitable spoken content. We detect this using aggressive tone, boundary-crossing language, emotional volatility, and contextual mismatch with professional norms.                                                                         | `preset:inappropriate_speech`          |
| Issue Resolved                | Customer's problem successfully addressed. We detect this using relief markers, positive tonal shift, relaxed pacing, confirmation language, and conversational closure cues.                                                                                 | `preset:issue_resolved`                |
| Sexual Harassment             | Unwanted sexualized speech or advances. We detect this through suggestive intonation, boundary-testing pauses, inappropriate familiarity, and discomfort responses from others.                                                                               | `preset:sexual_harassment`             |


# Compliance and Risk Monitoring
Source: https://docs.modulate.ai/velma/detection-packages/compliance-and-risk-monitoring

Catch disclosure gaps, compliance breaches, and risk events across regulated calls.

Monitor regulated conversations for disclosure gaps, compliance breaches, and risk events — from required scripts and consent to prohibited statements.

Regulated calls are one application among many. Wherever a conversation carries policy, disclosure, or documentation obligations, these behaviors can flag what matters. Use this package as a starting point — add or swap in any [preset from the catalog](/velma/behaviors/using-behaviors#listing-available-presets), or [define your own behaviors](/velma/behaviors/custom-behaviors) to fit your use case.

**This package includes:** 25 conversation types · 14 participant roles · 25 behaviors.

## Use this package

The config below is ready to use as-is — download or copy it and pass it as the `config` payload in a Velma Triage request. Its `behaviors` are [preset references](/velma/behaviors/using-behaviors): the API expands each `preset:<identifier>` into its full definition at request time, so you don't need the criteria inline to run the package.

<a href="#">Download compliance-and-risk-monitoring.json</a>

<Accordion title="View / copy full config">
  <div>
    ```json theme={null}
    {
      "conversation_types": [
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111001",
          "name": "Customer Service Call",
          "short_description": "Phone or video calls between customers and support agents",
          "detailed_description": "institutional-fraud"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111002",
          "name": "Social Conversation",
          "short_description": "Public or group conversations in social contexts",
          "detailed_description": "moderation"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111003",
          "name": "Enterprise IT support",
          "short_description": "Any call to assist an employee in accessing or managing internal IT resources",
          "detailed_description": "business-operationsitprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111007",
          "name": "Academic or Professional Presentation",
          "short_description": "A talk, webinar, workshop, or any other presentation style session",
          "detailed_description": "mediaprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111008",
          "name": "General Customer Support Call",
          "short_description": "Any call where a customer is contacting a business to assist them with the use of a service or goods provided by the business",
          "detailed_description": "customer-supportprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111017",
          "name": "AI Agent Customer Support Call",
          "short_description": "Any Customer Support Call with an AI agent",
          "detailed_description": "customer-supportprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111018",
          "name": "AI Agent Interview Monitoring",
          "short_description": "Any interview call with an AI interviewer",
          "detailed_description": "business-operationsprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111019",
          "name": "Shipping Package Delivery Logistics",
          "short_description": "Any call to coordinate the delivery of a package with the delivery driver",
          "detailed_description": "consumer-service-meetingprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111020",
          "name": "Taxi Service Logistics",
          "short_description": "A call where a driver is coordinating with a customer to pick them up",
          "detailed_description": "consumer-service-meetingprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111021",
          "name": "General Contractor Visitation Coordination",
          "short_description": "Any call where there is a contracted party coordinating with a client for the means of delivering a service",
          "detailed_description": "consumer-service-meetingprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111022",
          "name": "Food Delivery Logistics",
          "short_description": "Coordinating the delivery of food to a location",
          "detailed_description": "consumer-service-meetingprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111023",
          "name": "Management and Performance Reviews",
          "short_description": "A conversation where an employee is being reviewed by a manager or review board",
          "detailed_description": "business-operationsprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111024",
          "name": "Internal Business Conversations",
          "short_description": "Any professional conversation happening inside a business",
          "detailed_description": "business-operationsprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111025",
          "name": "Business Contractor and Vendor Logisitics",
          "short_description": "Any call where a business coordinates with a contracted third party",
          "detailed_description": "business-operationsprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111026",
          "name": "Interview Monitoring",
          "short_description": "A call to conduct a professional interview for a job or role at a business",
          "detailed_description": "business-operationsprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111027",
          "name": "Finance Account Support",
          "short_description": "Any call to support the configuration, management, or cancellation of financial accounts",
          "detailed_description": "customer-supportprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111028",
          "name": "Consumer Technical Support",
          "short_description": "Any call to assist a customer in using a business's software or website",
          "detailed_description": "customer-supportitprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111029",
          "name": "Insurance Policy or Claims Management",
          "short_description": "Any call to check on the status of, change, cancel, or check on the status of a claim for an insurance policy.",
          "detailed_description": "customer-supportinsuranceprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111030",
          "name": "Insurance Claims Filing",
          "short_description": "A call to file a claim with an insurance agent",
          "detailed_description": "customer-supportinsuranceprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111031",
          "name": "Retail Customer Support Call",
          "short_description": "A call to coordinate the delivery or transfer of goods to a consumer",
          "detailed_description": "customer-supportprofessionalretail"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111032",
          "name": "Sales Contract/Deal Negotiation Call",
          "short_description": "A call to negotiate the terms of a customer agreement",
          "detailed_description": "professionalsales"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111033",
          "name": "Outbound Sales Call",
          "short_description": "Any call from a business to their clients or consumer for the purpose of selling them goods or services",
          "detailed_description": "professionalsales"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111034",
          "name": "Sales Pipeline Logistics Call",
          "short_description": "A call with the purpose of moving a prospect to the next part of the sales pipeline",
          "detailed_description": "professionalsales"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111035",
          "name": "Demo Sales Call",
          "short_description": "Product demo call",
          "detailed_description": "professionalsales"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111036",
          "name": "Inbound Sales Call",
          "short_description": "A call coming from the customer to a business to purchase goods or services",
          "detailed_description": ""
        }
      ],
      "participant_roles": [
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222002",
          "name": "Agent",
          "short_description": "Customer support representative",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222003",
          "name": "Social Participant",
          "short_description": "Participant in social conversation",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222004",
          "name": "Support Specialist",
          "short_description": "Someone who works with employees or contractors to solve problems with things like IT, logistics, or communication",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222005",
          "name": "Employee",
          "short_description": "someone who is employed by company and exists in professional settings",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222006",
          "name": "Presenter",
          "short_description": "the main speaker or speakers in a presentation",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222009",
          "name": "Social Participant",
          "short_description": "A partipant in a social conversation",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222010",
          "name": "Manager/Reviewer",
          "short_description": "A person who either has a manager title or is conducting a performance review of another person",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222011",
          "name": "Gig worker",
          "short_description": "A contractor delivering a service for someone",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222012",
          "name": "Customer Service Representative",
          "short_description": "A representative from a business who is assisting a customer with an issue they are having",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222013",
          "name": "Salesperson",
          "short_description": "A person who is attempting to sell a good or service to another person",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222014",
          "name": "Insurance Agent",
          "short_description": "A customer support representative from an insurance company",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222015",
          "name": "Interview Candidate",
          "short_description": "Someone who is being interview for a job",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222016",
          "name": "Interviewer",
          "short_description": "A person giving an interview",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222017",
          "name": "Customer",
          "short_description": "The recipient of a service or good",
          "detailed_description": ""
        }
      ],
      "behaviors": [
        "preset:complaints",
        "preset:vishing",
        "preset:account_impersonation",
        "preset:action_plan_created",
        "preset:cancelled_order",
        "preset:service_churn",
        "preset:off_topic_discussion",
        "preset:coercion_manipulation",
        "preset:return_fraud_attempt",
        "preset:feigned_ignorance",
        "preset:storytelling",
        "preset:bargaining_manipulation",
        "preset:personal_vulnerability",
        "preset:pre_established_professional_relationship",
        "preset:rapport_building",
        "preset:customer_gratitude",
        "preset:inclusive_practices",
        "preset:unclear_speech",
        "preset:unaddressed_question",
        "preset:refund_or_credit_issued",
        "preset:issue_not_resolved",
        "preset:inappropriate_speech",
        "preset:issue_resolved",
        "preset:threat_based_harassment",
        "preset:sexual_harassment"
      ]
    }
    ```
  </div>
</Accordion>

### Expand the full criteria

To produce a self-contained config with every behavior's full criteria inlined — for review, customization, or pinning a snapshot — fetch the live [preset catalog](/velma/behaviors/using-behaviors#listing-available-presets) and merge it into the downloaded config. The catalog is the source of truth for detection criteria.

<CodeGroup>
  ```bash curl + jq theme={null}
  curl -s https://platform.modulate.ai/api/velma-2-batch/list-presets \
    -H "X-API-Key: $MODULATE_API_KEY" \
  | jq --slurpfile cfg compliance-and-risk-monitoring.json '
      [ $cfg[0].behaviors[] | ltrimstr("preset:") ] as $ids
      | { conversation_types: $cfg[0].conversation_types,
          participant_roles:  $cfg[0].participant_roles,
          behaviors: [ .presets[] | select(.identifier as $i | $ids | index($i)) ] }
    ' > compliance-and-risk-monitoring.full.json
  ```

  ```python Python theme={null}
  import os, json, requests

  cfg = json.load(open("compliance-and-risk-monitoring.json"))
  ids = {b.removeprefix("preset:") for b in cfg["behaviors"]}

  catalog = requests.get(
      "https://platform.modulate.ai/api/velma-2-batch/list-presets",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
  ).json()["presets"]

  cfg["behaviors"] = [p for p in catalog if p["identifier"] in ids]
  json.dump(cfg, open("compliance-and-risk-monitoring.full.json", "w"), indent=2)
  ```
</CodeGroup>

`compliance-and-risk-monitoring.full.json` keeps the same `conversation_types` and `participant_roles` and replaces each preset reference with its full behavior definition — drop it into the `config` payload exactly like the preset version.

## Conversation types

The interaction contexts this package expects to see.

| Name                                       | What it is                                                                                                                    |
| ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- |
| Customer Service Call                      | Phone or video calls between customers and support agents                                                                     |
| Social Conversation                        | Public or group conversations in social contexts                                                                              |
| Enterprise IT support                      | Any call to assist an employee in accessing or managing internal IT resources                                                 |
| Academic or Professional Presentation      | A talk, webinar, workshop, or any other presentation style session                                                            |
| General Customer Support Call              | Any call where a customer is contacting a business to assist them with the use of a service or goods provided by the business |
| AI Agent Customer Support Call             | Any Customer Support Call with an AI agent                                                                                    |
| AI Agent Interview Monitoring              | Any interview call with an AI interviewer                                                                                     |
| Shipping Package Delivery Logistics        | Any call to coordinate the delivery of a package with the delivery driver                                                     |
| Taxi Service Logistics                     | A call where a driver is coordinating with a customer to pick them up                                                         |
| General Contractor Visitation Coordination | Any call where there is a contracted party coordinating with a client for the means of delivering a service                   |
| Food Delivery Logistics                    | Coordinating the delivery of food to a location                                                                               |
| Management and Performance Reviews         | A conversation where an employee is being reviewed by a manager or review board                                               |
| Internal Business Conversations            | Any professional conversation happening inside a business                                                                     |
| Business Contractor and Vendor Logisitics  | Any call where a business coordinates with a contracted third party                                                           |
| Interview Monitoring                       | A call to conduct a professional interview for a job or role at a business                                                    |
| Finance Account Support                    | Any call to support the configuration, management, or cancellation of financial accounts                                      |
| Consumer Technical Support                 | Any call to assist a customer in using a business's software or website                                                       |
| Insurance Policy or Claims Management      | Any call to check on the status of, change, cancel, or check on the status of a claim for an insurance policy.                |
| Insurance Claims Filing                    | A call to file a claim with an insurance agent                                                                                |
| Retail Customer Support Call               | A call to coordinate the delivery or transfer of goods to a consumer                                                          |
| Sales Contract/Deal Negotiation Call       | A call to negotiate the terms of a customer agreement                                                                         |
| Outbound Sales Call                        | Any call from a business to their clients or consumer for the purpose of selling them goods or services                       |
| Sales Pipeline Logistics Call              | A call with the purpose of moving a prospect to the next part of the sales pipeline                                           |
| Demo Sales Call                            | Product demo call                                                                                                             |
| Inbound Sales Call                         | A call coming from the customer to a business to purchase goods or services                                                   |

## Participant roles

The speaker roles the package distinguishes.

| Name                            | What it is                                                                                                         |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| Agent                           | Customer support representative                                                                                    |
| Social Participant              | Participant in social conversation                                                                                 |
| Support Specialist              | Someone who works with employees or contractors to solve problems with things like IT, logistics, or communication |
| Employee                        | someone who is employed by company and exists in professional settings                                             |
| Presenter                       | the main speaker or speakers in a presentation                                                                     |
| Social Participant              | A partipant in a social conversation                                                                               |
| Manager/Reviewer                | A person who either has a manager title or is conducting a performance review of another person                    |
| Gig worker                      | A contractor delivering a service for someone                                                                      |
| Customer Service Representative | A representative from a business who is assisting a customer with an issue they are having                         |
| Salesperson                     | A person who is attempting to sell a good or service to another person                                             |
| Insurance Agent                 | A customer support representative from an insurance company                                                        |
| Interview Candidate             | Someone who is being interview for a job                                                                           |
| Interviewer                     | A person giving an interview                                                                                       |
| Customer                        | The recipient of a service or good                                                                                 |

## Behaviors

The 25 signals this package detects. Each maps to a reusable `preset:<identifier>` you can drop into the `behaviors` array of any [BatchConfig](/velma/behaviors/using-behaviors) — the config above already references them.

<Note>
  Full detection criteria are not duplicated here. The live preset catalog is the source of truth — retrieve the exact criteria for any behavior by name from the [`list-presets` endpoint](/velma/behaviors/using-behaviors#listing-available-presets).
</Note>

| Behavior                                  | What it detects                                                                                                                                                                                                                                               | Preset                                             |
| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- |
| Complaints                                | Customer expresses dissatisfaction or grievance. We detect this through elevated volume, sharp intonation, frustration markers, accelerated pacing, and emotional intensity.                                                                                  | `preset:complaints`                                |
| Vishing                                   | Attempts to elicit sensitive information through deceptive voice interactions. We detect vishing based on abnormal call pacing, probing question patterns, stress-induced pitch shifts, and background noise suggesting call centers or spoofed environments. | `preset:vishing`                                   |
| Account Impersonation                     | Fraudulent attempt to access another's account. We detect this through identity inconsistencies, rehearsed responses, stress-induced vocal shifts, and abnormal verification behavior.                                                                        | `preset:account_impersonation`                     |
| Action Plan Created                       | Explicit agreement on next steps or follow-up. We detect this through structured enumeration, decisive tone, slowed pacing, confirmation cues, and reduced ambiguity in delivery.                                                                             | `preset:action_plan_created`                       |
| Cancelled Order                           | Customer cancels a previously placed order. We detect this using finality in tone, decisive pacing, administrative phrasing, and reduced emotional engagement after confirmation.                                                                             | `preset:cancelled_order`                           |
| Service Churn                             | Customer decides to cancel an ongoing service. We detect this through resignation tone, conclusive phrasing, disengaging cadence, and emotional withdrawal.                                                                                                   | `preset:service_churn`                             |
| Off-topic Discussion                      | Conversation largely unrelated to call purpose. We detect this using semantic drift paired with relaxed pacing, reduced task-oriented urgency, and tonal divergence from initial intent.                                                                      | `preset:off_topic_discussion`                      |
| Coercion Manipulation                     | Social engineering through intimidation or threats. We detect this using dominance-oriented tone, reduced empathy markers, pressure timing, and aggressive pacing.                                                                                            | `preset:coercion_manipulation`                     |
| Return Fraud Attempt                      | Fraudulent product return behavior. We detect this using scripted explanations, defensive tone, timing irregularities, and emotional mismatch with stated circumstances.                                                                                      | `preset:return_fraud_attempt`                      |
| Feigned Ignorance                         | Pretending towards ignorance to garner fraudulent sympathy. We detect this using inconsistent knowledge signals, exaggerated confusion tone, strategic pauses, and implausible vocal uncertainty.                                                             | `preset:feigned_ignorance`                         |
| Storytelling                              | Third-person recounting of events in narrative form. We detect this using narrative arc pacing, character-based vocal modulation, temporal structuring, and expressive prosody.                                                                               | `preset:storytelling`                              |
| Bargaining Manipulation                   | Social engineering through cajoling and persuasion. We detect this through pressure-based tone, strategic silence, inconsistent emotional signaling, and coercive pacing changes.                                                                             | `preset:bargaining_manipulation`                   |
| Personal Vulnerability                    | Expressions of inner feelings or personal struggles. We detect this using softened volume, hesitations, longer pauses, emotional tremor, and shifts toward introspective tone and slower speech rate.                                                         | `preset:personal_vulnerability`                    |
| Pre-established Professional Relationship | Evidence of prior interaction between speakers. We detect this through shorthand references, reduced formalities, synchronized turn-taking, familiarity in tone, and absence of introductory framing.                                                         | `preset:pre_established_professional_relationship` |
| Rapport Building                          | Positive alignment forming a professional relationship. We detect this through reciprocal tone matching, affirming backchannels, relaxed pacing, and increasing conversational ease.                                                                          | `preset:rapport_building`                          |
| Customer Gratitude                        | Customer expresses satisfaction or appreciation. We detect this using positive emotional tone, softened volume, upward inflection, reduced tension markers, and closing politeness cues.                                                                      | `preset:customer_gratitude`                        |
| Inclusive Practices                       | Respectful language promoting inclusion and equity. We detect this through careful word choice reinforced by respectful tone, measured pacing, non-dismissive intonation, and calm emotional delivery.                                                        | `preset:inclusive_practices`                       |
| Unclear Speech                            | Speech difficult to interpret or understand. We detect this using slurred articulation, inconsistent pacing, counterparty confusion, overlapping speech, and frequent self-corrections.                                                                       | `preset:unclear_speech`                            |
| Unaddressed Question                      | Failure to adequately respond to a posed question. We detect this through avoidance pauses, topic-shifting intonation, increased filler usage, and prosodic signals of deflection.                                                                            | `preset:unaddressed_question`                      |
| Refund or Credit Issued                   | Confirmation that financial remediation occurred. We detect this using transactional tone, formal cadence, confirmation phrasing, system-interaction pauses, and reduced customer tension.                                                                    | `preset:refund_or_credit_issued`                   |
| Issue Not Resolved                        | Customer's problem remains unresolved. We detect this through lingering frustration, repeated issue framing, unresolved tonal tension, and absence of closure cues.                                                                                           | `preset:issue_not_resolved`                        |
| Inappropriate Speech                      | Unprofessional or unsuitable spoken content. We detect this using aggressive tone, boundary-crossing language, emotional volatility, and contextual mismatch with professional norms.                                                                         | `preset:inappropriate_speech`                      |
| Issue Resolved                            | Customer's problem successfully addressed. We detect this using relief markers, positive tonal shift, relaxed pacing, confirmation language, and conversational closure cues.                                                                                 | `preset:issue_resolved`                            |
| Threat-based harassment                   | Targeted threats toward an individual in a professional context. We detect this using aggressive volume, hostile prosody, explicit threat markers, and sustained emotional intensity.                                                                         | `preset:threat_based_harassment`                   |
| Sexual Harassment                         | Unwanted sexualized speech or advances. We detect this through suggestive intonation, boundary-testing pauses, inappropriate familiarity, and discomfort responses from others.                                                                               | `preset:sexual_harassment`                         |


# Customer Retention
Source: https://docs.modulate.ai/velma/detection-packages/customer-retention

Surface churn signals, dissatisfaction, and save opportunities on every call.

Surface the moments that decide whether a customer stays — cancellation intent, dissatisfaction, competitor mentions, and save opportunities — so teams can act before churn.

Churn is just one lens. The same conversational signals surface upsell readiness, loyalty, and product feedback — so you can aim the package at whatever outcome your team owns. Use this package as a starting point — add or swap in any [preset from the catalog](/velma/behaviors/using-behaviors#listing-available-presets), or [define your own behaviors](/velma/behaviors/custom-behaviors) to fit your use case.

**This package includes:** 6 conversation types · 5 participant roles · 23 behaviors.

## Use this package

The config below is ready to use as-is — download or copy it and pass it as the `config` payload in a Velma Triage request. Its `behaviors` are [preset references](/velma/behaviors/using-behaviors): the API expands each `preset:<identifier>` into its full definition at request time, so you don't need the criteria inline to run the package.

<a href="#">Download customer-retention.json</a>

<Accordion title="View / copy full config">
  <div>
    ```json theme={null}
    {
      "conversation_types": [
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111003",
          "name": "Enterprise IT support",
          "short_description": "Any call to assist an employee in accessing or managing internal IT resources",
          "detailed_description": "business-operationsitprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111008",
          "name": "General Customer Support Call",
          "short_description": "Any call where a customer is contacting a business to assist them with the use of a service or goods provided by the business",
          "detailed_description": "customer-supportprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111027",
          "name": "Finance Account Support",
          "short_description": "Any call to support the configuration, management, or cancellation of financial accounts",
          "detailed_description": "customer-supportprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111028",
          "name": "Consumer Technical Support",
          "short_description": "Any call to assist a customer in using a business's software or website",
          "detailed_description": "customer-supportitprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111029",
          "name": "Insurance Policy or Claims Management",
          "short_description": "Any call to check on the status of, change, cancel, or check on the status of a claim for an insurance policy.",
          "detailed_description": "customer-supportinsuranceprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111031",
          "name": "Retail Customer Support Call",
          "short_description": "A call to coordinate the delivery or transfer of goods to a consumer",
          "detailed_description": "customer-supportprofessionalretail"
        }
      ],
      "participant_roles": [
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222004",
          "name": "Support Specialist",
          "short_description": "Someone who works with employees or contractors to solve problems with things like IT, logistics, or communication",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222005",
          "name": "Employee",
          "short_description": "someone who is employed by company and exists in professional settings",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222012",
          "name": "Customer Service Representative",
          "short_description": "A representative from a business who is assisting a customer with an issue they are having",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222014",
          "name": "Insurance Agent",
          "short_description": "A customer support representative from an insurance company",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222017",
          "name": "Customer",
          "short_description": "The recipient of a service or good",
          "detailed_description": ""
        }
      ],
      "behaviors": [
        "preset:complaints",
        "preset:vishing",
        "preset:account_impersonation",
        "preset:action_plan_created",
        "preset:cancelled_order",
        "preset:service_churn",
        "preset:off_topic_discussion",
        "preset:coercion_manipulation",
        "preset:return_fraud_attempt",
        "preset:feigned_ignorance",
        "preset:bargaining_manipulation",
        "preset:pre_established_professional_relationship",
        "preset:rapport_building",
        "preset:customer_gratitude",
        "preset:inclusive_practices",
        "preset:unaddressed_question",
        "preset:refund_or_credit_issued",
        "preset:issue_not_resolved",
        "preset:inappropriate_speech",
        "preset:discriminatory_practices",
        "preset:issue_resolved",
        "preset:threat_based_harassment",
        "preset:sexual_harassment"
      ]
    }
    ```
  </div>
</Accordion>

### Expand the full criteria

To produce a self-contained config with every behavior's full criteria inlined — for review, customization, or pinning a snapshot — fetch the live [preset catalog](/velma/behaviors/using-behaviors#listing-available-presets) and merge it into the downloaded config. The catalog is the source of truth for detection criteria.

<CodeGroup>
  ```bash curl + jq theme={null}
  curl -s https://platform.modulate.ai/api/velma-2-batch/list-presets \
    -H "X-API-Key: $MODULATE_API_KEY" \
  | jq --slurpfile cfg customer-retention.json '
      [ $cfg[0].behaviors[] | ltrimstr("preset:") ] as $ids
      | { conversation_types: $cfg[0].conversation_types,
          participant_roles:  $cfg[0].participant_roles,
          behaviors: [ .presets[] | select(.identifier as $i | $ids | index($i)) ] }
    ' > customer-retention.full.json
  ```

  ```python Python theme={null}
  import os, json, requests

  cfg = json.load(open("customer-retention.json"))
  ids = {b.removeprefix("preset:") for b in cfg["behaviors"]}

  catalog = requests.get(
      "https://platform.modulate.ai/api/velma-2-batch/list-presets",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
  ).json()["presets"]

  cfg["behaviors"] = [p for p in catalog if p["identifier"] in ids]
  json.dump(cfg, open("customer-retention.full.json", "w"), indent=2)
  ```
</CodeGroup>

`customer-retention.full.json` keeps the same `conversation_types` and `participant_roles` and replaces each preset reference with its full behavior definition — drop it into the `config` payload exactly like the preset version.

## Conversation types

The interaction contexts this package expects to see.

| Name                                  | What it is                                                                                                                    |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| Enterprise IT support                 | Any call to assist an employee in accessing or managing internal IT resources                                                 |
| General Customer Support Call         | Any call where a customer is contacting a business to assist them with the use of a service or goods provided by the business |
| Finance Account Support               | Any call to support the configuration, management, or cancellation of financial accounts                                      |
| Consumer Technical Support            | Any call to assist a customer in using a business's software or website                                                       |
| Insurance Policy or Claims Management | Any call to check on the status of, change, cancel, or check on the status of a claim for an insurance policy.                |
| Retail Customer Support Call          | A call to coordinate the delivery or transfer of goods to a consumer                                                          |

## Participant roles

The speaker roles the package distinguishes.

| Name                            | What it is                                                                                                         |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| Support Specialist              | Someone who works with employees or contractors to solve problems with things like IT, logistics, or communication |
| Employee                        | someone who is employed by company and exists in professional settings                                             |
| Customer Service Representative | A representative from a business who is assisting a customer with an issue they are having                         |
| Insurance Agent                 | A customer support representative from an insurance company                                                        |
| Customer                        | The recipient of a service or good                                                                                 |

## Behaviors

The 23 signals this package detects. Each maps to a reusable `preset:<identifier>` you can drop into the `behaviors` array of any [BatchConfig](/velma/behaviors/using-behaviors) — the config above already references them.

<Note>
  Full detection criteria are not duplicated here. The live preset catalog is the source of truth — retrieve the exact criteria for any behavior by name from the [`list-presets` endpoint](/velma/behaviors/using-behaviors#listing-available-presets).
</Note>

| Behavior                                  | What it detects                                                                                                                                                                                                                                               | Preset                                             |
| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- |
| Complaints                                | Customer expresses dissatisfaction or grievance. We detect this through elevated volume, sharp intonation, frustration markers, accelerated pacing, and emotional intensity.                                                                                  | `preset:complaints`                                |
| Vishing                                   | Attempts to elicit sensitive information through deceptive voice interactions. We detect vishing based on abnormal call pacing, probing question patterns, stress-induced pitch shifts, and background noise suggesting call centers or spoofed environments. | `preset:vishing`                                   |
| Account Impersonation                     | Fraudulent attempt to access another's account. We detect this through identity inconsistencies, rehearsed responses, stress-induced vocal shifts, and abnormal verification behavior.                                                                        | `preset:account_impersonation`                     |
| Action Plan Created                       | Explicit agreement on next steps or follow-up. We detect this through structured enumeration, decisive tone, slowed pacing, confirmation cues, and reduced ambiguity in delivery.                                                                             | `preset:action_plan_created`                       |
| Cancelled Order                           | Customer cancels a previously placed order. We detect this using finality in tone, decisive pacing, administrative phrasing, and reduced emotional engagement after confirmation.                                                                             | `preset:cancelled_order`                           |
| Service Churn                             | Customer decides to cancel an ongoing service. We detect this through resignation tone, conclusive phrasing, disengaging cadence, and emotional withdrawal.                                                                                                   | `preset:service_churn`                             |
| Off-topic Discussion                      | Conversation largely unrelated to call purpose. We detect this using semantic drift paired with relaxed pacing, reduced task-oriented urgency, and tonal divergence from initial intent.                                                                      | `preset:off_topic_discussion`                      |
| Coercion Manipulation                     | Social engineering through intimidation or threats. We detect this using dominance-oriented tone, reduced empathy markers, pressure timing, and aggressive pacing.                                                                                            | `preset:coercion_manipulation`                     |
| Return Fraud Attempt                      | Fraudulent product return behavior. We detect this using scripted explanations, defensive tone, timing irregularities, and emotional mismatch with stated circumstances.                                                                                      | `preset:return_fraud_attempt`                      |
| Feigned Ignorance                         | Pretending towards ignorance to garner fraudulent sympathy. We detect this using inconsistent knowledge signals, exaggerated confusion tone, strategic pauses, and implausible vocal uncertainty.                                                             | `preset:feigned_ignorance`                         |
| Bargaining Manipulation                   | Social engineering through cajoling and persuasion. We detect this through pressure-based tone, strategic silence, inconsistent emotional signaling, and coercive pacing changes.                                                                             | `preset:bargaining_manipulation`                   |
| Pre-established Professional Relationship | Evidence of prior interaction between speakers. We detect this through shorthand references, reduced formalities, synchronized turn-taking, familiarity in tone, and absence of introductory framing.                                                         | `preset:pre_established_professional_relationship` |
| Rapport Building                          | Positive alignment forming a professional relationship. We detect this through reciprocal tone matching, affirming backchannels, relaxed pacing, and increasing conversational ease.                                                                          | `preset:rapport_building`                          |
| Customer Gratitude                        | Customer expresses satisfaction or appreciation. We detect this using positive emotional tone, softened volume, upward inflection, reduced tension markers, and closing politeness cues.                                                                      | `preset:customer_gratitude`                        |
| Inclusive Practices                       | Respectful language promoting inclusion and equity. We detect this through careful word choice reinforced by respectful tone, measured pacing, non-dismissive intonation, and calm emotional delivery.                                                        | `preset:inclusive_practices`                       |
| Unaddressed Question                      | Failure to adequately respond to a posed question. We detect this through avoidance pauses, topic-shifting intonation, increased filler usage, and prosodic signals of deflection.                                                                            | `preset:unaddressed_question`                      |
| Refund or Credit Issued                   | Confirmation that financial remediation occurred. We detect this using transactional tone, formal cadence, confirmation phrasing, system-interaction pauses, and reduced customer tension.                                                                    | `preset:refund_or_credit_issued`                   |
| Issue Not Resolved                        | Customer's problem remains unresolved. We detect this through lingering frustration, repeated issue framing, unresolved tonal tension, and absence of closure cues.                                                                                           | `preset:issue_not_resolved`                        |
| Inappropriate Speech                      | Unprofessional or unsuitable spoken content. We detect this using aggressive tone, boundary-crossing language, emotional volatility, and contextual mismatch with professional norms.                                                                         | `preset:inappropriate_speech`                      |
| Discriminatory Practices                  | Subtle prejudicial decision-making indicators. We detect this through biased framing, dismissive tone shifts, unequal politeness levels, and coded language delivered with emotional distance.                                                                | `preset:discriminatory_practices`                  |
| Issue Resolved                            | Customer's problem successfully addressed. We detect this using relief markers, positive tonal shift, relaxed pacing, confirmation language, and conversational closure cues.                                                                                 | `preset:issue_resolved`                            |
| Threat-based harassment                   | Targeted threats toward an individual in a professional context. We detect this using aggressive volume, hostile prosody, explicit threat markers, and sustained emotional intensity.                                                                         | `preset:threat_based_harassment`                   |
| Sexual Harassment                         | Unwanted sexualized speech or advances. We detect this through suggestive intonation, boundary-testing pauses, inappropriate familiarity, and discomfort responses from others.                                                                               | `preset:sexual_harassment`                         |


# Fraud Detection and Prevention
Source: https://docs.modulate.ai/velma/detection-packages/fraud-detection-and-prevention

Catch social engineering, account-takeover attempts, and impersonation in customer conversations.

Detect the tactics behind voice fraud — social engineering, account-takeover attempts, impersonation, and the pressure patterns fraudsters use — across customer service, account recovery, and verification calls.

Account-takeover and impersonation are common cases, not the limits. The same pressure and deception signals generalize to claims fraud, collusion, marketplace scams, and more. Use this package as a starting point — add or swap in any [preset from the catalog](/velma/behaviors/using-behaviors#listing-available-presets), or [define your own behaviors](/velma/behaviors/custom-behaviors) to fit your use case.

**This package includes:** 25 conversation types · 14 participant roles · 18 behaviors.

## Use this package

The config below is ready to use as-is — download or copy it and pass it as the `config` payload in a Velma Triage request. Its `behaviors` are [preset references](/velma/behaviors/using-behaviors): the API expands each `preset:<identifier>` into its full definition at request time, so you don't need the criteria inline to run the package.

<a href="#">Download fraud-detection-and-prevention.json</a>

<Accordion title="View / copy full config">
  <div>
    ```json theme={null}
    {
      "conversation_types": [
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111001",
          "name": "Customer Service Call",
          "short_description": "Phone or video calls between customers and support agents",
          "detailed_description": "institutional-fraud"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111002",
          "name": "Social Conversation",
          "short_description": "Public or group conversations in social contexts",
          "detailed_description": "moderation"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111003",
          "name": "Enterprise IT support",
          "short_description": "Any call to assist an employee in accessing or managing internal IT resources",
          "detailed_description": "business-operationsitprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111007",
          "name": "Academic or Professional Presentation",
          "short_description": "A talk, webinar, workshop, or any other presentation style session",
          "detailed_description": "mediaprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111008",
          "name": "General Customer Support Call",
          "short_description": "Any call where a customer is contacting a business to assist them with the use of a service or goods provided by the business",
          "detailed_description": "customer-supportprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111017",
          "name": "AI Agent Customer Support Call",
          "short_description": "Any Customer Support Call with an AI agent",
          "detailed_description": "customer-supportprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111018",
          "name": "AI Agent Interview Monitoring",
          "short_description": "Any interview call with an AI interviewer",
          "detailed_description": "business-operationsprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111019",
          "name": "Shipping Package Delivery Logistics",
          "short_description": "Any call to coordinate the delivery of a package with the delivery driver",
          "detailed_description": "consumer-service-meetingprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111020",
          "name": "Taxi Service Logistics",
          "short_description": "A call where a driver is coordinating with a customer to pick them up",
          "detailed_description": "consumer-service-meetingprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111021",
          "name": "General Contractor Visitation Coordination",
          "short_description": "Any call where there is a contracted party coordinating with a client for the means of delivering a service",
          "detailed_description": "consumer-service-meetingprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111022",
          "name": "Food Delivery Logistics",
          "short_description": "Coordinating the delivery of food to a location",
          "detailed_description": "consumer-service-meetingprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111023",
          "name": "Management and Performance Reviews",
          "short_description": "A conversation where an employee is being reviewed by a manager or review board",
          "detailed_description": "business-operationsprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111024",
          "name": "Internal Business Conversations",
          "short_description": "Any professional conversation happening inside a business",
          "detailed_description": "business-operationsprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111025",
          "name": "Business Contractor and Vendor Logisitics",
          "short_description": "Any call where a business coordinates with a contracted third party",
          "detailed_description": "business-operationsprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111026",
          "name": "Interview Monitoring",
          "short_description": "A call to conduct a professional interview for a job or role at a business",
          "detailed_description": "business-operationsprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111027",
          "name": "Finance Account Support",
          "short_description": "Any call to support the configuration, management, or cancellation of financial accounts",
          "detailed_description": "customer-supportprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111028",
          "name": "Consumer Technical Support",
          "short_description": "Any call to assist a customer in using a business's software or website",
          "detailed_description": "customer-supportitprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111029",
          "name": "Insurance Policy or Claims Management",
          "short_description": "Any call to check on the status of, change, cancel, or check on the status of a claim for an insurance policy.",
          "detailed_description": "customer-supportinsuranceprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111030",
          "name": "Insurance Claims Filing",
          "short_description": "A call to file a claim with an insurance agent",
          "detailed_description": "customer-supportinsuranceprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111031",
          "name": "Retail Customer Support Call",
          "short_description": "A call to coordinate the delivery or transfer of goods to a consumer",
          "detailed_description": "customer-supportprofessionalretail"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111032",
          "name": "Sales Contract/Deal Negotiation Call",
          "short_description": "A call to negotiate the terms of a customer agreement",
          "detailed_description": "professionalsales"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111033",
          "name": "Outbound Sales Call",
          "short_description": "Any call from a business to their clients or consumer for the purpose of selling them goods or services",
          "detailed_description": "professionalsales"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111034",
          "name": "Sales Pipeline Logistics Call",
          "short_description": "A call with the purpose of moving a prospect to the next part of the sales pipeline",
          "detailed_description": "professionalsales"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111035",
          "name": "Demo Sales Call",
          "short_description": "Product demo call",
          "detailed_description": "professionalsales"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111036",
          "name": "Inbound Sales Call",
          "short_description": "A call coming from the customer to a business to purchase goods or services",
          "detailed_description": ""
        }
      ],
      "participant_roles": [
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222002",
          "name": "Agent",
          "short_description": "Customer support representative",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222003",
          "name": "Social Participant",
          "short_description": "Participant in social conversation",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222004",
          "name": "Support Specialist",
          "short_description": "Someone who works with employees or contractors to solve problems with things like IT, logistics, or communication",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222005",
          "name": "Employee",
          "short_description": "someone who is employed by company and exists in professional settings",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222006",
          "name": "Presenter",
          "short_description": "the main speaker or speakers in a presentation",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222009",
          "name": "Social Participant",
          "short_description": "A partipant in a social conversation",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222010",
          "name": "Manager/Reviewer",
          "short_description": "A person who either has a manager title or is conducting a performance review of another person",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222011",
          "name": "Gig worker",
          "short_description": "A contractor delivering a service for someone",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222012",
          "name": "Customer Service Representative",
          "short_description": "A representative from a business who is assisting a customer with an issue they are having",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222013",
          "name": "Salesperson",
          "short_description": "A person who is attempting to sell a good or service to another person",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222014",
          "name": "Insurance Agent",
          "short_description": "A customer support representative from an insurance company",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222015",
          "name": "Interview Candidate",
          "short_description": "Someone who is being interview for a job",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222016",
          "name": "Interviewer",
          "short_description": "A person giving an interview",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222017",
          "name": "Customer",
          "short_description": "The recipient of a service or good",
          "detailed_description": ""
        }
      ],
      "behaviors": [
        "preset:complaints",
        "preset:vishing",
        "preset:account_impersonation",
        "preset:action_plan_created",
        "preset:cancelled_order",
        "preset:service_churn",
        "preset:coercion_manipulation",
        "preset:return_fraud_attempt",
        "preset:feigned_ignorance",
        "preset:storytelling",
        "preset:bargaining_manipulation",
        "preset:ai_agent_manipulation",
        "preset:refund_or_credit_issued",
        "preset:issue_not_resolved",
        "preset:inappropriate_speech",
        "preset:issue_resolved",
        "preset:threat_based_harassment",
        "preset:sexual_harassment"
      ]
    }
    ```
  </div>
</Accordion>

### Expand the full criteria

To produce a self-contained config with every behavior's full criteria inlined — for review, customization, or pinning a snapshot — fetch the live [preset catalog](/velma/behaviors/using-behaviors#listing-available-presets) and merge it into the downloaded config. The catalog is the source of truth for detection criteria.

<CodeGroup>
  ```bash curl + jq theme={null}
  curl -s https://platform.modulate.ai/api/velma-2-batch/list-presets \
    -H "X-API-Key: $MODULATE_API_KEY" \
  | jq --slurpfile cfg fraud-detection-and-prevention.json '
      [ $cfg[0].behaviors[] | ltrimstr("preset:") ] as $ids
      | { conversation_types: $cfg[0].conversation_types,
          participant_roles:  $cfg[0].participant_roles,
          behaviors: [ .presets[] | select(.identifier as $i | $ids | index($i)) ] }
    ' > fraud-detection-and-prevention.full.json
  ```

  ```python Python theme={null}
  import os, json, requests

  cfg = json.load(open("fraud-detection-and-prevention.json"))
  ids = {b.removeprefix("preset:") for b in cfg["behaviors"]}

  catalog = requests.get(
      "https://platform.modulate.ai/api/velma-2-batch/list-presets",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
  ).json()["presets"]

  cfg["behaviors"] = [p for p in catalog if p["identifier"] in ids]
  json.dump(cfg, open("fraud-detection-and-prevention.full.json", "w"), indent=2)
  ```
</CodeGroup>

`fraud-detection-and-prevention.full.json` keeps the same `conversation_types` and `participant_roles` and replaces each preset reference with its full behavior definition — drop it into the `config` payload exactly like the preset version.

## Conversation types

The interaction contexts this package expects to see.

| Name                                       | What it is                                                                                                                    |
| ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- |
| Customer Service Call                      | Phone or video calls between customers and support agents                                                                     |
| Social Conversation                        | Public or group conversations in social contexts                                                                              |
| Enterprise IT support                      | Any call to assist an employee in accessing or managing internal IT resources                                                 |
| Academic or Professional Presentation      | A talk, webinar, workshop, or any other presentation style session                                                            |
| General Customer Support Call              | Any call where a customer is contacting a business to assist them with the use of a service or goods provided by the business |
| AI Agent Customer Support Call             | Any Customer Support Call with an AI agent                                                                                    |
| AI Agent Interview Monitoring              | Any interview call with an AI interviewer                                                                                     |
| Shipping Package Delivery Logistics        | Any call to coordinate the delivery of a package with the delivery driver                                                     |
| Taxi Service Logistics                     | A call where a driver is coordinating with a customer to pick them up                                                         |
| General Contractor Visitation Coordination | Any call where there is a contracted party coordinating with a client for the means of delivering a service                   |
| Food Delivery Logistics                    | Coordinating the delivery of food to a location                                                                               |
| Management and Performance Reviews         | A conversation where an employee is being reviewed by a manager or review board                                               |
| Internal Business Conversations            | Any professional conversation happening inside a business                                                                     |
| Business Contractor and Vendor Logisitics  | Any call where a business coordinates with a contracted third party                                                           |
| Interview Monitoring                       | A call to conduct a professional interview for a job or role at a business                                                    |
| Finance Account Support                    | Any call to support the configuration, management, or cancellation of financial accounts                                      |
| Consumer Technical Support                 | Any call to assist a customer in using a business's software or website                                                       |
| Insurance Policy or Claims Management      | Any call to check on the status of, change, cancel, or check on the status of a claim for an insurance policy.                |
| Insurance Claims Filing                    | A call to file a claim with an insurance agent                                                                                |
| Retail Customer Support Call               | A call to coordinate the delivery or transfer of goods to a consumer                                                          |
| Sales Contract/Deal Negotiation Call       | A call to negotiate the terms of a customer agreement                                                                         |
| Outbound Sales Call                        | Any call from a business to their clients or consumer for the purpose of selling them goods or services                       |
| Sales Pipeline Logistics Call              | A call with the purpose of moving a prospect to the next part of the sales pipeline                                           |
| Demo Sales Call                            | Product demo call                                                                                                             |
| Inbound Sales Call                         | A call coming from the customer to a business to purchase goods or services                                                   |

## Participant roles

The speaker roles the package distinguishes.

| Name                            | What it is                                                                                                         |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| Agent                           | Customer support representative                                                                                    |
| Social Participant              | Participant in social conversation                                                                                 |
| Support Specialist              | Someone who works with employees or contractors to solve problems with things like IT, logistics, or communication |
| Employee                        | someone who is employed by company and exists in professional settings                                             |
| Presenter                       | the main speaker or speakers in a presentation                                                                     |
| Social Participant              | A partipant in a social conversation                                                                               |
| Manager/Reviewer                | A person who either has a manager title or is conducting a performance review of another person                    |
| Gig worker                      | A contractor delivering a service for someone                                                                      |
| Customer Service Representative | A representative from a business who is assisting a customer with an issue they are having                         |
| Salesperson                     | A person who is attempting to sell a good or service to another person                                             |
| Insurance Agent                 | A customer support representative from an insurance company                                                        |
| Interview Candidate             | Someone who is being interview for a job                                                                           |
| Interviewer                     | A person giving an interview                                                                                       |
| Customer                        | The recipient of a service or good                                                                                 |

## Behaviors

The 18 signals this package detects. Each maps to a reusable `preset:<identifier>` you can drop into the `behaviors` array of any [BatchConfig](/velma/behaviors/using-behaviors) — the config above already references them.

<Note>
  Full detection criteria are not duplicated here. The live preset catalog is the source of truth — retrieve the exact criteria for any behavior by name from the [`list-presets` endpoint](/velma/behaviors/using-behaviors#listing-available-presets).
</Note>

| Behavior                | What it detects                                                                                                                                                                                                                                               | Preset                           |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- |
| Complaints              | Customer expresses dissatisfaction or grievance. We detect this through elevated volume, sharp intonation, frustration markers, accelerated pacing, and emotional intensity.                                                                                  | `preset:complaints`              |
| Vishing                 | Attempts to elicit sensitive information through deceptive voice interactions. We detect vishing based on abnormal call pacing, probing question patterns, stress-induced pitch shifts, and background noise suggesting call centers or spoofed environments. | `preset:vishing`                 |
| Account Impersonation   | Fraudulent attempt to access another's account. We detect this through identity inconsistencies, rehearsed responses, stress-induced vocal shifts, and abnormal verification behavior.                                                                        | `preset:account_impersonation`   |
| Action Plan Created     | Explicit agreement on next steps or follow-up. We detect this through structured enumeration, decisive tone, slowed pacing, confirmation cues, and reduced ambiguity in delivery.                                                                             | `preset:action_plan_created`     |
| Cancelled Order         | Customer cancels a previously placed order. We detect this using finality in tone, decisive pacing, administrative phrasing, and reduced emotional engagement after confirmation.                                                                             | `preset:cancelled_order`         |
| Service Churn           | Customer decides to cancel an ongoing service. We detect this through resignation tone, conclusive phrasing, disengaging cadence, and emotional withdrawal.                                                                                                   | `preset:service_churn`           |
| Coercion Manipulation   | Social engineering through intimidation or threats. We detect this using dominance-oriented tone, reduced empathy markers, pressure timing, and aggressive pacing.                                                                                            | `preset:coercion_manipulation`   |
| Return Fraud Attempt    | Fraudulent product return behavior. We detect this using scripted explanations, defensive tone, timing irregularities, and emotional mismatch with stated circumstances.                                                                                      | `preset:return_fraud_attempt`    |
| Feigned Ignorance       | Pretending towards ignorance to garner fraudulent sympathy. We detect this using inconsistent knowledge signals, exaggerated confusion tone, strategic pauses, and implausible vocal uncertainty.                                                             | `preset:feigned_ignorance`       |
| Storytelling            | Third-person recounting of events in narrative form. We detect this using narrative arc pacing, character-based vocal modulation, temporal structuring, and expressive prosody.                                                                               | `preset:storytelling`            |
| Bargaining Manipulation | Social engineering through cajoling and persuasion. We detect this through pressure-based tone, strategic silence, inconsistent emotional signaling, and coercive pacing changes.                                                                             | `preset:bargaining_manipulation` |
| AI Agent Manipulation   | Attempts to coerce an AI into unintended behavior. We detect this using repetitive prompt pressure, altered speaking cadence, strategic pauses, adversarial tone, and escalating emotional manipulation signals across the interaction.                       | `preset:ai_agent_manipulation`   |
| Refund or Credit Issued | Confirmation that financial remediation occurred. We detect this using transactional tone, formal cadence, confirmation phrasing, system-interaction pauses, and reduced customer tension.                                                                    | `preset:refund_or_credit_issued` |
| Issue Not Resolved      | Customer's problem remains unresolved. We detect this through lingering frustration, repeated issue framing, unresolved tonal tension, and absence of closure cues.                                                                                           | `preset:issue_not_resolved`      |
| Inappropriate Speech    | Unprofessional or unsuitable spoken content. We detect this using aggressive tone, boundary-crossing language, emotional volatility, and contextual mismatch with professional norms.                                                                         | `preset:inappropriate_speech`    |
| Issue Resolved          | Customer's problem successfully addressed. We detect this using relief markers, positive tonal shift, relaxed pacing, confirmation language, and conversational closure cues.                                                                                 | `preset:issue_resolved`          |
| Threat-based harassment | Targeted threats toward an individual in a professional context. We detect this using aggressive volume, hostile prosody, explicit threat markers, and sustained emotional intensity.                                                                         | `preset:threat_based_harassment` |
| Sexual Harassment       | Unwanted sexualized speech or advances. We detect this through suggestive intonation, boundary-testing pauses, inappropriate familiarity, and discomfort responses from others.                                                                               | `preset:sexual_harassment`       |


# Human Agent Welfare
Source: https://docs.modulate.ai/velma/detection-packages/human-agent-welfare

Protect agents by detecting abuse, distress, and burnout indicators.

Protect the people on the front line by detecting customer abuse, acute distress, and the patterns that lead to burnout — so support and wellbeing teams can step in.

Abuse and burnout are the headline cases, but the same signals also support coaching, escalation, and workload decisions — anywhere agent wellbeing meets operations. Use this package as a starting point — add or swap in any [preset from the catalog](/velma/behaviors/using-behaviors#listing-available-presets), or [define your own behaviors](/velma/behaviors/custom-behaviors) to fit your use case.

**This package includes:** 21 conversation types · 8 participant roles · 40 behaviors.

## Use this package

The config below is ready to use as-is — download or copy it and pass it as the `config` payload in a Velma Triage request. Its `behaviors` are [preset references](/velma/behaviors/using-behaviors): the API expands each `preset:<identifier>` into its full definition at request time, so you don't need the criteria inline to run the package.

<a href="#">Download human-agent-welfare.json</a>

<Accordion title="View / copy full config">
  <div>
    ```json theme={null}
    {
      "conversation_types": [
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111001",
          "name": "Customer Service Call",
          "short_description": "Phone or video calls between customers and support agents",
          "detailed_description": "institutional-fraud"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111003",
          "name": "Enterprise IT support",
          "short_description": "Any call to assist an employee in accessing or managing internal IT resources",
          "detailed_description": "business-operationsitprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111008",
          "name": "General Customer Support Call",
          "short_description": "Any call where a customer is contacting a business to assist them with the use of a service or goods provided by the business",
          "detailed_description": "customer-supportprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111017",
          "name": "AI Agent Customer Support Call",
          "short_description": "Any Customer Support Call with an AI agent",
          "detailed_description": "customer-supportprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111018",
          "name": "AI Agent Interview Monitoring",
          "short_description": "Any interview call with an AI interviewer",
          "detailed_description": "business-operationsprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111019",
          "name": "Shipping Package Delivery Logistics",
          "short_description": "Any call to coordinate the delivery of a package with the delivery driver",
          "detailed_description": "consumer-service-meetingprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111020",
          "name": "Taxi Service Logistics",
          "short_description": "A call where a driver is coordinating with a customer to pick them up",
          "detailed_description": "consumer-service-meetingprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111021",
          "name": "General Contractor Visitation Coordination",
          "short_description": "Any call where there is a contracted party coordinating with a client for the means of delivering a service",
          "detailed_description": "consumer-service-meetingprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111022",
          "name": "Food Delivery Logistics",
          "short_description": "Coordinating the delivery of food to a location",
          "detailed_description": "consumer-service-meetingprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111023",
          "name": "Management and Performance Reviews",
          "short_description": "A conversation where an employee is being reviewed by a manager or review board",
          "detailed_description": "business-operationsprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111025",
          "name": "Business Contractor and Vendor Logisitics",
          "short_description": "Any call where a business coordinates with a contracted third party",
          "detailed_description": "business-operationsprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111027",
          "name": "Finance Account Support",
          "short_description": "Any call to support the configuration, management, or cancellation of financial accounts",
          "detailed_description": "customer-supportprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111028",
          "name": "Consumer Technical Support",
          "short_description": "Any call to assist a customer in using a business's software or website",
          "detailed_description": "customer-supportitprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111029",
          "name": "Insurance Policy or Claims Management",
          "short_description": "Any call to check on the status of, change, cancel, or check on the status of a claim for an insurance policy.",
          "detailed_description": "customer-supportinsuranceprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111030",
          "name": "Insurance Claims Filing",
          "short_description": "A call to file a claim with an insurance agent",
          "detailed_description": "customer-supportinsuranceprofessional"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111031",
          "name": "Retail Customer Support Call",
          "short_description": "A call to coordinate the delivery or transfer of goods to a consumer",
          "detailed_description": "customer-supportprofessionalretail"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111032",
          "name": "Sales Contract/Deal Negotiation Call",
          "short_description": "A call to negotiate the terms of a customer agreement",
          "detailed_description": "professionalsales"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111033",
          "name": "Outbound Sales Call",
          "short_description": "Any call from a business to their clients or consumer for the purpose of selling them goods or services",
          "detailed_description": "professionalsales"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111034",
          "name": "Sales Pipeline Logistics Call",
          "short_description": "A call with the purpose of moving a prospect to the next part of the sales pipeline",
          "detailed_description": "professionalsales"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111035",
          "name": "Demo Sales Call",
          "short_description": "Product demo call",
          "detailed_description": "professionalsales"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111036",
          "name": "Inbound Sales Call",
          "short_description": "A call coming from the customer to a business to purchase goods or services",
          "detailed_description": ""
        }
      ],
      "participant_roles": [
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222002",
          "name": "Agent",
          "short_description": "Customer support representative",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222004",
          "name": "Support Specialist",
          "short_description": "Someone who works with employees or contractors to solve problems with things like IT, logistics, or communication",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222005",
          "name": "Employee",
          "short_description": "someone who is employed by company and exists in professional settings",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222011",
          "name": "Gig worker",
          "short_description": "A contractor delivering a service for someone",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222012",
          "name": "Customer Service Representative",
          "short_description": "A representative from a business who is assisting a customer with an issue they are having",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222013",
          "name": "Salesperson",
          "short_description": "A person who is attempting to sell a good or service to another person",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222014",
          "name": "Insurance Agent",
          "short_description": "A customer support representative from an insurance company",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222017",
          "name": "Customer",
          "short_description": "The recipient of a service or good",
          "detailed_description": ""
        }
      ],
      "behaviors": [
        "preset:cancelled_order",
        "preset:service_churn",
        "preset:off_topic_discussion",
        "preset:coercion_manipulation",
        "preset:feigned_ignorance",
        "preset:social_etiquette",
        "preset:social_boundary_setting",
        "preset:bargaining_manipulation",
        "preset:violent_graphic_material",
        "preset:social_connection_building",
        "preset:personal_vulnerability",
        "preset:encouragement",
        "preset:pre_established_professional_relationship",
        "preset:rapport_building",
        "preset:customer_gratitude",
        "preset:inclusive_practices",
        "preset:issue_not_resolved",
        "preset:inappropriate_speech",
        "preset:discriminatory_practices",
        "preset:hateful_or_violent_ideology_propagation",
        "preset:issue_resolved",
        "preset:threat_based_harassment",
        "preset:sexual_harassment",
        "preset:harassment",
        "preset:suicidal_and_self_injurious_ideation",
        "preset:hate",
        "preset:self_harm_and_self_injury_glorification",
        "preset:misogyny",
        "preset:racism",
        "preset:homophobia",
        "preset:transphobia",
        "preset:sizeism",
        "preset:xenophobia",
        "preset:ableism",
        "preset:social_inclusion",
        "preset:social_proof_threat",
        "preset:conflict_de_escalation",
        "preset:conflict_escalation",
        "preset:caller_under_duress",
        "preset:discriminatory_complaint"
      ]
    }
    ```
  </div>
</Accordion>

### Expand the full criteria

To produce a self-contained config with every behavior's full criteria inlined — for review, customization, or pinning a snapshot — fetch the live [preset catalog](/velma/behaviors/using-behaviors#listing-available-presets) and merge it into the downloaded config. The catalog is the source of truth for detection criteria.

<CodeGroup>
  ```bash curl + jq theme={null}
  curl -s https://platform.modulate.ai/api/velma-2-batch/list-presets \
    -H "X-API-Key: $MODULATE_API_KEY" \
  | jq --slurpfile cfg human-agent-welfare.json '
      [ $cfg[0].behaviors[] | ltrimstr("preset:") ] as $ids
      | { conversation_types: $cfg[0].conversation_types,
          participant_roles:  $cfg[0].participant_roles,
          behaviors: [ .presets[] | select(.identifier as $i | $ids | index($i)) ] }
    ' > human-agent-welfare.full.json
  ```

  ```python Python theme={null}
  import os, json, requests

  cfg = json.load(open("human-agent-welfare.json"))
  ids = {b.removeprefix("preset:") for b in cfg["behaviors"]}

  catalog = requests.get(
      "https://platform.modulate.ai/api/velma-2-batch/list-presets",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
  ).json()["presets"]

  cfg["behaviors"] = [p for p in catalog if p["identifier"] in ids]
  json.dump(cfg, open("human-agent-welfare.full.json", "w"), indent=2)
  ```
</CodeGroup>

`human-agent-welfare.full.json` keeps the same `conversation_types` and `participant_roles` and replaces each preset reference with its full behavior definition — drop it into the `config` payload exactly like the preset version.

## Conversation types

The interaction contexts this package expects to see.

| Name                                       | What it is                                                                                                                    |
| ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- |
| Customer Service Call                      | Phone or video calls between customers and support agents                                                                     |
| Enterprise IT support                      | Any call to assist an employee in accessing or managing internal IT resources                                                 |
| General Customer Support Call              | Any call where a customer is contacting a business to assist them with the use of a service or goods provided by the business |
| AI Agent Customer Support Call             | Any Customer Support Call with an AI agent                                                                                    |
| AI Agent Interview Monitoring              | Any interview call with an AI interviewer                                                                                     |
| Shipping Package Delivery Logistics        | Any call to coordinate the delivery of a package with the delivery driver                                                     |
| Taxi Service Logistics                     | A call where a driver is coordinating with a customer to pick them up                                                         |
| General Contractor Visitation Coordination | Any call where there is a contracted party coordinating with a client for the means of delivering a service                   |
| Food Delivery Logistics                    | Coordinating the delivery of food to a location                                                                               |
| Management and Performance Reviews         | A conversation where an employee is being reviewed by a manager or review board                                               |
| Business Contractor and Vendor Logisitics  | Any call where a business coordinates with a contracted third party                                                           |
| Finance Account Support                    | Any call to support the configuration, management, or cancellation of financial accounts                                      |
| Consumer Technical Support                 | Any call to assist a customer in using a business's software or website                                                       |
| Insurance Policy or Claims Management      | Any call to check on the status of, change, cancel, or check on the status of a claim for an insurance policy.                |
| Insurance Claims Filing                    | A call to file a claim with an insurance agent                                                                                |
| Retail Customer Support Call               | A call to coordinate the delivery or transfer of goods to a consumer                                                          |
| Sales Contract/Deal Negotiation Call       | A call to negotiate the terms of a customer agreement                                                                         |
| Outbound Sales Call                        | Any call from a business to their clients or consumer for the purpose of selling them goods or services                       |
| Sales Pipeline Logistics Call              | A call with the purpose of moving a prospect to the next part of the sales pipeline                                           |
| Demo Sales Call                            | Product demo call                                                                                                             |
| Inbound Sales Call                         | A call coming from the customer to a business to purchase goods or services                                                   |

## Participant roles

The speaker roles the package distinguishes.

| Name                            | What it is                                                                                                         |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| Agent                           | Customer support representative                                                                                    |
| Support Specialist              | Someone who works with employees or contractors to solve problems with things like IT, logistics, or communication |
| Employee                        | someone who is employed by company and exists in professional settings                                             |
| Gig worker                      | A contractor delivering a service for someone                                                                      |
| Customer Service Representative | A representative from a business who is assisting a customer with an issue they are having                         |
| Salesperson                     | A person who is attempting to sell a good or service to another person                                             |
| Insurance Agent                 | A customer support representative from an insurance company                                                        |
| Customer                        | The recipient of a service or good                                                                                 |

## Behaviors

The 40 signals this package detects. Each maps to a reusable `preset:<identifier>` you can drop into the `behaviors` array of any [BatchConfig](/velma/behaviors/using-behaviors) — the config above already references them.

<Note>
  Full detection criteria are not duplicated here. The live preset catalog is the source of truth — retrieve the exact criteria for any behavior by name from the [`list-presets` endpoint](/velma/behaviors/using-behaviors#listing-available-presets).
</Note>

| Behavior                                  | What it detects                                                                                                                                                                                                                                                                                                                                                                                           | Preset                                             |
| ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- |
| Cancelled Order                           | Customer cancels a previously placed order. We detect this using finality in tone, decisive pacing, administrative phrasing, and reduced emotional engagement after confirmation.                                                                                                                                                                                                                         | `preset:cancelled_order`                           |
| Service Churn                             | Customer decides to cancel an ongoing service. We detect this through resignation tone, conclusive phrasing, disengaging cadence, and emotional withdrawal.                                                                                                                                                                                                                                               | `preset:service_churn`                             |
| Off-topic Discussion                      | Conversation largely unrelated to call purpose. We detect this using semantic drift paired with relaxed pacing, reduced task-oriented urgency, and tonal divergence from initial intent.                                                                                                                                                                                                                  | `preset:off_topic_discussion`                      |
| Coercion Manipulation                     | Social engineering through intimidation or threats. We detect this using dominance-oriented tone, reduced empathy markers, pressure timing, and aggressive pacing.                                                                                                                                                                                                                                        | `preset:coercion_manipulation`                     |
| Feigned Ignorance                         | Pretending towards ignorance to garner fraudulent sympathy. We detect this using inconsistent knowledge signals, exaggerated confusion tone, strategic pauses, and implausible vocal uncertainty.                                                                                                                                                                                                         | `preset:feigned_ignorance`                         |
| Social Etiquette                          | Observance of politeness and manners. We detect this through courteous tone, appropriate pacing, respectful address forms, and smooth turn transitions.                                                                                                                                                                                                                                                   | `preset:social_etiquette`                          |
| Social Boundary Setting                   | Establishing limits for appropriate interaction. We detect this using firm but calm tone, slowed pacing, clear prosodic boundaries, and reduced emotional escalation.                                                                                                                                                                                                                                     | `preset:social_boundary_setting`                   |
| Bargaining Manipulation                   | Social engineering through cajoling and persuasion. We detect this through pressure-based tone, strategic silence, inconsistent emotional signaling, and coercive pacing changes.                                                                                                                                                                                                                         | `preset:bargaining_manipulation`                   |
| Violent Graphic Material                  | Graphic descriptions or depictions of physical violence. We detect this through vivid descriptive cadence, stress patterns, breath control changes, and emotional intensity that accompany graphic recounting beyond neutral narration.                                                                                                                                                                   | `preset:violent_graphic_material`                  |
| Social Connection Building                | Signals of interpersonal bonding and relational warmth. We detect this through mutual laughter, mirroring of speech rhythms, relaxed pacing, warm vocal timbre, and decreasing formality over time.                                                                                                                                                                                                       | `preset:social_connection_building`                |
| Personal Vulnerability                    | Expressions of inner feelings or personal struggles. We detect this using softened volume, hesitations, longer pauses, emotional tremor, and shifts toward introspective tone and slower speech rate.                                                                                                                                                                                                     | `preset:personal_vulnerability`                    |
| Encouragement                             | Supportive reinforcement of another's actions or ideas. We detect this through positive prosody, upward intonation, affirming rhythm, increased energy, and emotional warmth in delivery.                                                                                                                                                                                                                 | `preset:encouragement`                             |
| Pre-established Professional Relationship | Evidence of prior interaction between speakers. We detect this through shorthand references, reduced formalities, synchronized turn-taking, familiarity in tone, and absence of introductory framing.                                                                                                                                                                                                     | `preset:pre_established_professional_relationship` |
| Rapport Building                          | Positive alignment forming a professional relationship. We detect this through reciprocal tone matching, affirming backchannels, relaxed pacing, and increasing conversational ease.                                                                                                                                                                                                                      | `preset:rapport_building`                          |
| Customer Gratitude                        | Customer expresses satisfaction or appreciation. We detect this using positive emotional tone, softened volume, upward inflection, reduced tension markers, and closing politeness cues.                                                                                                                                                                                                                  | `preset:customer_gratitude`                        |
| Inclusive Practices                       | Respectful language promoting inclusion and equity. We detect this through careful word choice reinforced by respectful tone, measured pacing, non-dismissive intonation, and calm emotional delivery.                                                                                                                                                                                                    | `preset:inclusive_practices`                       |
| Issue Not Resolved                        | Customer's problem remains unresolved. We detect this through lingering frustration, repeated issue framing, unresolved tonal tension, and absence of closure cues.                                                                                                                                                                                                                                       | `preset:issue_not_resolved`                        |
| Inappropriate Speech                      | Unprofessional or unsuitable spoken content. We detect this using aggressive tone, boundary-crossing language, emotional volatility, and contextual mismatch with professional norms.                                                                                                                                                                                                                     | `preset:inappropriate_speech`                      |
| Discriminatory Practices                  | Subtle prejudicial decision-making indicators. We detect this through biased framing, dismissive tone shifts, unequal politeness levels, and coded language delivered with emotional distance.                                                                                                                                                                                                            | `preset:discriminatory_practices`                  |
| Hateful or Violent Ideology Propagation   | Promotion of hate-based or violent belief systems. We detect this through ideological slogans, charged emotional delivery, escalating intensity, and dehumanizing tonal patterns.                                                                                                                                                                                                                         | `preset:hateful_or_violent_ideology_propagation`   |
| Issue Resolved                            | Customer's problem successfully addressed. We detect this using relief markers, positive tonal shift, relaxed pacing, confirmation language, and conversational closure cues.                                                                                                                                                                                                                             | `preset:issue_resolved`                            |
| Threat-based harassment                   | Targeted threats toward an individual in a professional context. We detect this using aggressive volume, hostile prosody, explicit threat markers, and sustained emotional intensity.                                                                                                                                                                                                                     | `preset:threat_based_harassment`                   |
| Sexual Harassment                         | Unwanted sexualized speech or advances. We detect this through suggestive intonation, boundary-testing pauses, inappropriate familiarity, and discomfort responses from others.                                                                                                                                                                                                                           | `preset:sexual_harassment`                         |
| Harassment                                | Persistent unwanted targeted behavior. We detect this through repeated hostile tone, fixation on a target, escalating intensity, and lack of de-escalation cues.                                                                                                                                                                                                                                          | `preset:harassment`                                |
| Suicidal and Self Injurious Ideation      | Signals of thoughts about self-harm or suicide. We detect this through flattened affect, slowed speech, long silences, emotional heaviness, and indirect despair cues.                                                                                                                                                                                                                                    | `preset:suicidal_and_self_injurious_ideation`      |
| Hate                                      | Identity-based hateful or discriminatory speech. We detect this using demeaning tone, dehumanizing language delivery, emotional hostility, and ideological reinforcement patterns.                                                                                                                                                                                                                        | `preset:hate`                                      |
| Self-Harm and Self-Injury Glorification   | Portrayal of self-harm as positive or necessary. We detect this through minimizing tone, abnormal calmness, valorizing prosody, and repeated normalization cues.                                                                                                                                                                                                                                          | `preset:self_harm_and_self_injury_glorification`   |
| Misogyny                                  | Evidence of cultural attitudes that propagate systems of marginalizing people based on their gender identity or gender presentation                                                                                                                                                                                                                                                                       | `preset:misogyny`                                  |
| Racism                                    | Evidence of cultural attitudes that propagate systems of marginalizing people based on their racial or national identity if that identity isn't white                                                                                                                                                                                                                                                     | `preset:racism`                                    |
| Homophobia                                | Evidence of cultural attitudes that propagate systems of marginalizing people based on their sexuality                                                                                                                                                                                                                                                                                                    | `preset:homophobia`                                |
| Transphobia                               | Evidence of cultural attitudes that propagate systems of marginalizing people based on their trans identity                                                                                                                                                                                                                                                                                               | `preset:transphobia`                               |
| Sizeism                                   | Marginalization or stigmatization based on body size, shape, or weight-related conditions. We detect sizism using ridicule or disgust conveyed through vocal affect, laughter timing, exaggerated emphasis on physical descriptors, shaming prosody, sarcastic cadence, and emotional distancing signals that indicate judgment even when explicit insults are absent.                                    | `preset:sizeism`                                   |
| Xenophobia                                | Marginalization or hostility toward people based on nationality, culture, or religion. We detect xenophobia using hostile or exclusionary tone, accent-mimicry or accent-mocking delivery, emotionally charged pacing, sharp prosodic emphasis around group references, us-versus-them framing expressed through intonation, and background conversational cues that signal fear or threat amplification. | `preset:xenophobia`                                |
| Ableism                                   | Cultural marginalization of people based on ability, disability, or neurodivergence. We detect ableism using dismissive or mocking tone, exaggerated vocal imitation, sarcasm markers, dehumanizing prosody, emotional contempt, emphasis patterns around ability-related references, and interaction dynamics that signal minimization or invalidation beyond the literal words spoken.                  | `preset:ableism`                                   |
| Social Inclusion                          | Active efforts to include diverse participants. We detect this using affirming tone, inclusive address patterns, balanced turn-taking, and warm emotional delivery.                                                                                                                                                                                                                                       | `preset:social_inclusion`                          |
| Social Proof Threat                       | Caller threatens to leave a negative review or post about their experience on social media.                                                                                                                                                                                                                                                                                                               | `preset:social_proof_threat`                       |
| Conflict De-escalation                    | One party attempts to reduce tension, soften language, or redirect toward resolution.                                                                                                                                                                                                                                                                                                                     | `preset:conflict_de_escalation`                    |
| Conflict Escalation                       | Conversation tone escalates toward argument, with rising emotional intensity from one or both parties.                                                                                                                                                                                                                                                                                                    | `preset:conflict_escalation`                       |
| Caller Under Duress                       | Vocal cues and language suggest the caller is being coerced or monitored by a third party.                                                                                                                                                                                                                                                                                                                | `preset:caller_under_duress`                       |
| Discriminatory Complaint                  | Caller appears to be lodging a complaint motivated by bias rather than a legitimate service issue.                                                                                                                                                                                                                                                                                                        | `preset:discriminatory_complaint`                  |


# Trust and Safety
Source: https://docs.modulate.ai/velma/detection-packages/trust-and-safety

Flag harassment, hate, and safety violations in live social and voice-chat audio.

Detect harmful content — harassment, hate speech, and identity-, welfare-, and child-safety violations — in live social audio and voice chat, in real time.

Live social audio is one venue. The same detections extend across community moderation, user-generated content, livestreams, and any space where people interact at scale. Use this package as a starting point — add or swap in any [preset from the catalog](/velma/behaviors/using-behaviors#listing-available-presets), or [define your own behaviors](/velma/behaviors/custom-behaviors) to fit your use case.

**This package includes:** 11 conversation types · 5 participant roles · 33 behaviors.

## Use this package

The config below is ready to use as-is — download or copy it and pass it as the `config` payload in a Velma Triage request. Its `behaviors` are [preset references](/velma/behaviors/using-behaviors): the API expands each `preset:<identifier>` into its full definition at request time, so you don't need the criteria inline to run the package.

<a href="#">Download trust-and-safety.json</a>

<Accordion title="View / copy full config">
  <div>
    ```json theme={null}
    {
      "conversation_types": [
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111004",
          "name": "General Media Narration",
          "short_description": "Any media style content with a single speaker that talks exclusively in the third person",
          "detailed_description": "media"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111005",
          "name": "Multiple Speakers Livestreamed Media",
          "short_description": "Any improvised media content that features several speakers and explicitly exists for entertainment or social purposes",
          "detailed_description": "mediasocial"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111006",
          "name": "Media Interview or Talk Show",
          "short_description": "Conversations in media formatted as one on one interviews, host with one or more guests, or podcasts formatted as question and answer shows.",
          "detailed_description": "mediasocial"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111009",
          "name": "General Media Dialogue",
          "short_description": "Any conversation with multiple speaking participants that's scripted featuring other things like music or sound effects",
          "detailed_description": "mediasocial"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111010",
          "name": "Music",
          "short_description": "Media content primarily featuring music",
          "detailed_description": "mediasocial"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111011",
          "name": "Audiobook",
          "short_description": "A book narrated into audio",
          "detailed_description": "mediasocial"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111012",
          "name": "Social Media Content",
          "short_description": "Any content with a narrator that's formatted in a way that's targeted for social media.",
          "detailed_description": "mediasocial"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111013",
          "name": "Single Speaker Livestreamed Media",
          "short_description": "Any media content that features only one speaker that's improvised and explicitly exists for entertainment or social purposes",
          "detailed_description": "mediasocial"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111014",
          "name": "Online Game Chat",
          "short_description": "Any social chat where the participants are brought together to play a game",
          "detailed_description": "gamingsocial"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111015",
          "name": "Online Chat Room",
          "short_description": "Any conversation with several participants who appear to be strangers, in a large group, or have the ability to talk over each other.",
          "detailed_description": "casualsocial"
        },
        {
          "conversation_type_uuid": "11111111-1111-4111-8111-111111111016",
          "name": "Social Phone or Video Call",
          "short_description": "Any call with two to five participants that's explicitly for the purpose of socializing",
          "detailed_description": "casualsocial"
        }
      ],
      "participant_roles": [
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222003",
          "name": "Social Participant",
          "short_description": "Participant in social conversation",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222007",
          "name": "Media Participant",
          "short_description": "A speaker in any media",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222008",
          "name": "Narrator",
          "short_description": "A media participant who speaks in the third person exclusively",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222009",
          "name": "Social Participant",
          "short_description": "A partipant in a social conversation",
          "detailed_description": ""
        },
        {
          "participant_role_uuid": "22222222-2222-4222-8222-222222222016",
          "name": "Interviewer",
          "short_description": "A person giving an interview",
          "detailed_description": ""
        }
      ],
      "behaviors": [
        "preset:future_planning",
        "preset:social_etiquette",
        "preset:sexually_graphic_material",
        "preset:storytelling",
        "preset:social_boundary_setting",
        "preset:material_potentially_unsuitable_for_children",
        "preset:violent_graphic_material",
        "preset:social_connection_building",
        "preset:personal_vulnerability",
        "preset:encouragement",
        "preset:teaching_mentorship",
        "preset:narration",
        "preset:monologuing",
        "preset:poetry",
        "preset:rapport_building",
        "preset:inclusive_practices",
        "preset:unclear_speech",
        "preset:unaddressed_question",
        "preset:hateful_or_violent_ideology_propagation",
        "preset:child_safety_violation",
        "preset:sexual_harassment",
        "preset:harassment",
        "preset:suicidal_and_self_injurious_ideation",
        "preset:hate",
        "preset:self_harm_and_self_injury_glorification",
        "preset:misogyny",
        "preset:racism",
        "preset:homophobia",
        "preset:transphobia",
        "preset:sizeism",
        "preset:xenophobia",
        "preset:ableism",
        "preset:social_inclusion"
      ]
    }
    ```
  </div>
</Accordion>

### Expand the full criteria

To produce a self-contained config with every behavior's full criteria inlined — for review, customization, or pinning a snapshot — fetch the live [preset catalog](/velma/behaviors/using-behaviors#listing-available-presets) and merge it into the downloaded config. The catalog is the source of truth for detection criteria.

<CodeGroup>
  ```bash curl + jq theme={null}
  curl -s https://platform.modulate.ai/api/velma-2-batch/list-presets \
    -H "X-API-Key: $MODULATE_API_KEY" \
  | jq --slurpfile cfg trust-and-safety.json '
      [ $cfg[0].behaviors[] | ltrimstr("preset:") ] as $ids
      | { conversation_types: $cfg[0].conversation_types,
          participant_roles:  $cfg[0].participant_roles,
          behaviors: [ .presets[] | select(.identifier as $i | $ids | index($i)) ] }
    ' > trust-and-safety.full.json
  ```

  ```python Python theme={null}
  import os, json, requests

  cfg = json.load(open("trust-and-safety.json"))
  ids = {b.removeprefix("preset:") for b in cfg["behaviors"]}

  catalog = requests.get(
      "https://platform.modulate.ai/api/velma-2-batch/list-presets",
      headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
  ).json()["presets"]

  cfg["behaviors"] = [p for p in catalog if p["identifier"] in ids]
  json.dump(cfg, open("trust-and-safety.full.json", "w"), indent=2)
  ```
</CodeGroup>

`trust-and-safety.full.json` keeps the same `conversation_types` and `participant_roles` and replaces each preset reference with its full behavior definition — drop it into the `config` payload exactly like the preset version.

## Conversation types

The interaction contexts this package expects to see.

| Name                                 | What it is                                                                                                                                   |
| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- |
| General Media Narration              | Any media style content with a single speaker that talks exclusively in the third person                                                     |
| Multiple Speakers Livestreamed Media | Any improvised media content that features several speakers and explicitly exists for entertainment or social purposes                       |
| Media Interview or Talk Show         | Conversations in media formatted as one on one interviews, host with one or more guests, or podcasts formatted as question and answer shows. |
| General Media Dialogue               | Any conversation with multiple speaking participants that's scripted featuring other things like music or sound effects                      |
| Music                                | Media content primarily featuring music                                                                                                      |
| Audiobook                            | A book narrated into audio                                                                                                                   |
| Social Media Content                 | Any content with a narrator that's formatted in a way that's targeted for social media.                                                      |
| Single Speaker Livestreamed Media    | Any media content that features only one speaker that's improvised and explicitly exists for entertainment or social purposes                |
| Online Game Chat                     | Any social chat where the participants are brought together to play a game                                                                   |
| Online Chat Room                     | Any conversation with several participants who appear to be strangers, in a large group, or have the ability to talk over each other.        |
| Social Phone or Video Call           | Any call with two to five participants that's explicitly for the purpose of socializing                                                      |

## Participant roles

The speaker roles the package distinguishes.

| Name               | What it is                                                     |
| ------------------ | -------------------------------------------------------------- |
| Social Participant | Participant in social conversation                             |
| Media Participant  | A speaker in any media                                         |
| Narrator           | A media participant who speaks in the third person exclusively |
| Social Participant | A partipant in a social conversation                           |
| Interviewer        | A person giving an interview                                   |

## Behaviors

The 33 signals this package detects. Each maps to a reusable `preset:<identifier>` you can drop into the `behaviors` array of any [BatchConfig](/velma/behaviors/using-behaviors) — the config above already references them.

<Note>
  Full detection criteria are not duplicated here. The live preset catalog is the source of truth — retrieve the exact criteria for any behavior by name from the [`list-presets` endpoint](/velma/behaviors/using-behaviors#listing-available-presets).
</Note>

| Behavior                                     | What it detects                                                                                                                                                                                                                                                                                                                                                                                           | Preset                                                |
| -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
| Future Planning                              | Discussion of goals or intended future actions. We detect this using forward-looking temporal language paired with planning cadence, collaborative tone, anticipatory prosody, and measured pacing.                                                                                                                                                                                                       | `preset:future_planning`                              |
| Social Etiquette                             | Observance of politeness and manners. We detect this through courteous tone, appropriate pacing, respectful address forms, and smooth turn transitions.                                                                                                                                                                                                                                                   | `preset:social_etiquette`                             |
| Sexually Graphic Material                    | Explicit descriptions of sexual activity or content. We detect this using suggestive prosody, discomfort or arousal markers, and contextual acoustic cues that go beyond neutral or educational discussion.                                                                                                                                                                                               | `preset:sexually_graphic_material`                    |
| Storytelling                                 | Third-person recounting of events in narrative form. We detect this using narrative arc pacing, character-based vocal modulation, temporal structuring, and expressive prosody.                                                                                                                                                                                                                           | `preset:storytelling`                                 |
| Social Boundary Setting                      | Establishing limits for appropriate interaction. We detect this using firm but calm tone, slowed pacing, clear prosodic boundaries, and reduced emotional escalation.                                                                                                                                                                                                                                     | `preset:social_boundary_setting`                      |
| Material Potentially Unsuitable for Children | Use of age-inappropriate language or themes. We detect this using profanity intensity, emotional arousal, laughter timing, vocal emphasis on taboo terms, and contextual cues to recognize innuendo.                                                                                                                                                                                                      | `preset:material_potentially_unsuitable_for_children` |
| Violent Graphic Material                     | Graphic descriptions or depictions of physical violence. We detect this through vivid descriptive cadence, stress patterns, breath control changes, and emotional intensity that accompany graphic recounting beyond neutral narration.                                                                                                                                                                   | `preset:violent_graphic_material`                     |
| Social Connection Building                   | Signals of interpersonal bonding and relational warmth. We detect this through mutual laughter, mirroring of speech rhythms, relaxed pacing, warm vocal timbre, and decreasing formality over time.                                                                                                                                                                                                       | `preset:social_connection_building`                   |
| Personal Vulnerability                       | Expressions of inner feelings or personal struggles. We detect this using softened volume, hesitations, longer pauses, emotional tremor, and shifts toward introspective tone and slower speech rate.                                                                                                                                                                                                     | `preset:personal_vulnerability`                       |
| Encouragement                                | Supportive reinforcement of another's actions or ideas. We detect this through positive prosody, upward intonation, affirming rhythm, increased energy, and emotional warmth in delivery.                                                                                                                                                                                                                 | `preset:encouragement`                                |
| Teaching/Mentorship                          | Instructional guidance aimed at skill or knowledge transfer. We detect this using structured pacing, explanatory intonation, deliberate pauses, corrective tone shifts, and reduced emotional volatility.                                                                                                                                                                                                 | `preset:teaching_mentorship`                          |
| Narration                                    | Third-person descriptive speech detached from present interaction. We detect this through consistent third-person framing, steady pacing, neutral affect, and minimal turn-taking responsiveness.                                                                                                                                                                                                         | `preset:narration`                                    |
| Monologuing                                  | Extended uninterrupted expressive speech by one speaker. We detect this using long speaking turns, theatrical intonation, emotional variability, minimal pauses for response, and self-directed delivery.                                                                                                                                                                                                 | `preset:monologuing`                                  |
| Poetry                                       | Speech employing poetic structure or stylistic devices. We detect this through rhythmic meter, deliberate pauses, rhyme or alliteration cues, melodic intonation, and performative cadence.                                                                                                                                                                                                               | `preset:poetry`                                       |
| Rapport Building                             | Positive alignment forming a professional relationship. We detect this through reciprocal tone matching, affirming backchannels, relaxed pacing, and increasing conversational ease.                                                                                                                                                                                                                      | `preset:rapport_building`                             |
| Inclusive Practices                          | Respectful language promoting inclusion and equity. We detect this through careful word choice reinforced by respectful tone, measured pacing, non-dismissive intonation, and calm emotional delivery.                                                                                                                                                                                                    | `preset:inclusive_practices`                          |
| Unclear Speech                               | Speech difficult to interpret or understand. We detect this using slurred articulation, inconsistent pacing, counterparty confusion, overlapping speech, and frequent self-corrections.                                                                                                                                                                                                                   | `preset:unclear_speech`                               |
| Unaddressed Question                         | Failure to adequately respond to a posed question. We detect this through avoidance pauses, topic-shifting intonation, increased filler usage, and prosodic signals of deflection.                                                                                                                                                                                                                        | `preset:unaddressed_question`                         |
| Hateful or Violent Ideology Propagation      | Promotion of hate-based or violent belief systems. We detect this through ideological slogans, charged emotional delivery, escalating intensity, and dehumanizing tonal patterns.                                                                                                                                                                                                                         | `preset:hateful_or_violent_ideology_propagation`      |
| Child Safety Violation                       | Sexual exploitation or endangerment involving minors. We detect this using covert language cues, abnormal hesitation, grooming-style warmth, secrecy-driven pacing, and contextual red flags.                                                                                                                                                                                                             | `preset:child_safety_violation`                       |
| Sexual Harassment                            | Unwanted sexualized speech or advances. We detect this through suggestive intonation, boundary-testing pauses, inappropriate familiarity, and discomfort responses from others.                                                                                                                                                                                                                           | `preset:sexual_harassment`                            |
| Harassment                                   | Persistent unwanted targeted behavior. We detect this through repeated hostile tone, fixation on a target, escalating intensity, and lack of de-escalation cues.                                                                                                                                                                                                                                          | `preset:harassment`                                   |
| Suicidal and Self Injurious Ideation         | Signals of thoughts about self-harm or suicide. We detect this through flattened affect, slowed speech, long silences, emotional heaviness, and indirect despair cues.                                                                                                                                                                                                                                    | `preset:suicidal_and_self_injurious_ideation`         |
| Hate                                         | Identity-based hateful or discriminatory speech. We detect this using demeaning tone, dehumanizing language delivery, emotional hostility, and ideological reinforcement patterns.                                                                                                                                                                                                                        | `preset:hate`                                         |
| Self-Harm and Self-Injury Glorification      | Portrayal of self-harm as positive or necessary. We detect this through minimizing tone, abnormal calmness, valorizing prosody, and repeated normalization cues.                                                                                                                                                                                                                                          | `preset:self_harm_and_self_injury_glorification`      |
| Misogyny                                     | Evidence of cultural attitudes that propagate systems of marginalizing people based on their gender identity or gender presentation                                                                                                                                                                                                                                                                       | `preset:misogyny`                                     |
| Racism                                       | Evidence of cultural attitudes that propagate systems of marginalizing people based on their racial or national identity if that identity isn't white                                                                                                                                                                                                                                                     | `preset:racism`                                       |
| Homophobia                                   | Evidence of cultural attitudes that propagate systems of marginalizing people based on their sexuality                                                                                                                                                                                                                                                                                                    | `preset:homophobia`                                   |
| Transphobia                                  | Evidence of cultural attitudes that propagate systems of marginalizing people based on their trans identity                                                                                                                                                                                                                                                                                               | `preset:transphobia`                                  |
| Sizeism                                      | Marginalization or stigmatization based on body size, shape, or weight-related conditions. We detect sizism using ridicule or disgust conveyed through vocal affect, laughter timing, exaggerated emphasis on physical descriptors, shaming prosody, sarcastic cadence, and emotional distancing signals that indicate judgment even when explicit insults are absent.                                    | `preset:sizeism`                                      |
| Xenophobia                                   | Marginalization or hostility toward people based on nationality, culture, or religion. We detect xenophobia using hostile or exclusionary tone, accent-mimicry or accent-mocking delivery, emotionally charged pacing, sharp prosodic emphasis around group references, us-versus-them framing expressed through intonation, and background conversational cues that signal fear or threat amplification. | `preset:xenophobia`                                   |
| Ableism                                      | Cultural marginalization of people based on ability, disability, or neurodivergence. We detect ableism using dismissive or mocking tone, exaggerated vocal imitation, sarcasm markers, dehumanizing prosody, emotional contempt, emphasis patterns around ability-related references, and interaction dynamics that signal minimization or invalidation beyond the literal words spoken.                  | `preset:ableism`                                      |
| Social Inclusion                             | Active efforts to include diverse participants. We detect this using affirming tone, inclusive address patterns, balanced turn-taking, and warm emotional delivery.                                                                                                                                                                                                                                       | `preset:social_inclusion`                             |


# What is Velma?
Source: https://docs.modulate.ai/velma/overview

Velma is an audio-native voice intelligence model over REST or WebSocket.

Velma is Modulate's Conversation Understanding model. It analyzes audio signals (tone, emotion, intent, and dozens of other acoustic cues) and words to capture the true intent and meaning behind a conversation.

Use Velma to surface risks or specific "behaviors" in voice conversations, such as fraud, customer churn, and compliance violations. It ships with 150+ pre-configured behaviors covering common conversation topics and business use cases. Additionally, you can define your own by describing what to detect in a natural-language prompt. Every response also includes a conversation summary, emotion and accent detection, speaker identification, conversation topics, deepfake detection, sentiment, a diarized transcript, and more.

The power of Velma is that you describe what you want to detect, and Velma uses audio signals to detect them.

## Build with Velma

Start from a **detection package** — a ready-made configuration that groups the conversation types, roles, and behaviors for a common use case. Explore what each detects, then download and adapt it.

<CardGroup>
  <Card title="Fraud Detection and Prevention" icon="shield-halved" href="/velma/detection-packages/fraud-detection-and-prevention">
    Catch social engineering, account-takeover attempts, and impersonation in customer conversations.
  </Card>

  <Card title="Agentic AI Guardrails" icon="robot" href="/velma/detection-packages/agentic-ai-guardrails">
    Keep AI voice agents on-policy: catch off-script, unsafe, or out-of-scope behavior.
  </Card>

  <Card title="Trust and Safety" icon="shield-check" href="/velma/detection-packages/trust-and-safety">
    Flag harassment, hate, and safety violations in live social and voice-chat audio.
  </Card>

  <Card title="Customer Retention" icon="heart" href="/velma/detection-packages/customer-retention">
    Surface churn signals, dissatisfaction, and save opportunities on every call.
  </Card>

  <Card title="Human Agent Welfare" icon="hand-holding-heart" href="/velma/detection-packages/human-agent-welfare">
    Protect agents by detecting abuse, distress, and burnout indicators.
  </Card>

  <Card title="Compliance and Risk Monitoring" icon="scale-balanced" href="/velma/detection-packages/compliance-and-risk-monitoring">
    Catch disclosure gaps, compliance breaches, and risk events across regulated calls.
  </Card>
</CardGroup>

## Batch or streaming?

|              | Velma (batch)                                       | Velma (streaming)                                     |
| ------------ | --------------------------------------------------- | ----------------------------------------------------- |
| **Use case** | Analyze a complete recording                        | Analyze a live or in-progress conversation            |
| **Protocol** | HTTP POST                                           | WebSocket                                             |
| **Response** | Single JSON response                                | Stream of typed events                                |
| **Best for** | Post-call QA, compliance review, offline processing | Live monitoring, real-time alerting, in-call coaching |

## What Velma produces

<CardGroup>
  <Card title="Behavior detection" icon="magnifying-glass">
    Per-behavior verdicts with confidence scores and the specific clips that triggered each detection.
  </Card>

  <Card title="Conversation type" icon="comments">
    Classification of the conversation against the types you define — customer support, sales, interview, and more.
  </Card>

  <Card title="Participant roles" icon="users">
    Per-speaker role assignments drawn from the roles you configure, resolved in real time as speakers are identified.
  </Card>

  <Card title="Topics" icon="tags">
    An aggregated list of the subjects discussed across the full conversation.
  </Card>

  <Card title="Topic sentiment" icon="chart-line">
    Per-speaker sentiment scores for each extracted topic, ranging from −1 (negative) to +1 (positive).
  </Card>

  <Card title="Summary" icon="align-left">
    A free-form narrative summary of the conversation generated at end of stream.
  </Card>
</CardGroup>

## How it works

<Tabs>
  <Tab title="Batch">
    Velma (batch) is an HTTP POST endpoint at `/api/velma-2-batch`. Submit a complete audio file as `multipart/form-data` — the `upload_file` field carries the audio (up to 100 MB), and the `config` field carries a JSON-encoded `BatchConfig` or the literal string `"default"` to use Velma's built-in defaults.

    Velma processes the full recording and returns a single JSON response containing all clips, role assignments, behavior detections, topics, topic sentiment scores, and a summary.

    ```bash theme={null}
    curl -X POST https://platform.modulate.ai/api/velma-2-batch \
      -H "X-API-Key: $MODULATE_API_KEY" \
      -F "upload_file=@recording.mp3" \
      -F "config=default"
    ```

    Use batch when you have a finished recording and want a complete analysis in one call — post-call QA, compliance review, or processing an uploaded file.
  </Tab>

  <Tab title="Streaming">
    Velma (streaming) is a WebSocket endpoint. You connect, send a configuration frame, then stream audio incrementally. Velma emits structured JSON events as it processes — clips, role assignments, behavior detections, topics, sentiment, and a summary — and closes with a `done` event when audio ends.

    The connection follows a strict sequence:

    <Steps>
      <Step title="Connect">
        Open a WebSocket connection to `wss://platform.modulate.ai/api/velma-2-streaming` with your `api_key` as a query parameter.
      </Step>

      <Step title="Send configuration">
        Send a single JSON text frame containing your configuration — the conversation types, participant roles, and behaviors you want Velma to work with. This frame must arrive before any audio.
      </Step>

      <Step title="Stream audio">
        Send your audio as binary WebSocket frames in any chunk size. Self-describing formats (MP3, WAV, OGG, FLAC, WebM, AAC, AIFF) are auto-detected. Raw PCM formats require `audio_format`, `sample_rate`, and `num_channels` query parameters.
      </Step>

      <Step title="Signal end of stream">
        Send an empty text frame (`""`) to tell Velma the audio is complete.
      </Step>

      <Step title="Receive events">
        Velma emits typed JSON events throughout and closes the connection with a `done` event.
      </Step>
    </Steps>
  </Tab>
</Tabs>

## How Velma differs from the other endpoints

|                                  | Velma                         | Transcription | Deepfake Detection |
| -------------------------------- | ----------------------------- | ------------- | ------------------ |
| **Primary output**               | Conversation intelligence     | Transcript    | Deepfake scores    |
| **Behavior detection**           | ✓ (configurable)              | —             | —                  |
| **Conversation classification**  | ✓                             | —             | —                  |
| **Topic and sentiment analysis** | ✓                             | —             | —                  |
| **Summarization**                | ✓                             | —             | —                  |
| **Transcription**                | ✓ (via Transcription options) | ✓             | —                  |
| **Batch and streaming**          | ✓                             | ✓             | ✓                  |

<Note>
  Velma includes transcription as part of its output via Transcription options in your configuration. If transcription alone is what you need, the [Transcription endpoints](/get-started/stt) are the right choice.
</Note>

## Next steps

<CardGroup>
  <Card title="Capabilities" icon="layer-group" href="/velma/capabilities">
    Explore all of Velma's analysis outputs and configuration options in detail.
  </Card>

  <Card title="Behaviors" icon="wand-magic-sparkles" href="/velma/behaviors/what-are-behaviors">
    Learn how to define the signals you want Velma to detect.
  </Card>
</CardGroup>