> ## Documentation Index
> Fetch the complete documentation index at: https://docs.modulate.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# AI Music Detection Streaming
> Real-time AI music detection over WebSocket. Per-window vocal-AI and instrumental-AI results are emitted progressively as audio arrives, followed by a final clip-level summary.
Real-time AI music detection over WebSocket. The client streams audio and receives per-window vocal-AI and instrumental-AI results as they become available, followed by a final clip-level summary on completion.
## Endpoint
```text theme={null}
wss://platform.modulate.ai/api/velma-2-ai-music-detection-streaming
```
## Authentication
Pass your API key as a query parameter on the connection URL:
```text theme={null}
wss://.../velma-2-ai-music-detection-streaming?api_key=YOUR_API_KEY&audio_format=mp3
```
## Connection parameters
| Parameter | Required | Description |
| - | - | - |
| `api_key` | Yes | Your API key |
| `audio_format` | Yes | Audio format - see supported formats below |
| `sample_rate` | Raw PCM only | Sample rate in Hz |
| `num_channels` | Raw PCM only | Number of channels (1-8) |
## Supported audio formats
**Container formats** - `sample_rate` and `num_channels` are ignored if supplied (the headers already carry this metadata):
`mp3`, `wav`, `flac`, `m4a`, `mp4`, `ogg`, `opus`, `webm`, `aac`, `aiff`, `wma`, `amr`, and `au`, plus 66 others
`3g2`, `3ga`, `3gp`, `3gpp`, `8svx`, `aa3`, `aac`, `ac3`, `act`, `adts`, `aif`, `aifc`, `aiff`,
`amb`, `amr`, `asf`, `at3`, `au`, `avr`, `awb`, `bwf`, `c2`, `caf`, `dss`, `dts`, `dtshd`,
`eac3`, `ec3`, `f4a`, `f4b`, `flac`, `gsm`, `iff`, `m2a`, `m2ts`, `m4a`, `m4b`, `m4r`, `m4v`,
`mka`, `mkv`, `mlp`, `mp+`, `mp1`, `mp2`, `mp3`, `mp4`, `mpa`, `mpc`, `mpga`, `mpp`, `mts`,
`oga`, `ogg`, `ogx`, `oma`, `omg`, `opus`, `paf`, `pvf`, `qcp`, `ra`, `rf64`, `rm`, `rmvb`,
`snd`, `svx`, `thd`, `ts`, `tta`, `voc`, `vqf`, `w64`, `wav`, `wave`, `weba`, `webm`, `wma`,
`wmv`
The MP4-family values (`mp4`, `m4a`, `m4b`, `m4r`, `m4v`, `3gp`, `3gpp`, `3ga`, `3g2`, `f4a`, `f4b`) must be sent in a streamable layout; otherwise the connection ends with an audio-processing error.
**Raw PCM formats** - `sample_rate` and `num_channels` are required:
`s8`, `s16le`, `s16be`, `s24le`, `s24be`, `s32le`, `s32be`, `u8`, `u16le`, `u16be`, `u24le`, `u24be`, `u32le`, `u32be`, `f32le`, `f32be`, `f64le`, `f64be`, `mulaw`, `alaw`, `g722`, `vox`
`g722` and `vox` are mono-only: `num_channels` must be `1`.
**Valid sample rates:** 8000, 11025, 16000, 22050, 32000, 44100, 48000, 96000
## Protocol
### Client -> server
| Message | Description |
| - | - |
| Binary frame | Chunk of audio bytes in the declared format (any size) |
| Empty text frame `""` | Signals end of stream |
### Server -> client
| Message | Description |
| - | - |
| `{"type": "window", "window": {...}}` | Per-window result - emitted for each completed 4-second window, in order |
| `{"type": "done", ...}` | Stream complete - clip-level verdict plus instrumental AI detection |
| `{"type": "error", "error": "..."}` | An error occurred - connection will close |
Vocal AI and instrumental AI are both scored independently on each 4-second window as audio arrives and reported in `window` messages - a window can carry both scores, one, or neither, depending on how much of each content type it contains. The window-level instrumental score is computed from a single short window, so it can be less accurate than the clip-level score in the final `done` message, which is computed from the full accumulated audio.
### Window object
| Field | Type | Description |
| - | - | - |
| `start_time_ms` | integer | Window start time in milliseconds |
| `end_time_ms` | integer | Window end time in milliseconds |
| `vocal_percentage` | float | Percentage of the window containing vocal content (0-100) |
| `vocal_ai_probability` | float or `null` | Probability the window contains AI-generated vocals (0-1). `null` if the window doesn't have enough vocal content to score, was too short to analyse, or no content was found to classify |
| `vocal_ai_confidence` | float or `null` | Confidence in `vocal_ai_probability` (0-1); `null` under the same conditions as that field |
| `instrumental_percentage` | float | Percentage of the window containing instrumental music content (0-100) |
| `instrumental_ai_probability` | float or `null` | Probability the window contains AI-generated instrumental content (0-1), from a single-window analysis (lower accuracy than the `done` message's clip-level score). `null` if the window is mostly silence, instrumental-AI detection is unavailable, or the window's audio could not be analysed |
| `instrumental_ai_confidence` | float or `null` | Confidence in `instrumental_ai_probability` (0-1); `null` under the same conditions as that field |
| `silence_percentage` | float | Percentage of the window containing neither vocal nor instrumental content (0-100) |
`vocal_ai_probability`/`vocal_ai_confidence` and `instrumental_ai_probability`/`instrumental_ai_confidence` are gated and scored independently, so a window can carry both pairs, one, or neither.
### Done object
| Field | Type | Description |
| - | - | - |
| `duration_ms` | integer | Total duration of the streamed audio in milliseconds |
| `window_count` | integer | Total number of windows analysed during the session |
| `primary_verdict` | string | Clip-level classification: `"ai-vocal-music"`, `"ai-instrumental"`, or `"not-ai-music"` |
| `vocal_percentage` | float | Average percentage of audio with vocal content, across all windows (0-100) |
| `vocal_ai_percentage` | float | Percentage of the full clip duration classified as AI-generated vocals (0-100), recomputed at end-of-stream. May differ from what the live per-window messages suggested |
| `vocal_ai_confidence` | float | Average confidence across frames classified as AI-generated vocals, from the end-of-stream recompute (0-1); `0` if none were found |
| `instrumental_percentage` | float | Average percentage of audio with instrumental content, across all windows (0-100) |
| `instrumental_ai_percentage` | float | AI detection score for the clip's instrumental content (0-100), computed once at end-of-stream from the full accumulated audio - more reliable than the live per-window `instrumental_ai_probability` values |
| `instrumental_ai_confidence` | float | Confidence in the instrumental AI assessment for the full clip (0-1) |
| `silence_percentage` | float | Average percentage of audio with neither vocal nor instrumental content (0-100) |
## WebSocket close codes
| Code | Meaning |
| - | - |
| `1000` | Normal closure after the `done` message, or after the connection completes |
| `1003` | Invalid or missing query parameters (unknown format, bad sample rate, missing `audio_format`) |
| `1011` | Internal server error, or processing ended before completion |
| `4001` | The `api_key` query parameter is missing or invalid |
| `4002` | Audio could not be decoded, or does not match the declared raw `audio_format` |
| `4003` | The request is not permitted |
| `4004` | The API key does not have access to this model |
| `4029` | The request could not be completed due to insufficient credits |
| `4030` | Concurrent request limit reached |
| `4031` | Monthly usage limit reached |
An `error` message is sent before the connection closes for every case above except `1000`.
## Rate limits
* Concurrent connection limits apply per model
## AsyncAPI
````yaml api/velma_2_ai_music_detection_streaming.yaml streaming-detection
id: streaming-detection
title: Streaming-detection
description: |
## WebSocket Close Codes
- `1000`: Normal closure after the `done` message, or after the
connection completes.
- `1003`: Invalid or missing query parameters (unknown `audio_format`,
invalid `sample_rate` or `num_channels`, a raw format missing
`sample_rate`/`num_channels`, `sample_rate`/`num_channels` supplied
without `audio_format`, or `audio_format` omitted). An `error` message
is sent before the close.
- `4002`: The audio could not be decoded, or does not match the
declared raw `audio_format`. An `error` message is sent before the
close.
- `1011`: Internal server error, or processing ended before completion.
An `error` message is sent before the close; when processing ends
without a `done` message, the `error` message is
"Unable to complete the request. Please try again."
- `4001`: The `api_key` query parameter is missing or invalid. An
`error` message is sent before the close.
- `4003`: The request is not permitted. An `error` message is sent
before the close.
- `4004`: The API key does not have access to this model. An `error`
message is sent before the close.
- `4029`: The request could not be completed due to insufficient credits.
An `error` message is sent before the close.
- `4030`: Concurrent request limit reached. An `error` message is sent
before the close.
- `4031`: Monthly usage limit reached. An `error` message is sent before
the close.
servers:
- id: production
protocol: wss
host: platform.modulate.ai
bindings: []
variables: []
address: /api/velma-2-ai-music-detection-streaming
parameters: []
bindings:
- protocol: ws
version: latest
value:
query:
type: object
required:
- api_key
- audio_format
properties:
api_key:
type: string
description: API key used for authentication and usage tracking.
audio_format:
type: string
description: >
Audio encoding format. Required for all connections. Raw formats
(headerless samples) also require `sample_rate` and
`num_channels`; container formats carry that metadata in the
stream, so those parameters are ignored if supplied. The
MP4-family
values (`mp4`, `m4a`, `m4b`, `m4r`, `m4v`, `3gp`, `3gpp`,
`3ga`, `3g2`, `f4a`, `f4b`) must be sent in a streamable
layout; otherwise the connection ends with an
audio-processing error.
The raw `g722` and `vox` formats are mono-only (`num_channels`
must be `1`).
enum:
- s8
- s16le
- s16be
- s24le
- s24be
- s32le
- s32be
- u8
- u16le
- u16be
- u24le
- u24be
- u32le
- u32be
- f32le
- f32be
- f64le
- f64be
- mulaw
- alaw
- g722
- vox
- 3g2
- 3ga
- 3gp
- 3gpp
- 8svx
- aa3
- aac
- ac3
- act
- adts
- aif
- aifc
- aiff
- amb
- amr
- asf
- at3
- au
- avr
- awb
- bwf
- c2
- caf
- dss
- dts
- dtshd
- eac3
- ec3
- f4a
- f4b
- flac
- gsm
- iff
- m2a
- m2ts
- m4a
- m4b
- m4r
- m4v
- mka
- mkv
- mlp
- mp+
- mp1
- mp2
- mp3
- mp4
- mpa
- mpc
- mpga
- mpp
- mts
- oga
- ogg
- ogx
- oma
- omg
- opus
- paf
- pvf
- qcp
- ra
- rf64
- rm
- rmvb
- snd
- svx
- thd
- ts
- tta
- voc
- vqf
- w64
- wav
- wave
- weba
- webm
- wma
- wmv
examples: &ref_0
- s16le
- f32le
- mulaw
- webm
sample_rate:
type: integer
description: |
Sample rate in Hz. **Required** when `audio_format` is a raw
format. Not required and ignored for container formats. Must
not be specified without `audio_format`.
enum:
- 8000
- 11025
- 16000
- 22050
- 32000
- 44100
- 48000
- 96000
examples: &ref_1
- 16000
- 48000
num_channels:
type: integer
description: |
Number of audio channels. **Required** when `audio_format` is
a raw format. Must be between 1 and 8. Not required and
ignored for container formats. Must not be specified without
`audio_format`.
minimum: 1
maximum: 8
examples: &ref_2
- 1
- 2
schemaProperties:
- name: query
type: object
required: false
properties:
- name: api_key
type: string
description: API key used for authentication and usage tracking.
required: true
- name: audio_format
type: string
description: >
Audio encoding format. Required for all connections. Raw formats
(headerless samples) also require `sample_rate` and
`num_channels`; container formats carry that metadata in the
stream, so those parameters are ignored if supplied. The
MP4-family
values (`mp4`, `m4a`, `m4b`, `m4r`, `m4v`, `3gp`, `3gpp`,
`3ga`, `3g2`, `f4a`, `f4b`) must be sent in a streamable
layout; otherwise the connection ends with an
audio-processing error.
The raw `g722` and `vox` formats are mono-only (`num_channels`
must be `1`).
enumValues:
- s8
- s16le
- s16be
- s24le
- s24be
- s32le
- s32be
- u8
- u16le
- u16be
- u24le
- u24be
- u32le
- u32be
- f32le
- f32be
- f64le
- f64be
- mulaw
- alaw
- g722
- vox
- 3g2
- 3ga
- 3gp
- 3gpp
- 8svx
- aa3
- aac
- ac3
- act
- adts
- aif
- aifc
- aiff
- amb
- amr
- asf
- at3
- au
- avr
- awb
- bwf
- c2
- caf
- dss
- dts
- dtshd
- eac3
- ec3
- f4a
- f4b
- flac
- gsm
- iff
- m2a
- m2ts
- m4a
- m4b
- m4r
- m4v
- mka
- mkv
- mlp
- mp+
- mp1
- mp2
- mp3
- mp4
- mpa
- mpc
- mpga
- mpp
- mts
- oga
- ogg
- ogx
- oma
- omg
- opus
- paf
- pvf
- qcp
- ra
- rf64
- rm
- rmvb
- snd
- svx
- thd
- ts
- tta
- voc
- vqf
- w64
- wav
- wave
- weba
- webm
- wma
- wmv
examples: *ref_0
required: true
- name: sample_rate
type: integer
description: |
Sample rate in Hz. **Required** when `audio_format` is a raw
format. Not required and ignored for container formats. Must
not be specified without `audio_format`.
enumValues:
- 8000
- 11025
- 16000
- 22050
- 32000
- 44100
- 48000
- 96000
examples: *ref_1
required: false
- name: num_channels
type: integer
description: |
Number of audio channels. **Required** when `audio_format` is
a raw format. Must be between 1 and 8. Not required and
ignored for container formats. Must not be specified without
`audio_format`.
examples: *ref_2
required: false
operations:
- &ref_31
id: sendAudio
title: Send audio
description: Stream audio data to the server.
type: send
messages:
- &ref_36
id: audioData
contentType: application/octet-stream
payload:
- type: string
format: binary
description: A binary frame containing audio samples.
x-parser-schema-id:
name: Audio Data
headers: []
jsonPayloadSchema:
type: string
format: binary
description: |
Audio data in the format declared by the `audio_format` query
parameter. For raw formats this is headerless samples; for
container formats this is the encoded byte stream. Frames may
be any size; the server buffers internally.
x-parser-schema-id:
title: Audio Data
description: A binary frame containing audio samples.
example: '{}'
bindings: []
extensions:
- id: x-parser-unique-object-id
value: audioData
bindings: []
extensions: &ref_3
- id: x-parser-unique-object-id
value: streaming-detection
- &ref_32
id: signalEnd
title: Signal end
description: Signal end of audio stream.
type: send
messages:
- &ref_37
id: endOfStream
contentType: text/plain
payload:
- type: string
const: ''
x-parser-schema-id:
name: End of Stream
description: >-
An empty text frame signalling that the client has finished
sending audio.
headers: []
jsonPayloadSchema:
type: string
const: ''
x-parser-schema-id:
title: End of Stream
description: >-
An empty text frame signalling that the client has finished sending
audio.
example: '{}'
bindings: []
extensions:
- id: x-parser-unique-object-id
value: endOfStream
bindings: []
extensions: *ref_3
- &ref_28
id: receiveWindowResult
title: Receive window result
description: Receive a per-window detection result.
type: receive
messages:
- &ref_33
id: windowResult
contentType: application/json
payload:
- name: Window Result
description: A single 4-second window detection result.
type: object
properties:
- name: type
type: string
description: window
required: true
- name: window
type: object
required: true
properties:
- name: start_time_ms
type: integer
description: Window start time in the audio stream, in milliseconds.
examples: &ref_4
- 0
required: true
- name: end_time_ms
type: integer
description: Window end time in the audio stream, in milliseconds.
examples: &ref_5
- 4000
required: true
- name: vocal_percentage
type: number
description: |
Percentage of this window that contains vocal content.
examples: &ref_6
- 87.5
required: true
- name: vocal_ai_probability
type: &ref_7
- number
- 'null'
description: >
Probability that this window contains AI-generated vocals.
`null` if the window does not have enough vocal content to
score, was too short to analyse, or no content was found
to classify - independent of
`instrumental_ai_probability`, so a window can carry both,
one, or neither.
examples: &ref_8
- 0.97
required: true
- name: vocal_ai_confidence
type: &ref_9
- number
- 'null'
description: >
Confidence in `vocal_ai_probability`. `null` under the
same conditions as that field.
examples: &ref_10
- 0.97
required: true
- name: instrumental_percentage
type: number
description: >
Percentage of this window that contains instrumental music
content.
examples: &ref_11
- 64.3
required: true
- name: instrumental_ai_probability
type: &ref_12
- number
- 'null'
description: >
Probability that this window contains AI-generated
instrumental content, from a single-window analysis (lower
accuracy than the `done` message's clip-level instrumental
score). `null` if the window is mostly silence,
instrumental-AI detection is unavailable, or the window's
audio could not be analysed - independent of
`vocal_ai_probability`, so a window can carry both, one,
or neither.
examples: &ref_13
- 0.42
required: true
- name: instrumental_ai_confidence
type: &ref_14
- number
- 'null'
description: >
Confidence in `instrumental_ai_probability`. `null` under
the same conditions as that field.
examples: &ref_15
- 0.16
required: true
- name: silence_percentage
type: number
description: >
Percentage of this window that contains neither vocal nor
instrumental content.
examples: &ref_16
- 3.5
required: true
headers: []
jsonPayloadSchema:
type: object
required:
- type
- window
properties:
type:
type: string
const: window
x-parser-schema-id:
window:
type: object
required:
- start_time_ms
- end_time_ms
- vocal_percentage
- vocal_ai_probability
- vocal_ai_confidence
- instrumental_percentage
- instrumental_ai_probability
- instrumental_ai_confidence
- silence_percentage
properties:
start_time_ms:
type: integer
minimum: 0
description: Window start time in the audio stream, in milliseconds.
examples: *ref_4
x-parser-schema-id:
end_time_ms:
type: integer
minimum: 0
description: Window end time in the audio stream, in milliseconds.
examples: *ref_5
x-parser-schema-id:
vocal_percentage:
type: number
format: double
minimum: 0
maximum: 100
description: |
Percentage of this window that contains vocal content.
examples: *ref_6
x-parser-schema-id:
vocal_ai_probability:
type: *ref_7
minimum: 0
maximum: 1
description: >
Probability that this window contains AI-generated vocals.
`null` if the window does not have enough vocal content to
score, was too short to analyse, or no content was found to
classify - independent of `instrumental_ai_probability`, so
a window can carry both, one, or neither.
examples: *ref_8
x-parser-schema-id:
vocal_ai_confidence:
type: *ref_9
minimum: 0
maximum: 1
description: >
Confidence in `vocal_ai_probability`. `null` under the same
conditions as that field.
examples: *ref_10
x-parser-schema-id:
instrumental_percentage:
type: number
format: double
minimum: 0
maximum: 100
description: >
Percentage of this window that contains instrumental music
content.
examples: *ref_11
x-parser-schema-id:
instrumental_ai_probability:
type: *ref_12
minimum: 0
maximum: 1
description: >
Probability that this window contains AI-generated
instrumental content, from a single-window analysis (lower
accuracy than the `done` message's clip-level instrumental
score). `null` if the window is mostly silence,
instrumental-AI detection is unavailable, or the window's
audio could not be analysed - independent of
`vocal_ai_probability`, so a window can carry both, one, or
neither.
examples: *ref_13
x-parser-schema-id:
instrumental_ai_confidence:
type: *ref_14
minimum: 0
maximum: 1
description: >
Confidence in `instrumental_ai_probability`. `null` under
the same conditions as that field.
examples: *ref_15
x-parser-schema-id:
silence_percentage:
type: number
format: double
minimum: 0
maximum: 100
description: >
Percentage of this window that contains neither vocal nor
instrumental content.
examples: *ref_16
x-parser-schema-id:
x-parser-schema-id: WindowResult
x-parser-schema-id: WindowResultMessage
title: Window Result
description: A single 4-second window detection result.
example: |-
{
"type": "",
"window": {
"start_time_ms": 123,
"end_time_ms": 123,
"vocal_percentage": 123,
"instrumental_percentage": 123,
"silence_percentage": 123
}
}
bindings: []
extensions:
- id: x-parser-unique-object-id
value: windowResult
bindings: []
extensions: *ref_3
- &ref_29
id: receiveDone
title: Receive done
description: Receive the final clip-level summary when analysis is complete.
type: receive
messages:
- &ref_34
id: done
contentType: application/json
payload:
- name: Done
description: Analysis complete - final clip-level verdict has been computed.
type: object
properties:
- name: type
type: string
description: done
required: true
- name: duration_ms
type: integer
description: Total duration of the streamed audio in milliseconds.
examples: &ref_17
- 89280
required: true
- name: window_count
type: integer
description: Total number of windows analysed during the session.
examples: &ref_18
- 22
required: true
- name: primary_verdict
type: string
description: >
Clip-level classification:
- `ai-vocal-music` - AI-generated music with a detected
synthetic
voice.
- `ai-instrumental` - AI-generated instrumental music with no
detectable synthetic voice.
- `not-ai-music` - the clip does not appear to contain
AI-generated music.
enumValues:
- ai-vocal-music
- ai-instrumental
- not-ai-music
examples: &ref_19
- ai-vocal-music
required: true
- name: vocal_percentage
type: number
description: >
Average percentage of the audio that contains vocal content,
across all windows.
examples: &ref_20
- 87.5
required: true
- name: vocal_ai_percentage
type: number
description: >
Clip-level AI-vocal score, on a 0-100 scale: percentage of the
full clip duration classified as AI-generated vocals,
recomputed at end-of-stream. May differ from what the live
per-window messages suggested.
examples: &ref_21
- 56.5
required: true
- name: vocal_ai_confidence
type: number
description: >
Average confidence across separated-audio frames classified as
AI-generated vocals, from the end-of-stream recompute. `0` if
no such frame was found.
examples: &ref_22
- 0.96
required: true
- name: instrumental_percentage
type: number
description: >
Average percentage of the audio that contains instrumental
music content, across all windows.
examples: &ref_23
- 64.3
required: true
- name: instrumental_ai_percentage
type: number
description: >
Clip-level AI-instrumental score, on a 0-100 scale, computed
once at end-of-stream from the full accumulated audio - a
larger amount of context than any single live window gets, so
this is more reliable than the per-window
`instrumental_ai_probability` values seen during streaming.
examples: &ref_24
- 42
required: true
- name: instrumental_ai_confidence
type: number
description: >
Confidence in the instrumental AI assessment for the full
clip.
examples: &ref_25
- 0.95
required: true
- name: silence_percentage
type: number
description: >
Average percentage of the audio that contains neither vocal
nor instrumental content, across all windows.
examples: &ref_26
- 3.51
required: true
headers: []
jsonPayloadSchema:
type: object
required:
- type
- duration_ms
- window_count
- primary_verdict
- vocal_percentage
- vocal_ai_percentage
- vocal_ai_confidence
- instrumental_percentage
- instrumental_ai_percentage
- instrumental_ai_confidence
- silence_percentage
properties:
type:
type: string
const: done
x-parser-schema-id:
duration_ms:
type: integer
minimum: 0
description: Total duration of the streamed audio in milliseconds.
examples: *ref_17
x-parser-schema-id:
window_count:
type: integer
minimum: 0
description: Total number of windows analysed during the session.
examples: *ref_18
x-parser-schema-id:
primary_verdict:
type: string
enum:
- ai-vocal-music
- ai-instrumental
- not-ai-music
description: >
Clip-level classification:
- `ai-vocal-music` - AI-generated music with a detected
synthetic
voice.
- `ai-instrumental` - AI-generated instrumental music with no
detectable synthetic voice.
- `not-ai-music` - the clip does not appear to contain
AI-generated music.
examples: *ref_19
x-parser-schema-id:
vocal_percentage:
type: number
format: double
minimum: 0
maximum: 100
description: >
Average percentage of the audio that contains vocal content,
across all windows.
examples: *ref_20
x-parser-schema-id:
vocal_ai_percentage:
type: number
format: double
minimum: 0
maximum: 100
description: >
Clip-level AI-vocal score, on a 0-100 scale: percentage of the
full clip duration classified as AI-generated vocals, recomputed
at end-of-stream. May differ from what the live per-window
messages suggested.
examples: *ref_21
x-parser-schema-id:
vocal_ai_confidence:
type: number
format: double
minimum: 0
maximum: 1
description: >
Average confidence across separated-audio frames classified as
AI-generated vocals, from the end-of-stream recompute. `0` if no
such frame was found.
examples: *ref_22
x-parser-schema-id:
instrumental_percentage:
type: number
format: double
minimum: 0
maximum: 100
description: >
Average percentage of the audio that contains instrumental music
content, across all windows.
examples: *ref_23
x-parser-schema-id:
instrumental_ai_percentage:
type: number
format: double
minimum: 0
maximum: 100
description: >
Clip-level AI-instrumental score, on a 0-100 scale, computed
once at end-of-stream from the full accumulated audio - a larger
amount of context than any single live window gets, so this is
more reliable than the per-window `instrumental_ai_probability`
values seen during streaming.
examples: *ref_24
x-parser-schema-id:
instrumental_ai_confidence:
type: number
format: double
minimum: 0
maximum: 1
description: |
Confidence in the instrumental AI assessment for the full clip.
examples: *ref_25
x-parser-schema-id:
silence_percentage:
type: number
format: double
minimum: 0
maximum: 100
description: >
Average percentage of the audio that contains neither vocal nor
instrumental content, across all windows.
examples: *ref_26
x-parser-schema-id:
x-parser-schema-id: DoneMessage
title: Done
description: Analysis complete - final clip-level verdict has been computed.
example: |-
{
"type": "",
"duration_ms": 123,
"window_count": 123,
"primary_verdict": "",
"vocal_percentage": 123,
"vocal_ai_percentage": 123,
"vocal_ai_confidence": 123,
"instrumental_percentage": 123,
"instrumental_ai_percentage": 123,
"instrumental_ai_confidence": 123,
"silence_percentage": 123
}
bindings: []
extensions:
- id: x-parser-unique-object-id
value: done
bindings: []
extensions: *ref_3
- &ref_30
id: receiveError
title: Receive error
description: Receive an error notification.
type: receive
messages:
- &ref_35
id: error
contentType: application/json
payload:
- name: Error
description: An error occurred during connection or analysis.
type: object
properties:
- name: type
type: string
description: error
required: true
- name: error
type: string
description: Human-readable error description.
examples: &ref_27
- Internal server error
- Unable to complete the request. Please try again.
- This request is not permitted.
- Missing API key.
- Invalid API key.
- This API key does not have access to this model.
- Insufficient credits.
- >-
Concurrent request limit reached. Please retry after your
in-flight requests complete.
- Monthly usage limit reached.
- Audio data does not match the declared audio_format.
- >-
The audio could not be processed. It may be corrupted or in
an unsupported format.
- >-
Invalid audio_format='mid'. Valid values: ['3g2', '3ga',
'3gp', '3gpp', '8svx', 'aa3', 'aac', 'ac3', 'act', 'adts',
'aif', 'aifc', 'aiff', 'alaw', 'amb', 'amr', 'asf', 'at3',
'au', 'avr', 'awb', 'bwf', 'c2', 'caf', 'dss', 'dts',
'dtshd', 'eac3', 'ec3', 'f32be', 'f32le', 'f4a', 'f4b',
'f64be', 'f64le', 'flac', 'g722', 'gsm', 'iff', 'm2a',
'm2ts', 'm4a', 'm4b', 'm4r', 'm4v', 'mka', 'mkv', 'mlp',
'mp+', 'mp1', 'mp2', 'mp3', 'mp4', 'mpa', 'mpc', 'mpga',
'mpp', 'mts', 'mulaw', 'oga', 'ogg', 'ogx', 'oma', 'omg',
'opus', 'paf', 'pvf', 'qcp', 'ra', 'rf64', 'rm', 'rmvb',
's16be', 's16le', 's24be', 's24le', 's32be', 's32le', 's8',
'snd', 'svx', 'thd', 'ts', 'tta', 'u16be', 'u16le', 'u24be',
'u24le', 'u32be', 'u32le', 'u8', 'voc', 'vox', 'vqf', 'w64',
'wav', 'wave', 'weba', 'webm', 'wma', 'wmv']
- >-
Raw audio formats require sample_rate and num_channels query
parameters
- >-
sample_rate and num_channels require audio_format to be
specified
- >-
Invalid sample_rate=50000. Valid values: [8000, 11025,
16000, 22050, 32000, 44100, 48000, 96000]
- Invalid num_channels=12. Must be between 1 and 8
- >-
audio_format query parameter is required. Example:
audio_format=s16le&sample_rate=16000&num_channels=1
required: true
headers: []
jsonPayloadSchema:
type: object
required:
- type
- error
properties:
type:
type: string
const: error
x-parser-schema-id:
error:
type: string
description: Human-readable error description.
examples: *ref_27
x-parser-schema-id:
x-parser-schema-id: ErrorResponse
title: Error
description: An error occurred during connection or analysis.
example: |-
{
"type": "",
"error": ""
}
bindings: []
extensions:
- id: x-parser-unique-object-id
value: error
bindings: []
extensions: *ref_3
sendOperations:
- *ref_28
- *ref_29
- *ref_30
receiveOperations:
- *ref_31
- *ref_32
sendMessages:
- *ref_33
- *ref_34
- *ref_35
receiveMessages:
- *ref_36
- *ref_37
extensions:
- id: x-parser-unique-object-id
value: streaming-detection
securitySchemes: []
````
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.