> ## Documentation Index > Fetch the complete documentation index at: https://docs.modulate.ai/llms.txt > Use this file to discover all available pages before exploring further. # Music & Speech Detection > Classify audio as music, speech, or neither, with frame-level probabilities. Batch and streaming. Music & Speech Detection classifies what kind of sound is present across a clip: music, speech, both, or neither. Common uses are skipping hold music, gating a pipeline so only speech is processed downstream, and locating where a broadcast switches between talk and track. `music_prob` and `speech_prob` are independent, not a split of 100%. A singer over a backing track scores high on both at once. This is not AI Music Detection, which answers whether music was model-generated. See [AI Music Detection](/get-started/ai-music-detection). | | Batch | Streaming | | - | - | - | | Protocol | HTTP POST | WebSocket | | Input | A complete file | Live audio frames | | Output | All frames at once, plus clip totals | Frames as they are classified, then the totals | | Use case | Sorting an archive, trimming hold music from recordings | Routing a live stream, real-time scene classification | ## Music & Speech Detection (batch) Returns `application/json`. Every field is always present. | Field | Type | Contents | | - | - | - | | `filename` | string | The submitted filename, or an empty string if none was sent. | | `duration_s` | number | Total duration in **seconds**, not milliseconds. | | `primary_label` | string | `music`, `speech`, `neither`, or `unknown`. | | `music_pct` | number | Percentage of the clip classified as containing music, 0 to 100. | | `speech_pct` | number | Percentage of the clip classified as containing speech, 0 to 100. | | `latency_ms` | number | Server-side inference time. | | `frames` | array | Per-frame results, ordered, covering the full duration. | | `frames[].start_time_ms` | integer | Frame start. | | `frames[].end_time_ms` | integer | Frame end. Each frame spans 192 ms. | | `frames[].music_prob` | number | Probability the frame contains music, 0 to 1, to four decimal places. | | `frames[].speech_prob` | number | Probability the frame contains speech, 0 to 1, to four decimal places. | | `primary_label` | Condition | | - | - | | `music` | Music covers at least as much of the clip as speech, and more than zero. | | `speech` | Speech covers more of the clip than music, and more than zero. | | `neither` | Neither reached the dominant threshold for any part of the clip. | | `unknown` | No frames could be produced from the audio. | Note that duration here is `duration_s` in seconds. Most other Modulate endpoints report `duration_ms`. ### Try it ```bash curl + jq theme={null} curl -sS -X POST https://platform.modulate.ai/api/velma-2-music-detection-batch \ -H "X-API-Key: $MODULATE_API_KEY" \ -F "upload_file=@audio.mp3" \ | jq -r 'if .detail then "error: \(.detail)" else "\(.primary_label): music \(.music_pct)%, speech \(.speech_pct)%", "\(.duration_s)s, \(.frames | length) frames of 192 ms", (.frames[:3][] | " \(.start_time_ms)-\(.end_time_ms)ms music=\(.music_prob) speech=\(.speech_prob)") end' ``` ```python Python theme={null} import os, requests response = requests.post( "https://platform.modulate.ai/api/velma-2-music-detection-batch", headers={"X-API-Key": os.environ["MODULATE_API_KEY"]}, files={"upload_file": open("audio.mp3", "rb")}, ) response.raise_for_status() result = response.json() print(f"{result['primary_label']}: music {result['music_pct']}%, speech {result['speech_pct']}%") print(f"{result['duration_s']}s, {len(result['frames'])} frames of 192 ms") # frames covers the whole clip; the first three are enough to see the shape for f in result["frames"][:3]: print(f" {f['start_time_ms']}-{f['end_time_ms']}ms " f"music={f['music_prob']} speech={f['speech_prob']}") ``` ```javascript JavaScript theme={null} import fs from "fs"; import FormData from "form-data"; const form = new FormData(); form.append("upload_file", fs.createReadStream("audio.mp3"), { filename: "audio.mp3" }); const response = await fetch( "https://platform.modulate.ai/api/velma-2-music-detection-batch", { method: "POST", headers: { "X-API-Key": process.env.MODULATE_API_KEY, ...form.getHeaders() }, body: form, } ); const result = await response.json(); if (!response.ok) { console.error(`error: ${result.detail}`); process.exit(1); } console.log(`${result.primary_label}: music ${result.music_pct}%, speech ${result.speech_pct}%`); console.log(`${result.duration_s}s, ${result.frames.length} frames of 192 ms`); // frames covers the whole clip; the first three are enough to see the shape for (const f of result.frames.slice(0, 3)) { console.log(` ${f.start_time_ms}-${f.end_time_ms}ms music=${f.music_prob} speech=${f.speech_prob}`); } ``` ```text Output theme={null} speech: music 13.3%, speech 86.7% 5.76s, 30 frames of 192 ms 0-192ms music=0.0213 speech=0.9888 192-384ms music=0.0204 speech=0.9931 384-576ms music=0.0198 speech=0.9942 ``` ```json theme={null} { "filename": "my_audio.wav", "duration_s": 5.76, "primary_label": "speech", "music_pct": 13.3, "speech_pct": 86.7, "latency_ms": 1243.5, "frames": [ { "start_time_ms": 0, "end_time_ms": 192, "music_prob": 0.0213, "speech_prob": 0.9888 }, { "start_time_ms": 192, "end_time_ms": 384, "music_prob": 0.0204, "speech_prob": 0.9931 }, { "start_time_ms": 384, "end_time_ms": 576, "music_prob": 0.0198, "speech_prob": 0.9942 } ] } ``` `frames` is abridged above. A 5.76 second clip returns all 30 of its 192 ms frames. ### What you can configure | Form field | Default | Effect | | - | - | - | | `upload_file` | *required* | The audio to analyze. This endpoint takes no other parameters. | ### Audio formats Accepted extensions include `.mp3`, `.wav`, `.flac`, `.m4a`, `.mp4`, `.ogg`, `.opus`, `.webm`, `.aac`, `.aiff`, and `.mov`, plus 85 others. `.3g2`, `.3ga`, `.3gp`, `.3gpp`, `.8svx`, `.aa3`, `.aac`, `.ac3`, `.act`, `.adts`, `.aif`, `.aifc`, `.aiff`, `.alac`, `.amb`, `.amr`, `.ape`, `.asf`, `.at3`, `.au`, `.avi`, `.avr`, `.awb`, `.bwf`, `.c2`, `.caf`, `.dss`, `.dts`, `.dtshd`, `.eac3`, `.ec3`, `.f4a`, `.f4b`, `.flac`, `.flv`, `.gsm`, `.iff`, `.m2a`, `.m2ts`, `.m4a`, `.m4b`, `.m4r`, `.m4v`, `.mka`, `.mkv`, `.mlp`, `.mmf`, `.mov`, `.mp+`, `.mp1`, `.mp2`, `.mp3`, `.mp4`, `.mpa`, `.mpc`, `.mpeg`, `.mpg`, `.mpga`, `.mpp`, `.mts`, `.mxf`, `.nist`, `.oga`, `.ogg`, `.ogx`, `.oma`, `.omg`, `.opus`, `.paf`, `.pvf`, `.qcp`, `.ra`, `.rf64`, `.rka`, `.rm`, `.rmvb`, `.sf`, `.shn`, `.snd`, `.sph`, `.svx`, `.tak`, `.thd`, `.ts`, `.tta`, `.vob`, `.voc`, `.vqf`, `.w64`, `.wav`, `.wave`, `.weba`, `.webm`, `.wma`, `.wmv`, `.wv` Maximum file size is 100 MB. Empty files are rejected with `400`. ## Music & Speech Detection (streaming) One `frame` message per completed 192 ms window, then a `done` message carrying clip totals computed across everything received. | Message | Payload | | - | - | | `frame` | `frame` object with `start_time_ms`, `end_time_ms`, `music_prob`, `speech_prob`. Same shape as a batch frame. | | `done` | `duration_ms`, `frame_count`, `music_pct`, `speech_pct`, `primary_label`. The server closes after this. | | `error` | `error` string. The server closes after this. | Note that `done` reports `duration_ms` in milliseconds, where the batch response reports `duration_s` in seconds. Send audio as binary WebSocket frames of any size; the server buffers and windows internally. Send an empty text frame (`""`) to end the stream. The server then flushes remaining audio, delivers final frames, sends `done`, and closes. ### Try it ```python Python theme={null} import os, asyncio, json, websockets API_KEY = os.environ["MODULATE_API_KEY"] AUDIO_FILE = "audio.mp3" CHUNK_SIZE = 65536 async def stream(): url = ( f"wss://platform.modulate.ai/api/velma-2-music-detection-streaming" f"?api_key={API_KEY}&audio_format=mp3" ) async with websockets.connect(url) as ws: async def send(): with open(AUDIO_FILE, "rb") as f: while chunk := f.read(CHUNK_SIZE): await ws.send(chunk) await ws.send("") # end of stream async def receive(): async for message in ws: msg = json.loads(message) if msg["type"] == "frame": f = msg["frame"] print(f" {f['start_time_ms']}-{f['end_time_ms']}ms " f"music={f['music_prob']} speech={f['speech_prob']}") elif msg["type"] == "done": print(f"done: {msg['duration_ms']}ms of audio, {msg['frame_count']} frames") print(f"{msg['primary_label']}: music {msg['music_pct']}%, " f"speech {msg['speech_pct']}%") break elif msg["type"] == "error": raise RuntimeError(msg["error"]) await asyncio.gather(send(), receive()) asyncio.run(stream()) ``` ```javascript JavaScript theme={null} import { WebSocket } from "ws"; import { createReadStream } from "fs"; const url = `wss://platform.modulate.ai/api/velma-2-music-detection-streaming` + `?api_key=${process.env.MODULATE_API_KEY}&audio_format=mp3`; const ws = new WebSocket(url); ws.on("open", () => { const stream = createReadStream("audio.mp3", { highWaterMark: 65536 }); stream.on("data", (chunk) => ws.send(chunk)); stream.on("end", () => ws.send("")); }); ws.on("message", (data) => { const msg = JSON.parse(data); if (msg.type === "frame") { const f = msg.frame; console.log(` ${f.start_time_ms}-${f.end_time_ms}ms music=${f.music_prob} speech=${f.speech_prob}`); } else if (msg.type === "done") { console.log(`done: ${msg.duration_ms}ms of audio, ${msg.frame_count} frames`); console.log(`${msg.primary_label}: music ${msg.music_pct}%, speech ${msg.speech_pct}%`); ws.close(); } else if (msg.type === "error") { console.error(`error: ${msg.error}`); } }); // A rejected handshake closes without sending an error frame first. ws.on("close", (code) => { if (code !== 1000) console.error(`error: connection closed ${code}`); }); ws.on("error", (err) => console.error(`error: ${err.message}`)); ``` ```text Output theme={null} 0-192ms music=0.0213 speech=0.9888 192-384ms music=0.0204 speech=0.9931 ... done: 15360ms of audio, 80 frames speech: music 42.5%, speech 57.5% ``` One `frame` line is printed per 192 ms of audio, so a 15 second clip prints 80 of them before `done`. ```json theme={null} { "type": "frame", "frame": { "start_time_ms": 0, "end_time_ms": 192, "music_prob": 0.0213, "speech_prob": 0.9888 } } { "type": "frame", "frame": { "start_time_ms": 192, "end_time_ms": 384, "music_prob": 0.0204, "speech_prob": 0.9931 } } { "type": "done", "duration_ms": 15360, "frame_count": 80, "music_pct": 42.5, "speech_pct": 57.5, "primary_label": "speech" } ``` A stream ends with an empty text frame. A shell pipe sends binary frames only, so `curl` and `websocat` cannot close a stream cleanly. Use one of the clients above. ### What you can configure | Query parameter | Default | Effect | | - | - | - | | `api_key` | *required* | The API key. WebSocket connections authenticate through the query string, not a header. | | `audio_format` | *required* | How the server decodes the bytes sent. Required on every connection, including container formats. | | `sample_rate` | *required for raw* | Sample rate in Hz. Required for raw formats. Must not be sent for container formats. | | `num_channels` | *required for raw* | 1 to 8. Required for raw formats. Must not be sent for container formats. | ### Audio formats `audio_format` is required on every connection, including self-describing containers. Omitting it closes the connection with code `1003`. **Container formats.** Accepted values include `mp3`, `wav`, `flac`, `m4a`, `mp4`, `ogg`, `opus`, `webm`, `aac`, `aiff`, `wma`, `amr`, and `au`, plus 66 others. The stream header carries sample rate and channel count, so `sample_rate` and `num_channels` are ignored if supplied. `3g2`, `3ga`, `3gp`, `3gpp`, `8svx`, `aa3`, `aac`, `ac3`, `act`, `adts`, `aif`, `aifc`, `aiff`, `amb`, `amr`, `asf`, `at3`, `au`, `avr`, `awb`, `bwf`, `c2`, `caf`, `dss`, `dts`, `dtshd`, `eac3`, `ec3`, `f4a`, `f4b`, `flac`, `gsm`, `iff`, `m2a`, `m2ts`, `m4a`, `m4b`, `m4r`, `m4v`, `mka`, `mkv`, `mlp`, `mp+`, `mp1`, `mp2`, `mp3`, `mp4`, `mpa`, `mpc`, `mpga`, `mpp`, `mts`, `oga`, `ogg`, `ogx`, `oma`, `omg`, `opus`, `paf`, `pvf`, `qcp`, `ra`, `rf64`, `rm`, `rmvb`, `snd`, `svx`, `thd`, `ts`, `tta`, `voc`, `vqf`, `w64`, `wav`, `wave`, `weba`, `webm`, `wma`, `wmv` The MP4-family values (`mp4`, `m4a`, `m4b`, `m4r`, `m4v`, `3gp`, `3gpp`, `3ga`, `3g2`, `f4a`, `f4b`) must be sent in a streamable layout. Anything else ends the connection with an audio-processing error. **Raw formats.** `s8`, `s16le`, `s16be`, `s24le`, `s24be`, `s32le`, `s32be`, `u8`, `u16le`, `u16be`, `u24le`, `u24be`, `u32le`, `u32be`, `f32le`, `f32be`, `f64le`, `f64be`, `mulaw`, `alaw`, `g722`, `vox`. These are headerless, so `sample_rate` and `num_channels` are both required. `g722` and `vox` are mono-only: `num_channels` must be `1`. `sample_rate` accepts `8000`, `11025`, `16000`, `22050`, `32000`, `44100`, `48000`, `96000`. Common raw configurations: | Source | `audio_format` | `sample_rate` | `num_channels` | | - | - | - | - | | Native app default | `s16le` | `16000` | `1` | | Web Audio API (`AudioWorklet`) | `f32le` | `48000` | `1` | | Native stereo capture | `s16le` | `48000` | `2` | | Telephony (mu-law) | `mulaw` | `8000` | `1` | | Telephony (A-law) | `alaw` | `8000` | `1` | To convert a file to raw PCM: ```bash theme={null} ffmpeg -i audio.mp3 -ar 16000 -ac 1 -f s16le audio.raw ``` Format parameters are validated at connection time, closing with `1003`, and again while decoding, closing with `4002`. The second catches undecodable audio and a mismatch between the declared `audio_format` and the bytes sent. ## API reference * [Music & Speech Detection Batch](/api-reference/music-detection/batch) * [Music & Speech Detection Streaming](/api-reference/music-detection/streaming) This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.