Music & Speech Detection

Music & Speech Detection classifies what kind of sound is present across a clip: music, speech, both, or neither. Common uses are skipping hold music, gating a pipeline so only speech is processed downstream, and locating where a broadcast switches between talk and track. music_prob and speech_prob are independent, not a split of 100%. A singer over a backing track scores high on both at once. This is not AI Music Detection, which answers whether music was model-generated. See AI Music Detection.

	Batch	Streaming
Protocol	HTTP POST	WebSocket
Input	A complete file	Live audio frames
Output	All frames at once, plus clip totals	Frames as they are classified, then the totals
Use case	Sorting an archive, trimming hold music from recordings	Routing a live stream, real-time scene classification

Music & Speech Detection (batch)

Returns application/json. Every field is always present.

Field	Type	Contents
`filename`	string	The submitted filename, or an empty string if none was sent.
`duration_s`	number	Total duration in seconds, not milliseconds.
`primary_label`	string	`music`, `speech`, `neither`, or `unknown`.
`music_pct`	number	Percentage of the clip classified as containing music, 0 to 100.
`speech_pct`	number	Percentage of the clip classified as containing speech, 0 to 100.
`latency_ms`	number	Server-side inference time.
`frames`	array	Per-frame results, ordered, covering the full duration.
`frames[].start_time_ms`	integer	Frame start.
`frames[].end_time_ms`	integer	Frame end. Each frame spans 192 ms.
`frames[].music_prob`	number	Probability the frame contains music, 0 to 1, to four decimal places.
`frames[].speech_prob`	number	Probability the frame contains speech, 0 to 1, to four decimal places.

`primary_label`	Condition
`music`	Music covers at least as much of the clip as speech, and more than zero.
`speech`	Speech covers more of the clip than music, and more than zero.
`neither`	Neither reached the dominant threshold for any part of the clip.
`unknown`	No frames could be produced from the audio.

Note that duration here is duration_s in seconds. Most other Modulate endpoints report duration_ms.

Try it

curl -X POST https://platform.modulate.ai/api/velma-2-music-detection-batch \
  -H "X-API-Key: $MODULATE_API_KEY" \
  -F "upload_file=@audio.mp3"

import os, requests

response = requests.post(
    "https://platform.modulate.ai/api/velma-2-music-detection-batch",
    headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
    files={"upload_file": open("audio.mp3", "rb")},
)
response.raise_for_status()
result = response.json()

print(f"{result['primary_label']}: music {result['music_pct']}%, speech {result['speech_pct']}%")
for frame in result["frames"]:
    print(f"  {frame['start_time_ms']}ms - {frame['end_time_ms']}ms  "
          f"music={frame['music_prob']:.4f}  speech={frame['speech_prob']:.4f}")

import fs from "fs";
import FormData from "form-data";

const form = new FormData();
form.append("upload_file", fs.createReadStream("audio.mp3"), { filename: "audio.mp3" });

const response = await fetch(
  "https://platform.modulate.ai/api/velma-2-music-detection-batch",
  {
    method: "POST",
    headers: { "X-API-Key": process.env.MODULATE_API_KEY, ...form.getHeaders() },
    body: form,
  }
);

const result = await response.json();
console.log(`${result.primary_label}: music ${result.music_pct}%, speech ${result.speech_pct}%`);

Response

{
  "filename": "my_audio.wav",
  "duration_s": 5.76,
  "primary_label": "speech",
  "music_pct": 0.0,
  "speech_pct": 86.7,
  "latency_ms": 1243.5,
  "frames": [
    { "start_time_ms": 0,   "end_time_ms": 192, "music_prob": 0.0213, "speech_prob": 0.9888 },
    { "start_time_ms": 192, "end_time_ms": 384, "music_prob": 0.0204, "speech_prob": 0.9931 }
  ]
}

What you can configure

Form field	Default	Effect
`upload_file`	required	The audio to analyze. This endpoint takes no other parameters.

Audio formats

Accepted extensions: .aac, .flac, .m4a, .mp3, .mp4, .ogg, .opus, .wav. Maximum file size is 100 MB. Empty files are rejected with 400.

Music & Speech Detection (streaming)

One frame message per completed 192 ms window, then a done message carrying clip totals computed across everything received.

Message	Payload
`frame`	`frame` object with `start_time_ms`, `end_time_ms`, `music_prob`, `speech_prob`. Same shape as a batch frame.
`done`	`duration_ms`, `frame_count`, `music_pct`, `speech_pct`, `primary_label`. The server closes after this.
`error`	`error` string. The server closes after this.

Note that done reports duration_ms in milliseconds, where the batch response reports duration_s in seconds. Send audio as binary WebSocket frames of any size; the server buffers and windows internally. Send an empty text frame ("") to end the stream. The server then flushes remaining audio, delivers final frames, sends done, and closes.

Try it

websocat "wss://platform.modulate.ai/api/velma-2-music-detection-streaming?api_key=$MODULATE_API_KEY&audio_format=mp3" \
  --binary - < audio.mp3

import os, asyncio, json, websockets

API_KEY = os.environ["MODULATE_API_KEY"]
AUDIO_FILE = "audio.mp3"
CHUNK_SIZE = 65536

async def stream():
    url = (
        f"wss://platform.modulate.ai/api/velma-2-music-detection-streaming"
        f"?api_key={API_KEY}&audio_format=mp3"
    )
    async with websockets.connect(url) as ws:
        async def send():
            with open(AUDIO_FILE, "rb") as f:
                while chunk := f.read(CHUNK_SIZE):
                    await ws.send(chunk)
            await ws.send("")  # end of stream

        async def receive():
            async for message in ws:
                msg = json.loads(message)
                if msg["type"] == "frame":
                    f = msg["frame"]
                    print(f"  {f['start_time_ms']}ms - {f['end_time_ms']}ms  "
                          f"music={f['music_prob']:.4f}  speech={f['speech_prob']:.4f}")
                elif msg["type"] == "done":
                    print(f"Done: {msg['duration_ms']}ms, {msg['frame_count']} frames")
                    print(f"{msg['primary_label']}: music {msg['music_pct']}%, "
                          f"speech {msg['speech_pct']}%")
                    break
                elif msg["type"] == "error":
                    raise RuntimeError(msg["error"])

        await asyncio.gather(send(), receive())

asyncio.run(stream())

import { WebSocket } from "ws";
import { createReadStream } from "fs";

const url =
  `wss://platform.modulate.ai/api/velma-2-music-detection-streaming` +
  `?api_key=${process.env.MODULATE_API_KEY}&audio_format=mp3`;

const ws = new WebSocket(url);

ws.on("open", () => {
  const stream = createReadStream("audio.mp3", { highWaterMark: 65536 });
  stream.on("data", (chunk) => ws.send(chunk));
  stream.on("end", () => ws.send(""));
});

ws.on("message", (data) => {
  const msg = JSON.parse(data);
  if (msg.type === "frame") {
    const f = msg.frame;
    console.log(`${f.start_time_ms}ms - ${f.end_time_ms}ms  music=${f.music_prob}  speech=${f.speech_prob}`);
  } else if (msg.type === "done") {
    console.log(`${msg.primary_label}: music ${msg.music_pct}%, speech ${msg.speech_pct}%`);
    ws.close();
  } else if (msg.type === "error") {
    throw new Error(msg.error);
  }
});

Messages received

{ "type": "frame", "frame": { "start_time_ms": 0, "end_time_ms": 192, "music_prob": 0.0213, "speech_prob": 0.9888 } }
{ "type": "frame", "frame": { "start_time_ms": 192, "end_time_ms": 384, "music_prob": 0.0204, "speech_prob": 0.9931 } }
{ "type": "done", "duration_ms": 15360, "frame_count": 80, "music_pct": 42.5, "speech_pct": 57.5, "primary_label": "speech" }

WebSocket endpoints cannot be exercised with cURL. For command-line testing use websocat.

What you can configure

Query parameter	Default	Effect
`api_key`	required	The API key. WebSocket connections authenticate through the query string, not a header.
`audio_format`	required	How the server decodes the bytes sent. Required on every connection, including container formats.
`sample_rate`	required for raw	Sample rate in Hz. Required for raw formats. Must not be sent for container formats.
`num_channels`	required for raw	1 to 8. Required for raw formats. Must not be sent for container formats.

Audio formats

audio_format is required on every connection, including self-describing containers. Omitting it closes the connection with code 1003.

Streaming accepts a narrower set than batch. m4a, mp4, and opus are batch-only. Container formats. wav, mp3, ogg, flac, webm, aac, aiff. The stream carries sample rate and channel count, so sample_rate and num_channels must be omitted. Raw formats. s8, s16le, s16be, s24le, s24be, s32le, s32be, u8, u16le, u16be, u24le, u24be, u32le, u32be, f32le, f32be, f64le, f64be, mulaw, alaw. These are headerless, so sample_rate and num_channels are both required. sample_rate accepts 8000, 11025, 16000, 22050, 32000, 44100, 48000, 96000. Common raw configurations:

Source	`audio_format`	`sample_rate`	`num_channels`
Native app default	`s16le`	`16000`	`1`
Web Audio API (`AudioWorklet`)	`f32le`	`48000`	`1`
Native stereo capture	`s16le`	`48000`	`2`
Telephony (mu-law)	`mulaw`	`8000`	`1`
Telephony (A-law)	`alaw`	`8000`	`1`

To convert a file to raw PCM:

ffmpeg -i audio.mp3 -ar 16000 -ac 1 -f s16le audio.raw

Format parameters are validated at connection time, closing with 1003, and again while decoding, closing with 4002. The second catches undecodable audio and a mismatch between the declared audio_format and the bytes sent.

Get started

By capability

Guides

Music & Speech Detection

Music & Speech Detection (batch)

Try it

What you can configure

Audio formats

Music & Speech Detection (streaming)

Try it

What you can configure

Audio formats

API reference

​Music & Speech Detection (batch)

​Try it

​What you can configure

​Audio formats

​Music & Speech Detection (streaming)

​Try it

​What you can configure

​Audio formats

​API reference

Music & Speech Detection (batch)

Try it

What you can configure

Audio formats

Music & Speech Detection (streaming)

Try it

What you can configure

Audio formats

API reference