Audio Event Detection

Audio Event Detection identifies non-speech sounds in a recording: instruments, human vocalizations such as laughter and coughing, and environmental noises such as a knock or a gunshot. The response lists the events it confidently detected, so its shape follows the contents of the audio. The endpoint produces no transcript, diarization, or PII/PHI tagging.

Audio Event Detection (batch)

Returns application/json.

Field	Type	Contents
`probs`	object	One entry per confidently detected event, each a probability between 0 and 1. Empty when nothing was detected.
`duration_ms`	integer	Length of the processed audio, in milliseconds.

probs carries only the events the model confidently detected, so it is not a fixed set of keys and is frequently short. When nothing clears the threshold it is {}. Both fields are always present on a 200. Entries are independent: each is that event’s own probability, they are not mutually exclusive, and they are not expected to sum to 1. Threshold each event you care about on its own.

Worth knowing: an omitted event was not detected with sufficient confidence. That is not the same as the event being absent from the audio, so a missing key is not evidence that the sound did not occur.

Because entries are independent, overlapping sounds each score on their own merits. A clip with applause over laughter returns both near 1, rather than splitting one budget between them. Key names are case-sensitive, capitalized, and underscore-separated throughout.

Try it

curl -sS -X POST https://platform.modulate.ai/api/velma-2-audio-event-classifier \
  -H "X-API-Key: $MODULATE_API_KEY" \
  -F "upload_file=@audio.mp3" \
| jq -r 'if .detail then "error: \(.detail)" else
    "\(.duration_ms) ms",
    (if (.probs | length) == 0 then "no event detected" else empty end),
    (.probs | to_entries | sort_by(-.value)[] | "  \(.key): \(.value)")
  end'

import os, requests

response = requests.post(
    "https://platform.modulate.ai/api/velma-2-audio-event-classifier",
    headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
    files={"upload_file": open("audio.mp3", "rb")},
)
response.raise_for_status()
result = response.json()

probs = result["probs"]

print(f"{result['duration_ms']} ms")
if not probs:
    print("no event detected")

# Entries are independent, so threshold each one; they do not sum to 1.
for event, p in sorted(probs.items(), key=lambda item: -item[1]):
    print(f"  {event}: {p}")

# A key that is absent was not confidently detected, which is not the same
# as the sound being absent from the audio.
if probs.get("Crying", 0) >= 0.5:
    print("crying detected")

import fs from "fs";
import FormData from "form-data";

const form = new FormData();
form.append("upload_file", fs.createReadStream("audio.mp3"), { filename: "audio.mp3" });

const response = await fetch(
  "https://platform.modulate.ai/api/velma-2-audio-event-classifier",
  {
    method: "POST",
    headers: { "X-API-Key": process.env.MODULATE_API_KEY, ...form.getHeaders() },
    body: form,
  }
);

const result = await response.json();

if (!response.ok) {
  console.error(`error: ${result.detail}`);
  process.exit(1);
}
const probs = result.probs;

console.log(`${result.duration_ms} ms`);
if (Object.keys(probs).length === 0) console.log("no event detected");

// Entries are independent, so threshold each one; they do not sum to 1.
for (const [event, p] of Object.entries(probs).sort((a, b) => b[1] - a[1])) {
  console.log(`  ${event}: ${p}`);
}

// A key that is absent was not confidently detected, which is not the same
// as the sound being absent from the audio.
if ((probs.Crying ?? 0) >= 0.5) console.log("crying detected");

Output

2000 ms
  Applause: 0.998
  Laughter: 0.946

Applause and laughter both score near 1 on the same clip. Entries are independent, so overlapping sounds do not divide a single budget between them.

Response

{
  "probs": {
    "Applause": 0.998,
    "Laughter": 0.946
  },
  "duration_ms": 2000
}

When no event clears the confidence threshold, probs is empty:

{
  "probs": {},
  "duration_ms": 5400
}

What you can configure

Form field	Default	Effect
`upload_file`	required	The audio to analyze.

There are no other parameters. The model scores a fixed vocabulary of 42 events, but the response carries only those it detected.

Audio formats

Accepted extensions: .aac, .aiff, .flac, .mov, .mp3, .mp4, .ogg, .opus, .wav, .webm. Maximum file size is 100 MB. Empty files are rejected with 400, as are files whose audio cannot be decoded.

Supported events

Any of these 42 can appear in probs: Acoustic_guitar, Applause, Bark, Bass_drum, Burping_or_eructation, Bus, Cello, Chime, Clarinet, Computer_keyboard, Cough, Cowbell, Crying, Double_bass, Drawer_open_or_close, Electric_piano, Fart, Finger_snapping, Fireworks, Flute, Glockenspiel, Gong, Gunshot_or_gunfire, Harmonica, Hi_hat, Keys_jangling, Knock, Laughter, Meow, Microwave_oven, Oboe, Saxophone, Scissors, Shatter, Snare_drum, Squeak, Tambourine, Tearing, Telephone, Trumpet, Violin_or_fiddle, Writing.

API reference

Audio Event Detection Batch

Get started

By capability

Build end to end

Guides

Account admin

Audio Event Detection