Audio Event Detection

Audio Event Detection identifies non-speech sounds in a recording: instruments, human vocalizations such as laughter and coughing, and environmental noises such as a knock or a gunshot. Every supported event receives a probability on every call, so the response shape does not change with the contents of the audio. The endpoint produces no transcript, diarization, or PII/PHI tagging.

Audio Event Detection (batch)

Returns application/json.

Field	Type	Contents
`probs`	object	A probability between 0 and 1 for each of the 42 supported events. Every key is always present.
`probs.cry`	number	Probability that the audio contains crying. Scored independently of every other key.
`duration_ms`	integer	Length of the processed audio, in milliseconds.

Worth knowing: cry and the other 41 events are not on the same scale. cry is an independent probability, while the other 41 keys are drawn from one shared distribution that sums to 1 across them. Comparing cry against Laughter compares two different quantities.

Because the 41 shared keys sum to 1, they rank the most prominent event rather than reporting independent detections. A clip containing both applause and laughter splits probability between Applause and Laughter, so neither reaches the value it would reach alone. Read those keys by taking the highest rather than by thresholding each one. cry is independent, so a threshold on cry is meaningful. Key names are case-sensitive and inconsistent in form: cry is the only lowercase key, Hi-hat is the only key containing a hyphen, and the rest are capitalized with underscores.

Try it

curl -X POST https://platform.modulate.ai/api/velma-2-audio-event-classifier \
  -H "X-API-Key: $MODULATE_API_KEY" \
  -F "upload_file=@audio.mp3"

import os, requests

response = requests.post(
    "https://platform.modulate.ai/api/velma-2-audio-event-classifier",
    headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
    files={"upload_file": open("audio.mp3", "rb")},
)
response.raise_for_status()
result = response.json()

probs = result["probs"]

# The shared keys sum to 1, so rank them rather than thresholding each one.
shared = {event: p for event, p in probs.items() if event != "cry"}
top_event, top_prob = max(shared.items(), key=lambda item: item[1])

print(f"{result['duration_ms']} ms")
print(f"most prominent event: {top_event} ({top_prob:.3f})")
print(f"crying: {probs['cry']:.3f}")

import fs from "fs";
import FormData from "form-data";

const form = new FormData();
form.append("upload_file", fs.createReadStream("audio.mp3"), { filename: "audio.mp3" });

const response = await fetch(
  "https://platform.modulate.ai/api/velma-2-audio-event-classifier",
  {
    method: "POST",
    headers: { "X-API-Key": process.env.MODULATE_API_KEY, ...form.getHeaders() },
    body: form,
  }
);

const result = await response.json();
const { cry, ...shared } = result.probs;

// The shared keys sum to 1, so rank them rather than thresholding each one.
const [topEvent, topProb] = Object.entries(shared).sort((a, b) => b[1] - a[1])[0];

console.log(`${result.duration_ms} ms`);
console.log(`most prominent event: ${topEvent} (${topProb.toFixed(3)})`);
console.log(`crying: ${cry.toFixed(3)}`);

Response

{
  "probs": {
    "cry": 0.01,
    "Acoustic_guitar": 0.002,
    "Applause": 0.03,
    "Bark": 0.001,
    "Bass_drum": 0.001,
    "Burping_or_eructation": 0.002,
    "Bus": 0.001,
    "Cello": 0.001,
    "Chime": 0.001,
    "Clarinet": 0.001,
    "Computer_keyboard": 0.001,
    "Cough": 0.01,
    "Cowbell": 0.001,
    "Double_bass": 0.001,
    "Drawer_open_or_close": 0.001,
    "Electric_piano": 0.001,
    "Fart": 0.002,
    "Finger_snapping": 0.005,
    "Fireworks": 0.001,
    "Flute": 0.001,
    "Glockenspiel": 0.001,
    "Gong": 0.001,
    "Gunshot_or_gunfire": 0.001,
    "Harmonica": 0.001,
    "Hi-hat": 0.001,
    "Keys_jangling": 0.001,
    "Knock": 0.002,
    "Laughter": 0.913,
    "Meow": 0.001,
    "Microwave_oven": 0.001,
    "Oboe": 0.001,
    "Saxophone": 0.001,
    "Scissors": 0.001,
    "Shatter": 0.001,
    "Snare_drum": 0.001,
    "Squeak": 0.001,
    "Tambourine": 0.001,
    "Tearing": 0.001,
    "Telephone": 0.002,
    "Trumpet": 0.001,
    "Violin_or_fiddle": 0.001,
    "Writing": 0.001
  },
  "duration_ms": 2000
}

What you can configure

Form field	Default	Effect
`upload_file`	required	The audio to analyze.

There are no other parameters. The set of scored events is fixed, and the response always carries all 42 keys.

Audio formats

Accepted extensions: .aac, .aiff, .flac, .mov, .mp3, .mp4, .ogg, .opus, .wav, .webm. Maximum file size is 100 MB. Empty files are rejected with 400, as are files whose audio cannot be decoded.

Supported events

cry is scored independently. These 41 are drawn from one shared distribution: Acoustic_guitar, Applause, Bark, Bass_drum, Burping_or_eructation, Bus, Cello, Chime, Clarinet, Computer_keyboard, Cough, Cowbell, Double_bass, Drawer_open_or_close, Electric_piano, Fart, Finger_snapping, Fireworks, Flute, Glockenspiel, Gong, Gunshot_or_gunfire, Harmonica, Hi-hat, Keys_jangling, Knock, Laughter, Meow, Microwave_oven, Oboe, Saxophone, Scissors, Shatter, Snare_drum, Squeak, Tambourine, Tearing, Telephone, Trumpet, Violin_or_fiddle, Writing.

API reference

Audio Event Detection Batch

Get started

By capability

Build end to end

Guides

Audio Event Detection