Emotion Detection

Emotion Detection classifies emotional tone from the voice signal. Classification is acoustic: it reflects how something was said, not the words used. Two recordings of identical text can receive different labels if the delivery differs. The endpoint produces no transcript, diarization, or enrichment data.

Emotion Detection (batch)

Returns application/json.

Field	Type	Contents
`emotion`	string	The label for the whole file. Always present, never null, at any file length.
`time_series`	array	Consecutive fixed-length windows over the file, each with its own label. Empty when the file is shorter than one window.
`time_series[].start_ms`	integer	Window start, in milliseconds from the beginning of the file.
`time_series[].duration_ms`	integer	Window length.
`time_series[].emotion`	string	The label for that window.

There is no top-level duration field. Derive audio length from the last window if it is needed.

Try it

curl -X POST https://platform.modulate.ai/api/velma-2-emotion-batch \
  -H "X-API-Key: $MODULATE_API_KEY" \
  -F "upload_file=@audio.mp3"

import os, requests

response = requests.post(
    "https://platform.modulate.ai/api/velma-2-emotion-batch",
    headers={"X-API-Key": os.environ["MODULATE_API_KEY"]},
    files={"upload_file": open("audio.mp3", "rb")},
)
response.raise_for_status()
result = response.json()

print(f"Overall: {result['emotion']}")
for window in result["time_series"]:
    end = window["start_ms"] + window["duration_ms"]
    print(f"  {window['start_ms']}-{end} ms: {window['emotion']}")

import fs from "fs";
import FormData from "form-data";

const form = new FormData();
form.append("upload_file", fs.createReadStream("audio.mp3"), { filename: "audio.mp3" });

const response = await fetch(
  "https://platform.modulate.ai/api/velma-2-emotion-batch",
  {
    method: "POST",
    headers: { "X-API-Key": process.env.MODULATE_API_KEY, ...form.getHeaders() },
    body: form,
  }
);

const result = await response.json();
console.log(`Overall: ${result.emotion}`);
for (const w of result.time_series) {
  console.log(`  ${w.start_ms}-${w.start_ms + w.duration_ms} ms: ${w.emotion}`);
}

Response

{
  "emotion": "Happy",
  "time_series": [
    { "start_ms": 0, "duration_ms": 15000, "emotion": "Happy" },
    { "start_ms": 15000, "duration_ms": 15000, "emotion": "Neutral" }
  ]
}

What you can configure

Form field	Default	Effect
`upload_file`	required	The audio to analyze.
`use_ensemble`	`false`	Set `true` for a slower, more thorough analysis. Labels may change and latency increases.
`training_permitted`	`true`	Set `false` to exclude this request’s audio from model improvement.

Audio formats

Accepted extensions: .aac, .aiff, .flac, .mov, .mp3, .mp4, .ogg, .opus, .wav, .webm. Maximum file size is 100 MB. Empty files are rejected with 400.

Labels

Neutral, Calm, Happy, Amused, Excited, Proud, Affectionate, Interested, Hopeful, Frustrated, Angry, Contemptuous, Concerned, Afraid, Sad, Ashamed, Bored, Tired, Surprised, Anxious, Stressed, Disgusted, Disappointed, Confused, Relieved, Confident. The same set is used by the emotion_signal enrichment on Multilingual Transcription.

Working with the time series

Windows are consecutive and cover the file from the start. A trailing remainder shorter than one full window is dropped. A file shorter than a single window returns an empty time_series, but emotion is still present, so the whole-file label is safe to rely on at any length.

result = response.json()

overall = result["emotion"]

if result["time_series"]:
    shifts = [w["emotion"] for w in result["time_series"]]
    print(f"{overall} overall; window by window: {shifts}")
else:
    print(f"{overall} (file shorter than one window)")

Emotion alongside a transcript

Multilingual Transcription accepts emotion_signal=true, which attaches an emotion label to every utterance from the same label set, in the same call as the transcript. Use that when a transcript is also needed. This endpoint exists for whole-file classification with no transcript. See Transcription.

API reference

Emotion Detection Batch

Get started

By capability

Build end to end

Guides

Emotion Detection