Paper: “Decoding Ambiguous Emotions with Test-Time Scaling in Audio-Language Models” - Jia et al., February 2026. arxiv.org/abs/2602.03873
Emotion recognition in speech is harder than it looks. Most systems treat it as a classification problem - you give it audio, it tells you whether the person sounds happy, sad, angry, or neutral. Clean, discrete, done.
Real emotional expression isn’t like that. The same sentence can sound simultaneously frustrated and resigned. A laugh can be joyful or bitter. Human annotators often disagree on what they’re hearing - not because they’re bad at their jobs, but because the emotion genuinely isn’t singular.
This paper tackles that messiness head-on. The authors introduce the first benchmark specifically designed for ambiguous emotion recognition in speech, using modern audio-language models (ALMs) - large multimodal models that can process audio and reason about it in natural language. They then ask: can test-time scaling (TTS) - giving the model more computation at inference time - help these models handle emotional ambiguity better?
The short answer: yes, meaningfully so. But the relationship between model size, scaling strategy, and ambiguity is more nuanced than a simple “more compute = better.”
The paper is motivated by two separate observations that hadn’t been brought together before.
Firstly, audio-language models have become remarkably capable at affective reasoning. Unlike older classifiers trained on narrow emotion datasets, models like Qwen2-Audio or GPT-4o can process raw speech and generate nuanced descriptions of what they hear. But no one had systematically tested whether this capability holds up specifically on ambiguous cases - the utterances where even humans can’t agree.
Secondly, test-time scaling has been shown to improve performance on hard reasoning tasks in NLP by giving models more “thinking time” (via strategies like chain-of-thought sampling, majority voting, or self-consistency). If a model is uncertain about a classification, asking it to reason more carefully, or sampling multiple answers and aggregating them, can reduce errors.
The hypothesis is that emotional ambiguity in speech is precisely the kind of hard, uncertain case where test-time scaling should help - and that the benefit should be largest on utterances with the most inter-annotator disagreement.
The authors evaluate eight state-of-the-art audio-language models across three speech emotion recognition datasets, testing five different test-time scaling strategies.
Models tested include Qwen2-Audio, Ultravox v0.3 and v0.4, Gemini 2 Flash, GPT-4o, and others - spanning a range of model sizes and architectures.
Datasets include three well-known speech emotion benchmarks: IEMOCAP, EmoV-DB, and CREMA-D. Crucially, the authors identify ambiguous subsets within these datasets - utterances where annotators showed low agreement - and evaluate models separately on ambiguous versus clear-cut samples.
Test-time scaling strategies vary in how they use additional compute: some use majority voting across multiple sampled outputs, some prompt the model to reason step-by-step before answering (chain-of-thought), and others use self-consistency methods that aggregate multiple reasoning chains.
The evaluation is systematic: every combination of model, dataset, ambiguity level, and TTS strategy is tested, giving a comprehensive picture of what actually works and what doesn’t.
A few key findings emerge:
Test-time scaling helps, especially on ambiguous samples. Across most model-dataset combinations, applying TTS strategies improves emotion recognition accuracy, with the gains concentrated on the ambiguous utterances. Clear emotional speech benefits less - the models already get those right most of the time.
Model capacity interacts with TTS in non-trivial ways. Larger, more capable models tend to benefit more from TTS strategies. Smaller models sometimes perform worse under certain scaling approaches, suggesting they lack the representational capacity to make use of additional reasoning steps.
No single TTS strategy dominates. Majority voting, chain-of-thought, and self-consistency each have regimes where they outperform the others. The best strategy depends on the model and the dataset. This is practically inconvenient but scientifically interesting - it suggests that emotional ambiguity isn’t a uniform challenge that one technique can cleanly solve.
GPT-4o and Gemini 2 Flash show the strongest baseline performance, but even they struggle on genuinely ambiguous samples without TTS support. The benchmark exposes a real ceiling in current model capabilities when emotional signals are inherently mixed.
What this paper is really doing is reframing the emotion recognition problem. The dominant framing - “classify this utterance into one of N emotion categories” - breaks down when the emotion genuinely doesn’t fit a single category. The authors argue that this isn’t a data quality problem to be cleaned away; it’s a fundamental feature of how human emotion works.
By building a benchmark around ambiguity rather than against it, and showing that test-time scaling provides meaningful improvements on exactly these hard cases, the paper points towards a more realistic direction for emotionally intelligent AI: systems that reason about affective uncertainty rather than collapsing it into false confidence.
For socially aware conversational AI - assistants, therapy tools, accessibility systems - this matters a lot. A system that says “you sound frustrated” when you’re actually expressing bittersweet relief isn’t just inaccurate; it’s socially disconnected. Getting emotion recognition right on the hard cases is where the real work is.
The benchmark itself is probably the most durable contribution here. A standardised way to evaluate models specifically on emotional ambiguity gives the field something to optimise against - and the interaction effects between model capacity, TTS strategy, and ambiguity level are rich enough that there’s clearly more to explore.