whatmodel.sh / GPT-4o Audio Preview (2024-12-17)

GPT-4o Audio Preview (2024-12-17)

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

GPT-4o Audio Preview (2024-12-17) is a closed-weights language model from OpenAI, released December 2024. Licence: proprietary.

Catalogue identity notes

Comparison-only. OpenAI audio-input API snapshot 'gpt-4o-audio-preview-2024-12-17' (exact identifier and release date in AHELM Table 3, arXiv 2508.21376v2). Distinct from the text GPT-4o snapshots. No run route or price recorded.

48.8±2.4comparison only: no current run option verifiedfrom 10 benchmark results across 1 source

$ how it compares

The LLMs ranked either side of GPT-4o Audio Preview (2024-12-17), out of 828 with a score, then more from OpenAI.

#modelscorewhereRAMreleased
437Nova Lite Amazon49±2api $0.10/M2024-12
438MegaBeam-Mistral-7B-512k AWS (aws-prototyping)49±4local~6.1GB2024-07
439GPT-4o Audio Preview (2024-12-17) OpenAI49±2comparison only2024-12
440Claude 3 Sonnet Anthropic49±1comparison only2024-02
441AceReason-Nemotron-1.1-7B NVIDIA49±2local~6GB2025-06
more from OpenAI
1GPT-6 Astra OpenAI87±1api $20.00/M2026-09
3GPT-6.1 Sol OpenAI85±1api $4.00/M2026-09
7GPT-6 Sol OpenAI83±1api $4.00/M2026-09

see all LLMs · find the ones that fit your machine

$ what it's good at

other tests10 not in a task · 97% of its score
testresultvs the others in its rundifficultycountswhy
AHELM Speech Robust Bench (LibriSpeech-clean with noise) transcription WER45.1% ✎▂·█··▆▄·▂▂·▄ beats 3/13avg 33%▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 49%typical
AHELM CoRe-Bench conversational audio reasoning (pseudo-exact match)35.9% ✎▂·▄··▂▂·▄▂█▄ beats 2/13medium · avg 53%▇▇▇▇▇▇▇ 18%at its level
AHELM LibriSpeech transcription by audio LLMs (WER)9.5% ✎▃···▃·█▃▅▃▅█ beats 5/13avg 11%▇▇▇▇ 10%blunt
AHELM MUStARD audio sarcasm detection (exact match)58.3% ✎▂·······▆█▆▆ beats 6/13very hard · avg 57%▇▇ 5.9%blunt
AHELM VoxCeleb2 same-speaker detection (exact match)57.5% ✎▂·▂·▂█▂▄▄··▄ beats 9/13very hard · avg 54%▇▇ 5.5%sharp · too hard for it · noisy
AHELM AIR-Bench Foundation music subsets (exact match)56.0% ✎▃█▃▅▃·▃·▃▅▃▃ beats 6/13medium · avg 57%▇▇ 4.4%blunt · at its level
AHELM MELD audio emotion classification (pseudo-exact match)49.7% ✎▃··▃▃▃·▃▅▅█▅ beats 12/13medium · avg 40%▇ 2.6%blunt · at its level
AHELM FLEURS 7-language transcription by audio LLMs (WER)45.6% ✎▄▂···▂▆▂█▂·▂ beats 10/13avg 80%▇ 1.6%blunt · noisy
2 more tests
testresultvs the others in its rundifficultycountswhy
AHELM Multilingual LibriSpeech 7-language transcription by audio LLMs (WER)7.3% ✎▂▂▂··▂▂·▂▂█▆ beats 8/13avg 13%▇ 0.2%blunt · noisy
AHELM VocalSound human-sound recognition (prefix exact match)83.7% ✎██▄▄·█·█·█▄▄ beats 11/13easy · avg 67%▇ 0.1%blunt · at its level · noisy

$ where to run it

No verified API or download recorded yet · add one

$ how the score was worked out · AHELM: A Holistic Evaluation… 97%

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
AHELM: A Holistic Evaluation of Audio-Language Models (Stanford CRFM, arXiv:2508.21376v2)10▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 97%CC BY 4.0
prior · pull toward the middle▇ 3%
weighted average= 48.8

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json