Deprecated: The each() function is deprecated. This message will be suppressed on further calls in /home/zhenxiangba/zhenxiangba.com/public_html/phproxy-improved-master/index.php on line 456 Paper page - MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic
Evaluation of Audio General Intelligence
Please give a thumbs up to this comment if you found it helpful!
\n
If you want recommendations for any Paper on Hugging Face checkout this Space
\n
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
\n","updatedAt":"2025-08-21T01:37:11.348Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7116696238517761},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2508.13992","authors":[{"_id":"68a539856cf0bf898542ec92","user":{"_id":"65203f31d6dec04046139874","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65203f31d6dec04046139874/1LDWW4KblYeIDve9F_kUA.png","isPro":false,"fullname":"Sonal Kumar","user":"sonalkum","type":"user"},"name":"Sonal Kumar","status":"claimed_verified","statusLastChangedAt":"2025-08-28T08:56:41.879Z","hidden":false},{"_id":"68a539856cf0bf898542ec93","name":"Šimon Sedláček","hidden":false},{"_id":"68a539856cf0bf898542ec94","user":{"_id":"673663f604dd4f1afc771035","avatarUrl":"/avatars/37fe4c0f1e3b3da7d6c1e145780b4697.svg","isPro":false,"fullname":"Vaibhavi Lokegaonkar","user":"vlokegaonkar","type":"user"},"name":"Vaibhavi Lokegaonkar","status":"claimed_verified","statusLastChangedAt":"2026-02-06T19:00:41.344Z","hidden":false},{"_id":"68a539856cf0bf898542ec95","user":{"_id":"6329da8459950c1d279afea2","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1668763140756-6329da8459950c1d279afea2.jpeg","isPro":false,"fullname":"Fernando López Gavilánez","user":"ferugit","type":"user"},"name":"Fernando López","status":"claimed_verified","statusLastChangedAt":"2025-08-28T08:56:36.936Z","hidden":false},{"_id":"68a539856cf0bf898542ec96","name":"Wenyi Yu","hidden":false},{"_id":"68a539856cf0bf898542ec97","user":{"_id":"6316edb01cc81c5e95eb6100","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6316edb01cc81c5e95eb6100/r0L1kMMOUQENc1vpsbyxk.jpeg","isPro":false,"fullname":"Nishit Anand","user":"nishitanand","type":"user"},"name":"Nishit Anand","status":"claimed_verified","statusLastChangedAt":"2025-08-28T08:56:38.738Z","hidden":false},{"_id":"68a539856cf0bf898542ec98","name":"Hyeonggon Ryu","hidden":false},{"_id":"68a539856cf0bf898542ec99","name":"Lichang Chen","hidden":false},{"_id":"68a539856cf0bf898542ec9a","name":"Maxim Plička","hidden":false},{"_id":"68a539856cf0bf898542ec9b","name":"Miroslav Hlaváček","hidden":false},{"_id":"68a539856cf0bf898542ec9c","name":"William Fineas Ellingwood","hidden":false},{"_id":"68a539856cf0bf898542ec9d","name":"Sathvik Udupa","hidden":false},{"_id":"68a539856cf0bf898542ec9e","name":"Siyuan Hou","hidden":false},{"_id":"68a539856cf0bf898542ec9f","name":"Allison Ferner","hidden":false},{"_id":"68a539856cf0bf898542eca0","name":"Sara Barahona","hidden":false},{"_id":"68a539856cf0bf898542eca1","name":"Cecilia Bolaños","hidden":false},{"_id":"68a539856cf0bf898542eca2","name":"Satish Rahi","hidden":false},{"_id":"68a539856cf0bf898542eca3","name":"Laura Herrera-Alarcón","hidden":false},{"_id":"68a539856cf0bf898542eca4","name":"Satvik Dixit","hidden":false},{"_id":"68a539856cf0bf898542eca5","name":"Siddhi Patil","hidden":false},{"_id":"68a539856cf0bf898542eca6","name":"Soham Deshmukh","hidden":false},{"_id":"68a539856cf0bf898542eca7","user":{"_id":"63fc96a0b9db84750cec216f","avatarUrl":"/avatars/eb8cb2518260de4851f8f6aa8bc5508c.svg","isPro":false,"fullname":"Lasha Koroshinadze","user":"lashahub","type":"user"},"name":"Lasha Koroshinadze","status":"claimed_verified","statusLastChangedAt":"2026-01-28T11:34:25.515Z","hidden":false},{"_id":"68a539856cf0bf898542eca8","name":"Yao Liu","hidden":false},{"_id":"68a539856cf0bf898542eca9","name":"Leibny Paola Garcia Perera","hidden":false},{"_id":"68a539856cf0bf898542ecaa","name":"Eleni Zanou","hidden":false},{"_id":"68a539856cf0bf898542ecab","name":"Themos Stafylakis","hidden":false},{"_id":"68a539856cf0bf898542ecac","name":"Joon Son Chung","hidden":false},{"_id":"68a539856cf0bf898542ecad","name":"David Harwath","hidden":false},{"_id":"68a539856cf0bf898542ecae","name":"Chao Zhang","hidden":false},{"_id":"68a539856cf0bf898542ecaf","name":"Dinesh Manocha","hidden":false},{"_id":"68a539856cf0bf898542ecb0","name":"Alicia Lozano-Diez","hidden":false},{"_id":"68a539856cf0bf898542ecb1","name":"Santosh Kesiraju","hidden":false},{"_id":"68a539856cf0bf898542ecb2","user":{"_id":"62c9664eb34e600d7eaa4beb","avatarUrl":"/avatars/ca23ecdec2d31c99ecce97d9b180ae0c.svg","isPro":false,"fullname":"Ghosh","user":"Sreyan88","type":"user"},"name":"Sreyan Ghosh","status":"claimed_verified","statusLastChangedAt":"2025-08-20T08:51:15.060Z","hidden":false},{"_id":"68a539856cf0bf898542ecb3","user":{"_id":"65b8f0339b7250e205100190","avatarUrl":"/avatars/29df91154c5911a44ef503bb3580ac81.svg","isPro":false,"fullname":"Ramani Duraiswami","user":"RamaniD","type":"user"},"name":"Ramani Duraiswami","status":"claimed_verified","statusLastChangedAt":"2025-09-01T07:54:34.436Z","hidden":false}],"publishedAt":"2025-08-19T16:33:49.000Z","submittedOnDailyAt":"2025-08-20T01:34:00.837Z","title":"MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic\n Evaluation of Audio General Intelligence","submittedOnDailyBy":{"_id":"62c9664eb34e600d7eaa4beb","avatarUrl":"/avatars/ca23ecdec2d31c99ecce97d9b180ae0c.svg","isPro":false,"fullname":"Ghosh","user":"Sreyan88","type":"user"},"summary":"Audio comprehension-including speech, non-speech sounds, and music-is\nessential for achieving human-level intelligence. Consequently, AI agents must\ndemonstrate holistic audio understanding to qualify as generally intelligent.\nHowever, evaluating auditory intelligence comprehensively remains challenging.\nTo address this gap, we introduce MMAU-Pro, the most comprehensive and\nrigorously curated benchmark for assessing audio intelligence in AI systems.\nMMAU-Pro contains 5,305 instances, where each instance has one or more audios\npaired with human expert-generated question-answer pairs, spanning speech,\nsound, music, and their combinations. Unlike existing benchmarks, MMAU-Pro\nevaluates auditory intelligence across 49 unique skills and multiple complex\ndimensions, including long-form audio comprehension, spatial audio reasoning,\nmulti-audio understanding, among others. All questions are meticulously\ndesigned to require deliberate multi-hop reasoning, including both\nmultiple-choice and open-ended response formats. Importantly, audio data is\nsourced directly ``from the wild\" rather than from existing datasets with known\ndistributions. We evaluate 22 leading open-source and proprietary multimodal AI\nmodels, revealing significant limitations: even state-of-the-art models such as\nGemini 2.5 Flash and Audio Flamingo 3 achieve only 59.2% and 51.7% accuracy,\nrespectively, approaching random performance in multiple categories. Our\nextensive analysis highlights specific shortcomings and provides novel\ninsights, offering actionable perspectives for the community to enhance future\nAI systems' progression toward audio general intelligence. The benchmark and\ncode is available at https://sonalkum.github.io/mmau-pro.","upvotes":7,"discussionId":"68a539866cf0bf898542ecb4","projectPage":"https://sonalkum.github.io/mmau-pro","ai_summary":"MMAU-Pro is a comprehensive benchmark for evaluating audio intelligence in AI systems, assessing 49 unique skills across speech, sound, music, and their combinations, revealing significant limitations in current models.","ai_keywords":["MMAU-Pro","long-form audio comprehension","spatial audio reasoning","multi-audio understanding","multi-hop reasoning","multimodal AI models","Gemini 2.5 Flash","Audio Flamingo 3"]},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"65391169f186c8b4a893ff5e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65391169f186c8b4a893ff5e/P24mT-eGklClmc5hOJK_s.png","isPro":false,"fullname":"Šimon Sedláček","user":"pirxus","type":"user"},{"_id":"65b8f0339b7250e205100190","avatarUrl":"/avatars/29df91154c5911a44ef503bb3580ac81.svg","isPro":false,"fullname":"Ramani Duraiswami","user":"RamaniD","type":"user"},{"_id":"6601e0b1cab56b7c6a77badb","avatarUrl":"/avatars/0cd529c390e0d74225813efd03f3d189.svg","isPro":false,"fullname":"Satvik Dixit","user":"satvik-dixit","type":"user"},{"_id":"6316edb01cc81c5e95eb6100","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6316edb01cc81c5e95eb6100/r0L1kMMOUQENc1vpsbyxk.jpeg","isPro":false,"fullname":"Nishit Anand","user":"nishitanand","type":"user"},{"_id":"6329da8459950c1d279afea2","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1668763140756-6329da8459950c1d279afea2.jpeg","isPro":false,"fullname":"Fernando López Gavilánez","user":"ferugit","type":"user"},{"_id":"65203f31d6dec04046139874","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65203f31d6dec04046139874/1LDWW4KblYeIDve9F_kUA.png","isPro":false,"fullname":"Sonal Kumar","user":"sonalkum","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0}">
MMAU-Pro is a comprehensive benchmark for evaluating audio intelligence in AI systems, assessing 49 unique skills across speech, sound, music, and their combinations, revealing significant limitations in current models.
AI-generated summary
Audio comprehension-including speech, non-speech sounds, and music-is
essential for achieving human-level intelligence. Consequently, AI agents must
demonstrate holistic audio understanding to qualify as generally intelligent.
However, evaluating auditory intelligence comprehensively remains challenging.
To address this gap, we introduce MMAU-Pro, the most comprehensive and
rigorously curated benchmark for assessing audio intelligence in AI systems.
MMAU-Pro contains 5,305 instances, where each instance has one or more audios
paired with human expert-generated question-answer pairs, spanning speech,
sound, music, and their combinations. Unlike existing benchmarks, MMAU-Pro
evaluates auditory intelligence across 49 unique skills and multiple complex
dimensions, including long-form audio comprehension, spatial audio reasoning,
multi-audio understanding, among others. All questions are meticulously
designed to require deliberate multi-hop reasoning, including both
multiple-choice and open-ended response formats. Importantly, audio data is
sourced directly ``from the wild" rather than from existing datasets with known
distributions. We evaluate 22 leading open-source and proprietary multimodal AI
models, revealing significant limitations: even state-of-the-art models such as
Gemini 2.5 Flash and Audio Flamingo 3 achieve only 59.2% and 51.7% accuracy,
respectively, approaching random performance in multiple categories. Our
extensive analysis highlights specific shortcomings and provides novel
insights, offering actionable perspectives for the community to enhance future
AI systems' progression toward audio general intelligence. The benchmark and
code is available at https://sonalkum.github.io/mmau-pro.