https://github.com/baichuan-inc/Baichuan-Omni-1.5

\n","updatedAt":"2025-01-28T05:34:49.737Z","author":{"_id":"60f1abe7544c2adfd699860c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674929746905-60f1abe7544c2adfd699860c.jpeg","fullname":"AK","name":"akhaliq","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":9177,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7248987555503845},"editors":["akhaliq"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674929746905-60f1abe7544c2adfd699860c.jpeg"],"reactions":[],"isReport":false}},{"id":"6799859081dd5cb012c07d3a","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":317,"isUserFollowing":false},"createdAt":"2025-01-29T01:34:08.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction](https://huggingface.co/papers/2501.01957) (2025)\n* [Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition](https://huggingface.co/papers/2412.09501) (2024)\n* [Valley2: Exploring Multimodal Models with Scalable Vision-Language Design](https://huggingface.co/papers/2501.05901) (2025)\n* [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://huggingface.co/papers/2412.05271) (2024)\n* [OpenOmni: Large Language Models Pivot Zero-shot Omnimodal Alignment across Language with Real-time Self-Aware Emotional Speech Synthesis](https://huggingface.co/papers/2501.04561) (2025)\n* [From Specific-MLLM to Omni-MLLM: A Survey about the MLLMs alligned with Multi-Modality](https://huggingface.co/papers/2412.11694) (2024)\n* [Optimizing Vision-Language Interactions Through Decoder-Only Models](https://huggingface.co/papers/2412.10758) (2024)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2025-01-29T01:34:08.380Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":317,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7080268859863281},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2501.15368","authors":[{"_id":"67986c6822990ae89bb71fb9","user":{"_id":"6797cc0ff386b10d1609e3ff","avatarUrl":"/avatars/3ec1020e974ed01f60a46150501171da.svg","isPro":false,"fullname":"Yadong Li","user":"AdamLee1","type":"user"},"name":"Yadong Li","status":"admin_assigned","statusLastChangedAt":"2025-01-28T10:29:02.659Z","hidden":false},{"_id":"67986c6822990ae89bb71fba","name":"Jun Liu","hidden":false},{"_id":"67986c6822990ae89bb71fbb","name":"Tao Zhang","hidden":false},{"_id":"67986c6822990ae89bb71fbc","name":"Tao Zhang","hidden":false},{"_id":"67986c6822990ae89bb71fbd","name":"Song Chen","hidden":false},{"_id":"67986c6822990ae89bb71fbe","user":{"_id":"646c3ced3e2a7b06594bbaa4","avatarUrl":"/avatars/6e2d0e2f35e159a7832919a454583ab1.svg","isPro":false,"fullname":"李天鹏","user":"yuanshuai","type":"user"},"name":"Tianpeng Li","status":"claimed_verified","statusLastChangedAt":"2025-09-03T08:33:25.668Z","hidden":false},{"_id":"67986c6822990ae89bb71fbf","name":"Zehuan Li","hidden":false},{"_id":"67986c6822990ae89bb71fc0","name":"Lijun Liu","hidden":false},{"_id":"67986c6822990ae89bb71fc1","name":"Lingfeng Ming","hidden":false},{"_id":"67986c6822990ae89bb71fc2","name":"Guosheng Dong","hidden":false},{"_id":"67986c6822990ae89bb71fc3","name":"Da Pan","hidden":false},{"_id":"67986c6822990ae89bb71fc4","name":"Chong Li","hidden":false},{"_id":"67986c6822990ae89bb71fc5","user":{"_id":"65758b44769f3ee9bd44f237","avatarUrl":"/avatars/b1a231b43db901f1dc3d6ebb2df2984d.svg","isPro":false,"fullname":"fyb","user":"undobug","type":"user"},"name":"Yuanbo Fang","status":"claimed_verified","statusLastChangedAt":"2025-10-20T12:17:34.975Z","hidden":false},{"_id":"67986c6822990ae89bb71fc6","user":{"_id":"6455ec9bd808eebdefc4ceec","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6455ec9bd808eebdefc4ceec/AcwIUKFKjUgPpfJq07zQs.jpeg","isPro":false,"fullname":"Dongdong Kuang","user":"kingsley01","type":"user"},"name":"Dongdong Kuang","status":"claimed_verified","statusLastChangedAt":"2025-01-29T17:17:34.592Z","hidden":false},{"_id":"67986c6822990ae89bb71fc7","user":{"_id":"66af35946149eb45a6730a8f","avatarUrl":"/avatars/7f613f925b5d57798e03c0320661247e.svg","isPro":false,"fullname":"Mingrui Wang","user":"ruillm","type":"user"},"name":"Mingrui Wang","status":"claimed_verified","statusLastChangedAt":"2025-01-29T17:17:32.373Z","hidden":false},{"_id":"67986c6822990ae89bb71fc8","name":"Chenglin Zhu","hidden":false},{"_id":"67986c6822990ae89bb71fc9","name":"Youwei Zhang","hidden":false},{"_id":"67986c6822990ae89bb71fca","name":"Hongyu Guo","hidden":false},{"_id":"67986c6822990ae89bb71fcb","name":"Fengyu Zhang","hidden":false},{"_id":"67986c6822990ae89bb71fcc","user":{"_id":"65e71ef39cf349af2940b317","avatarUrl":"/avatars/fc1cd8d3510946fc947d67b16b51834b.svg","isPro":false,"fullname":"Yuran Wang","user":"Ryann829","type":"user"},"name":"Yuran Wang","status":"claimed_verified","statusLastChangedAt":"2025-01-28T13:52:53.610Z","hidden":false},{"_id":"67986c6822990ae89bb71fcd","name":"Bowen Ding","hidden":false},{"_id":"67986c6822990ae89bb71fce","user":{"_id":"665eccf5ffd59344a22533a8","avatarUrl":"/avatars/2ae2710753ce34a04937384bc6dddf70.svg","isPro":false,"fullname":"Wei Song (SII)","user":"Songweii","type":"user"},"name":"Wei Song","status":"claimed_verified","statusLastChangedAt":"2025-03-24T08:55:52.565Z","hidden":false},{"_id":"67986c6822990ae89bb71fcf","name":"Xu Li","hidden":false},{"_id":"67986c6822990ae89bb71fd0","name":"Yuqi Huo","hidden":false},{"_id":"67986c6822990ae89bb71fd1","name":"Zheng Liang","hidden":false},{"_id":"67986c6822990ae89bb71fd2","name":"Shusen Zhang","hidden":false},{"_id":"67986c6822990ae89bb71fd3","name":"Xin Wu","hidden":false},{"_id":"67986c6822990ae89bb71fd4","name":"Shuai Zhao","hidden":false},{"_id":"67986c6822990ae89bb71fd5","name":"Linchu Xiong","hidden":false},{"_id":"67986c6822990ae89bb71fd6","name":"Yozhen Wu","hidden":false},{"_id":"67986c6822990ae89bb71fd7","name":"Jiahui Ye","hidden":false},{"_id":"67986c6822990ae89bb71fd8","name":"Wenhao Lu","hidden":false},{"_id":"67986c6822990ae89bb71fd9","name":"Bowen Li","hidden":false},{"_id":"67986c6822990ae89bb71fda","name":"Yan Zhang","hidden":false},{"_id":"67986c6822990ae89bb71fdb","name":"Yaqi Zhou","hidden":false},{"_id":"67986c6822990ae89bb71fdc","name":"Xin Chen","hidden":false},{"_id":"67986c6822990ae89bb71fdd","name":"Lei Su","hidden":false},{"_id":"67986c6822990ae89bb71fde","name":"Hongda Zhang","hidden":false},{"_id":"67986c6822990ae89bb71fdf","name":"Fuzhong Chen","hidden":false},{"_id":"67986c6822990ae89bb71fe0","name":"Xuezhen Dong","hidden":false},{"_id":"67986c6822990ae89bb71fe1","name":"Na Nie","hidden":false},{"_id":"67986c6822990ae89bb71fe2","name":"Zhiying Wu","hidden":false},{"_id":"67986c6822990ae89bb71fe3","name":"Bin Xiao","hidden":false},{"_id":"67986c6822990ae89bb71fe4","name":"Ting Li","hidden":false},{"_id":"67986c6822990ae89bb71fe5","name":"Shunya Dang","hidden":false},{"_id":"67986c6822990ae89bb71fe6","name":"Ping Zhang","hidden":false},{"_id":"67986c6822990ae89bb71fe7","name":"Yijia Sun","hidden":false},{"_id":"67986c6822990ae89bb71fe8","name":"Jincheng Wu","hidden":false},{"_id":"67986c6822990ae89bb71fe9","name":"Jinjie Yang","hidden":false},{"_id":"67986c6822990ae89bb71fea","name":"Xionghai Lin","hidden":false},{"_id":"67986c6822990ae89bb71feb","name":"Zhi Ma","hidden":false},{"_id":"67986c6822990ae89bb71fec","name":"Kegeng Wu","hidden":false},{"_id":"67986c6822990ae89bb71fed","name":"Jia li","hidden":false},{"_id":"67986c6822990ae89bb71fee","name":"Aiyuan Yang","hidden":false},{"_id":"67986c6822990ae89bb71fef","name":"Hui Liu","hidden":false},{"_id":"67986c6822990ae89bb71ff0","name":"Jianqiang Zhang","hidden":false},{"_id":"67986c6822990ae89bb71ff1","name":"Xiaoxi Chen","hidden":false},{"_id":"67986c6822990ae89bb71ff2","name":"Guangwei Ai","hidden":false},{"_id":"67986c6822990ae89bb71ff3","name":"Wentao Zhang","hidden":false},{"_id":"67986c6822990ae89bb71ff4","name":"Yicong Chen","hidden":false},{"_id":"67986c6822990ae89bb71ff5","name":"Xiaoqin Huang","hidden":false},{"_id":"67986c6822990ae89bb71ff6","name":"Kun Li","hidden":false},{"_id":"67986c6822990ae89bb71ff7","name":"Wenjing Luo","hidden":false},{"_id":"67986c6822990ae89bb71ff8","name":"Yifei Duan","hidden":false},{"_id":"67986c6822990ae89bb71ff9","name":"Lingling Zhu","hidden":false},{"_id":"67986c6822990ae89bb71ffa","name":"Ran Xiao","hidden":false},{"_id":"67986c6822990ae89bb71ffb","name":"Zhe Su","hidden":false},{"_id":"67986c6822990ae89bb71ffc","name":"Jiani Pu","hidden":false},{"_id":"67986c6822990ae89bb71ffd","name":"Dian Wang","hidden":false},{"_id":"67986c6822990ae89bb71ffe","name":"Xu Jia","hidden":false},{"_id":"67986c6822990ae89bb71fff","name":"Tianyu Zhang","hidden":false},{"_id":"67986c6822990ae89bb72000","name":"Mengyu Ai","hidden":false},{"_id":"67986c6822990ae89bb72001","name":"Mang Wang","hidden":false},{"_id":"67986c6822990ae89bb72002","name":"Yujing Qiao","hidden":false},{"_id":"67986c6822990ae89bb72003","name":"Lei Zhang","hidden":false},{"_id":"67986c6822990ae89bb72004","name":"Yanjun Shen","hidden":false},{"_id":"67986c6822990ae89bb72005","user":{"_id":"641c45c921964f8f6d451d16","avatarUrl":"/avatars/da06cc603f8f9ee46ddb7dc72aae5bec.svg","isPro":false,"fullname":"FanYang","user":"fairyang","type":"user"},"name":"Fan Yang","status":"claimed_verified","statusLastChangedAt":"2025-09-03T08:33:23.951Z","hidden":false},{"_id":"67986c6822990ae89bb72006","name":"Miao Zhen","hidden":false},{"_id":"67986c6822990ae89bb72007","user":{"_id":"658670184f349f95cf7d2252","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/658670184f349f95cf7d2252/MfYwxDS1w2kIvav2GvE_U.jpeg","isPro":false,"fullname":"Jie","user":"Jayok6","type":"user"},"name":"Yijie Zhou","status":"claimed_verified","statusLastChangedAt":"2025-09-03T08:33:18.065Z","hidden":false},{"_id":"67986c6822990ae89bb72008","user":{"_id":"66254042ea4f4ed066a77a1e","avatarUrl":"/avatars/c749b77417431bb364f6fa2189eabaa2.svg","isPro":false,"fullname":"Mingyang Chen","user":"anselcmy","type":"user"},"name":"Mingyang Chen","status":"claimed_verified","statusLastChangedAt":"2025-09-03T08:33:27.715Z","hidden":false},{"_id":"67986c6822990ae89bb72009","user":{"_id":"6464dd5234acce85aea186c7","avatarUrl":"/avatars/3428029b0ae5f12885092c7aea588065.svg","isPro":false,"fullname":"lifei","user":"lifei926926","type":"user"},"name":"Fei Li","status":"claimed_verified","statusLastChangedAt":"2026-01-12T14:17:05.902Z","hidden":false},{"_id":"67986c6822990ae89bb7200a","name":"Chenzheng Zhu","hidden":false},{"_id":"67986c6822990ae89bb7200b","name":"Keer Lu","hidden":false},{"_id":"67986c6822990ae89bb7200c","name":"Yaqi Zhao","hidden":false},{"_id":"67986c6822990ae89bb7200d","name":"Hao Liang","hidden":false},{"_id":"67986c6822990ae89bb7200e","name":"Youquan Li","hidden":false},{"_id":"67986c6822990ae89bb7200f","name":"Yanzhao Qin","hidden":false},{"_id":"67986c6822990ae89bb72010","name":"Linzhuang Sun","hidden":false},{"_id":"67986c6822990ae89bb72011","name":"Jianhua Xu","hidden":false},{"_id":"67986c6822990ae89bb72012","user":{"_id":"6436bb0dd58a5ea528c55acb","avatarUrl":"/avatars/df17b66780e14e07bbe4625f068a94ad.svg","isPro":false,"fullname":"Alvin Sun","user":"AlvinSunYooo","type":"user"},"name":"Haoze Sun","status":"claimed_verified","statusLastChangedAt":"2025-02-11T10:03:25.944Z","hidden":false},{"_id":"67986c6822990ae89bb72013","user":{"_id":"6415947858a690df103af49f","avatarUrl":"/avatars/38aec23b869833bceb25b9250809b419.svg","isPro":false,"fullname":"lma","user":"lin5547","type":"user"},"name":"Mingan Lin","status":"claimed_verified","statusLastChangedAt":"2025-02-12T09:17:07.915Z","hidden":false},{"_id":"67986c6822990ae89bb72014","name":"Zenan Zhou","hidden":false},{"_id":"67986c6822990ae89bb72015","user":{"_id":"6501587887b370a56ad2608e","avatarUrl":"/avatars/6779baaa8ed9032de55a2f78e1f52e20.svg","isPro":false,"fullname":"Wei-Peng Chen","user":"whenfra","type":"user"},"name":"Weipeng Chen","status":"admin_assigned","statusLastChangedAt":"2025-01-28T13:54:48.451Z","hidden":false}],"publishedAt":"2025-01-26T02:19:03.000Z","submittedOnDailyAt":"2025-01-28T03:04:49.721Z","title":"Baichuan-Omni-1.5 Technical Report","submittedOnDailyBy":{"_id":"60f1abe7544c2adfd699860c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674929746905-60f1abe7544c2adfd699860c.jpeg","isPro":false,"fullname":"AK","user":"akhaliq","type":"user"},"summary":"We introduce Baichuan-Omni-1.5, an omni-modal model that not only has\nomni-modal understanding capabilities but also provides end-to-end audio\ngeneration capabilities. To achieve fluent and high-quality interaction across\nmodalities without compromising the capabilities of any modality, we\nprioritized optimizing three key aspects. First, we establish a comprehensive\ndata cleaning and synthesis pipeline for multimodal data, obtaining about 500B\nhigh-quality data (text, audio, and vision). Second, an audio-tokenizer\n(Baichuan-Audio-Tokenizer) has been designed to capture both semantic and\nacoustic information from audio, enabling seamless integration and enhanced\ncompatibility with MLLM. Lastly, we designed a multi-stage training strategy\nthat progressively integrates multimodal alignment and multitask fine-tuning,\nensuring effective synergy across all modalities. Baichuan-Omni-1.5 leads\ncontemporary models (including GPT4o-mini and MiniCPM-o 2.6) in terms of\ncomprehensive omni-modal capabilities. Notably, it achieves results comparable\nto leading models such as Qwen2-VL-72B across various multimodal medical\nbenchmarks.","upvotes":60,"discussionId":"67986c6b22990ae89bb720aa","githubRepo":"https://github.com/baichuan-inc/Baichuan-Omni-1.5","githubRepoAddedBy":"auto","ai_summary":"Baichuan-Omni-1.5 is an omni-modal model with end-to-end audio generation, featuring a comprehensive data pipeline, audio-tokenizer, and multi-stage training strategy for superior performance across multimodal tasks.","ai_keywords":["omni-modal model","audio-tokenizer","multimodal data","semantic information","acoustic information","MLLM","multimodal alignment","multitask fine-tuning","omni-modal capabilities","Qwen2-VL-72B","multimodal medical benchmarks"],"githubStars":185},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"665eccf5ffd59344a22533a8","avatarUrl":"/avatars/2ae2710753ce34a04937384bc6dddf70.svg","isPro":false,"fullname":"Wei Song (SII)","user":"Songweii","type":"user"},{"_id":"6415947858a690df103af49f","avatarUrl":"/avatars/38aec23b869833bceb25b9250809b419.svg","isPro":false,"fullname":"lma","user":"lin5547","type":"user"},{"_id":"668d4e50ed63008dfaa78304","avatarUrl":"/avatars/80854a3c6b4b7c70cd46694d4cf7296a.svg","isPro":false,"fullname":"Zenan Zhou","user":"Zenan11","type":"user"},{"_id":"6486bf03373f79a5290b519c","avatarUrl":"/avatars/8700c1d9621ad4026da8a56badcd51be.svg","isPro":false,"fullname":"adonlee","user":"adonlee","type":"user"},{"_id":"65decc75beffeb39ba679eba","avatarUrl":"/avatars/735b678bd5863a0c1b1bdd3bbf8858fa.svg","isPro":true,"fullname":"r","user":"oceansweep","type":"user"},{"_id":"6217599529500f41901123f8","avatarUrl":"/avatars/8a0fe54e53fe6527c70a78598a0cd941.svg","isPro":false,"fullname":"Hao Liang","user":"lhbit20010120","type":"user"},{"_id":"66713617f698ab519b66bfba","avatarUrl":"/avatars/c8dfc68e7bd8e4888453773570263df4.svg","isPro":false,"fullname":"enhui ma ","user":"estrellla","type":"user"},{"_id":"6436bb0dd58a5ea528c55acb","avatarUrl":"/avatars/df17b66780e14e07bbe4625f068a94ad.svg","isPro":false,"fullname":"Alvin Sun","user":"AlvinSunYooo","type":"user"},{"_id":"64636db1c615cbc1244749cf","avatarUrl":"/avatars/48a08d6bb8b676dd4a12789fcc20143b.svg","isPro":false,"fullname":"liu","user":"mrlijun","type":"user"},{"_id":"64127b9fac08ffb707937231","avatarUrl":"/avatars/33d406b8d1f319af5a4e3c2dc59ea7f2.svg","isPro":false,"fullname":"Ding Bowen","user":"Daniel21Ding","type":"user"},{"_id":"64a84de2eb47b3552285ef74","avatarUrl":"/avatars/114e0cc393d0aea9680f3af6d84d6f46.svg","isPro":false,"fullname":"Eni Grand","user":"Enigrand","type":"user"},{"_id":"65758b44769f3ee9bd44f237","avatarUrl":"/avatars/b1a231b43db901f1dc3d6ebb2df2984d.svg","isPro":false,"fullname":"fyb","user":"undobug","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":2}">

Papers

arxiv:2501.15368

Baichuan-Omni-1.5 Technical Report

Published on Jan 26, 2025

· Submitted by

AK on Jan 28, 2025

#2 Paper of the day

Upvote

Authors:

Yadong Li ,

Tianpeng Li ,

Yuanbo Fang ,

Dongdong Kuang ,

Mingrui Wang ,

Yuran Wang ,

Wei Song

Abstract

Baichuan-Omni-1.5 is an omni-modal model with end-to-end audio generation, featuring a comprehensive data pipeline, audio-tokenizer, and multi-stage training strategy for superior performance across multimodal tasks.

AI-generated summary

We introduce Baichuan-Omni-1.5, an omni-modal model that not only has omni-modal understanding capabilities but also provides end-to-end audio generation capabilities. To achieve fluent and high-quality interaction across modalities without compromising the capabilities of any modality, we prioritized optimizing three key aspects. First, we establish a comprehensive data cleaning and synthesis pipeline for multimodal data, obtaining about 500B high-quality data (text, audio, and vision). Second, an audio-tokenizer (Baichuan-Audio-Tokenizer) has been designed to capture both semantic and acoustic information from audio, enabling seamless integration and enhanced compatibility with MLLM. Lastly, we designed a multi-stage training strategy that progressively integrates multimodal alignment and multitask fine-tuning, ensuring effective synergy across all modalities. Baichuan-Omni-1.5 leads contemporary models (including GPT4o-mini and MiniCPM-o 2.6) in terms of comprehensive omni-modal capabilities. Notably, it achieves results comparable to leading models such as Qwen2-VL-72B across various multimodal medical benchmarks.