Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Boosting Large Language Model for Speech Synthesis: An Empirical Study (2023)
SpeechComposer: Unifying Multiple Speech Tasks with Prompt Composition (2024)
Audiobox: Unified Audio Generation with Natural Language Prompts (2023)
ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024)
Enhancing the Stability of LLM-based Speech Generation Systems through Self-Supervised Representations (2024)

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2024-02-10T01:20:43.512Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7660372853279114},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}},{"id":"671430d7485b75297d626d32","author":{"_id":"656e85572c331f3e079a2df0","avatarUrl":"/avatars/c0f2b142a58c6c46dbb6b9bed0aa7ff0.svg","fullname":"mismo","name":"Mimismo","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false},"createdAt":"2024-10-19T22:21:11.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Hello","html":"

Hello

\n","updatedAt":"2024-10-19T22:21:11.334Z","author":{"_id":"656e85572c331f3e079a2df0","avatarUrl":"/avatars/c0f2b142a58c6c46dbb6b9bed0aa7ff0.svg","fullname":"mismo","name":"Mimismo","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.33811527490615845},"editors":["Mimismo"],"editorAvatarUrls":["/avatars/c0f2b142a58c6c46dbb6b9bed0aa7ff0.svg"],"reactions":[],"isReport":false},"replies":[{"id":"695dcd2b1731e22c71c05260","author":{"_id":"68aea2b3867504c32e0c3382","avatarUrl":"/avatars/eadda83362a149e1dabeb3a0ea812357.svg","fullname":"J S","name":"songj-renesas","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false},"createdAt":"2026-01-07T03:04:11.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Hi","html":"

\n","updatedAt":"2026-01-07T03:04:11.710Z","author":{"_id":"68aea2b3867504c32e0c3382","avatarUrl":"/avatars/eadda83362a149e1dabeb3a0ea812357.svg","fullname":"J S","name":"songj-renesas","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"de","probability":0.5160623788833618},"editors":["songj-renesas"],"editorAvatarUrls":["/avatars/eadda83362a149e1dabeb3a0ea812357.svg"],"reactions":[],"isReport":false,"parentCommentId":"671430d7485b75297d626d32"}}]}],"primaryEmailConfirmed":false,"paper":{"id":"2402.05755","authors":[{"_id":"65c5984702262fe3b4fb9c57","user":{"_id":"62e7b942c9e83245ea5e14ec","avatarUrl":"/avatars/3223dae7ca6b80e6dc780c1b002b1524.svg","isPro":false,"fullname":"Tu Anh Nguyen","user":"ntuanh","type":"user"},"name":"Tu Anh Nguyen","status":"admin_assigned","statusLastChangedAt":"2024-02-09T10:17:00.027Z","hidden":false},{"_id":"65c5984702262fe3b4fb9c58","user":{"_id":"5e6b7153d4cd9779932a7600","avatarUrl":"/avatars/c1304f100758c5e2a54fd4d47f638841.svg","isPro":false,"fullname":"Benjamin Muller","user":"benjamin-mlr","type":"user"},"name":"Benjamin Muller","status":"admin_assigned","statusLastChangedAt":"2024-02-09T10:17:06.496Z","hidden":false},{"_id":"65c5984702262fe3b4fb9c59","user":{"_id":"65b3c0ee05c25412bb6aab2c","avatarUrl":"/avatars/2b3be41c899a29e2a0954dd7cd6cd5ab.svg","isPro":false,"fullname":"Bokai Yu","user":"bokaiyu","type":"user"},"name":"Bokai Yu","status":"admin_assigned","statusLastChangedAt":"2024-02-09T10:17:26.496Z","hidden":false},{"_id":"65c5984702262fe3b4fb9c5a","name":"Marta R. Costa-jussa","hidden":false},{"_id":"65c5984702262fe3b4fb9c5b","user":{"_id":"64e243dff8934329dd72098f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64e243dff8934329dd72098f/JbB4zvlendHSRXjx_iC0j.jpeg","isPro":false,"fullname":"Maha Elbayad","user":"elbayadm","type":"user"},"name":"Maha Elbayad","status":"admin_assigned","statusLastChangedAt":"2024-02-09T10:17:42.732Z","hidden":false},{"_id":"65c5984702262fe3b4fb9c5c","user":{"_id":"62e190f3926f4892a4c8099e","avatarUrl":"/avatars/2168f769c526362f9ed35473f4b30047.svg","isPro":false,"fullname":"Sravya Popuri","user":"sravyapopuri388","type":"user"},"name":"Sravya Popuri","status":"admin_assigned","statusLastChangedAt":"2024-02-09T10:17:48.409Z","hidden":false},{"_id":"65c5984702262fe3b4fb9c5d","user":{"_id":"6331a09fe092098b57bc1d83","avatarUrl":"/avatars/cdbc4be17c160b5e1f957b6c1ae29e22.svg","isPro":false,"fullname":"Paul-Ambroise Duquenne","user":"padqn","type":"user"},"name":"Paul-Ambroise Duquenne","status":"admin_assigned","statusLastChangedAt":"2024-02-09T10:17:54.553Z","hidden":false},{"_id":"65c5984702262fe3b4fb9c5e","user":{"_id":"64d25f2035577e62a9df31a8","avatarUrl":"/avatars/fdaeb527f084d21e35de5a625c3629d6.svg","isPro":false,"fullname":"robin algayres","user":"robinalgayres","type":"user"},"name":"Robin Algayres","status":"admin_assigned","statusLastChangedAt":"2024-02-09T10:18:00.654Z","hidden":false},{"_id":"65c5984702262fe3b4fb9c5f","user":{"_id":"64dcfecb249785efad9a514d","avatarUrl":"/avatars/77de68494021f298bf2561b2ea9ffef8.svg","isPro":false,"fullname":"Ruslan Mavlyutov","user":"mavlyutovrus","type":"user"},"name":"Ruslan Mavlyutov","status":"admin_assigned","statusLastChangedAt":"2024-02-09T10:18:06.330Z","hidden":false},{"_id":"65c5984702262fe3b4fb9c60","user":{"_id":"62a8fa984d933c74bf410c16","avatarUrl":"/avatars/73519deba3176be9c23d49f749aee5da.svg","isPro":false,"fullname":"Itai Gat","user":"itaigat","type":"user"},"name":"Itai Gat","status":"admin_assigned","statusLastChangedAt":"2024-02-09T10:18:12.797Z","hidden":false},{"_id":"65c5984702262fe3b4fb9c61","user":{"_id":"630eac7931970d1cd4fbacf2","avatarUrl":"/avatars/b7ccbddfa745db854dc342be1327cd53.svg","isPro":false,"fullname":"Gabriel Synnaeve","user":"gsynnaeve","type":"user"},"name":"Gabriel Synnaeve","status":"admin_assigned","statusLastChangedAt":"2024-02-09T10:18:19.465Z","hidden":false},{"_id":"65c5984702262fe3b4fb9c62","user":{"_id":"634deff6be5a827d4874546a","avatarUrl":"/avatars/05e0ad93d9c8c7e572c6f73feb649aeb.svg","isPro":false,"fullname":"Juan Pino","user":"juanpino","type":"user"},"name":"Juan Pino","status":"admin_assigned","statusLastChangedAt":"2024-02-09T10:18:25.364Z","hidden":false},{"_id":"65c5984702262fe3b4fb9c63","name":"Benoit Sagot","hidden":false},{"_id":"65c5984702262fe3b4fb9c64","user":{"_id":"63317d2118711776b4663c3a","avatarUrl":"/avatars/7dedd1934c1000b6f81a2a37ec348347.svg","isPro":false,"fullname":"Emmanuel Dupoux","user":"edupoux","type":"user"},"name":"Emmanuel Dupoux","status":"admin_assigned","statusLastChangedAt":"2024-02-09T10:18:34.382Z","hidden":false}],"publishedAt":"2024-02-08T15:39:32.000Z","submittedOnDailyAt":"2024-02-09T00:43:11.866Z","title":"SpiRit-LM: Interleaved Spoken and Written Language Model","submittedOnDailyBy":{"_id":"60f1abe7544c2adfd699860c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674929746905-60f1abe7544c2adfd699860c.jpeg","isPro":false,"fullname":"AK","user":"akhaliq","type":"user"},"summary":"We introduce SPIRIT-LM, a foundation multimodal language model that freely\nmixes text and speech. Our model is based on a pretrained text language model\nthat we extend to the speech modality by continuously training it on text and\nspeech units. Speech and text sequences are concatenated as a single set of\ntokens, and trained with a word-level interleaving method using a small\nautomatically-curated speech-text parallel corpus. SPIRIT-LM comes in two\nversions: a BASE version that uses speech semantic units and an EXPRESSIVE\nversion that models expressivity using pitch and style units in addition to the\nsemantic units. For both versions, the text is encoded with subword BPE tokens.\nThe resulting model displays both the semantic abilities of text models and the\nexpressive abilities of speech models. Additionally, we demonstrate that\nSPIRIT-LM is able to learn new tasks in a few-shot fashion across modalities\n(i.e. ASR, TTS, Speech Classification).","upvotes":15,"discussionId":"65c5984702262fe3b4fb9c82","githubRepo":"https://github.com/facebookresearch/spiritlm","githubRepoAddedBy":"auto","ai_summary":"A foundation multimodal language model combines text and speech using speech and text units, supporting semantic and expressive capabilities and few-shot learning across tasks.","ai_keywords":["multimodal language model","speech semantic units","subword BPE tokens","expressivity","pitch","style units","few-shot learning","ASR","TTS","Speech Classification"],"githubStars":927},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6538119803519fddb4a17e10","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6538119803519fddb4a17e10/ffJMkdx-rM7VvLTCM6ri_.jpeg","isPro":false,"fullname":"samusenps","user":"samusenps","type":"user"},{"_id":"5fcc1929563427b03e9af259","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1620642852901-5fcc1929563427b03e9af259.jpeg","isPro":false,"fullname":"David Adelani","user":"Davlan","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"62e190f3926f4892a4c8099e","avatarUrl":"/avatars/2168f769c526362f9ed35473f4b30047.svg","isPro":false,"fullname":"Sravya Popuri","user":"sravyapopuri388","type":"user"},{"_id":"65c799a127736b5b8692af0d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/UfLffJms7n9QKigV6in5P.jpeg","isPro":false,"fullname":"Nina Gaz ","user":"Nina69","type":"user"},{"_id":"65b3c0ee05c25412bb6aab2c","avatarUrl":"/avatars/2b3be41c899a29e2a0954dd7cd6cd5ab.svg","isPro":false,"fullname":"Bokai Yu","user":"bokaiyu","type":"user"},{"_id":"6186ddf6a7717cb375090c01","avatarUrl":"/avatars/716b6a7d1094c8036b2a8a7b9063e8aa.svg","isPro":true,"fullname":"Julien BLANCHON","user":"blanchon","type":"user"},{"_id":"63732ebbbd81fae2b3aaf3fb","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1669551186189-63732ebbbd81fae2b3aaf3fb.jpeg","isPro":false,"fullname":"Knut Jägersberg","user":"KnutJaegersberg","type":"user"},{"_id":"6093a02dc4a92d63a91c5236","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6093a02dc4a92d63a91c5236/yUte6V0FU0BvVFAbON-9n.jpeg","isPro":true,"fullname":"Diwank Tomer","user":"diwank","type":"user"},{"_id":"66dbf4eda4bb9c25bacb61f4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66dbf4eda4bb9c25bacb61f4/SuZXBaUNCOvemU2rZkrFY.jpeg","isPro":false,"fullname":"Rin Intachuen","user":"RinRin32","type":"user"},{"_id":"65831c22489f3b53a02097c5","avatarUrl":"/avatars/6bbf39c6de16123c966511ddc34b53f1.svg","isPro":false,"fullname":"Nikolay Rodionov","user":"eldodi","type":"user"},{"_id":"631997c9307cb811990314c1","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/631997c9307cb811990314c1/5YkjJpf_c9iearMTOsToK.png","isPro":false,"fullname":"Hazem Abdelazim","user":"Hazem-Abdelazim","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0}">

Papers

arxiv:2402.05755

SpiRit-LM: Interleaved Spoken and Written Language Model

Published on Feb 8, 2024

· Submitted by

AK on Feb 9, 2024

Upvote

Authors:

Tu Anh Nguyen ,

Benjamin Muller ,

Bokai Yu ,

Maha Elbayad ,

Sravya Popuri ,

Paul-Ambroise Duquenne ,

Robin Algayres ,

Ruslan Mavlyutov ,

Itai Gat ,

Gabriel Synnaeve ,

Juan Pino ,

Emmanuel Dupoux

Abstract

A foundation multimodal language model combines text and speech using speech and text units, supporting semantic and expressive capabilities and few-shot learning across tasks.

AI-generated summary

We introduce SPIRIT-LM, a foundation multimodal language model that freely mixes text and speech. Our model is based on a pretrained text language model that we extend to the speech modality by continuously training it on text and speech units. Speech and text sequences are concatenated as a single set of tokens, and trained with a word-level interleaving method using a small automatically-curated speech-text parallel corpus. SPIRIT-LM comes in two versions: a BASE version that uses speech semantic units and an EXPRESSIVE version that models expressivity using pitch and style units in addition to the semantic units. For both versions, the text is encoded with subword BPE tokens. The resulting model displays both the semantic abilities of text models and the expressive abilities of speech models. Additionally, we demonstrate that SPIRIT-LM is able to learn new tasks in a few-shot fashion across modalities (i.e. ASR, TTS, Speech Classification).