Question for the authors, how does this technique compare to Discrete Diffusion with Planned Denoising (https://arxiv.org/abs/2410.06264)?
\n","updatedAt":"2025-02-17T05:27:49.444Z","author":{"_id":"63b6f2e752c02ae8acbaa4d8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1672934038280-noauth.jpeg","fullname":"Habibullah Akbar","name":"ChavyvAkvar","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":19,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7894180417060852},"editors":["ChavyvAkvar"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1672934038280-noauth.jpeg"],"reactions":[],"isReport":false},"replies":[{"id":"67b3510d7a49eaea085545fb","author":{"_id":"640dd9a5fdeaae13908208a7","avatarUrl":"/avatars/61f8f1d5f6ef1c4af1f47285e9cc0217.svg","fullname":"nieshen","name":"nieshen","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":15,"isUserFollowing":false},"createdAt":"2025-02-17T15:09:01.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Thank you for your question! The paper you mentioned, DDPO, requires training both the planner and the denoiser simultaneously, while LLaDA only needs to train a denoiser. This leads to differences in both the training and sampling processes of DDPO and LLaDA.","html":"Thank you for your question! The paper you mentioned, DDPO, requires training both the planner and the denoiser simultaneously, while LLaDA only needs to train a denoiser. This leads to differences in both the training and sampling processes of DDPO and LLaDA.
\n","updatedAt":"2025-02-17T15:09:01.669Z","author":{"_id":"640dd9a5fdeaae13908208a7","avatarUrl":"/avatars/61f8f1d5f6ef1c4af1f47285e9cc0217.svg","fullname":"nieshen","name":"nieshen","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":15,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9405416250228882},"editors":["nieshen"],"editorAvatarUrls":["/avatars/61f8f1d5f6ef1c4af1f47285e9cc0217.svg"],"reactions":[{"reaction":"👍","users":["mscheong","phpeteur","maverick84","pengxiang"],"count":4}],"isReport":false,"parentCommentId":"67b2c8d57b65819c9ee72404"}}]},{"id":"67b35f5086e153dc10786119","author":{"_id":"65e01fd5574e5aa0e9c9d4f9","avatarUrl":"/avatars/8a066fc283f208001ce34e9c58673eb3.svg","fullname":"SPARSH TEWATIA","name":"sparsh35","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false},"createdAt":"2025-02-17T16:09:52.000Z","type":"comment","data":{"edited":true,"hidden":false,"latest":{"raw":"Will you release the pre training dataset in future as it will allow to reproduce and maybe help the community to train different ablations for comparisons ? Anyway thanks for wonderful work 🥳","html":"Will you release the pre training dataset in future as it will allow to reproduce and maybe help the community to train different ablations for comparisons ? Anyway thanks for wonderful work 🥳
\n","updatedAt":"2025-02-17T16:11:49.345Z","author":{"_id":"65e01fd5574e5aa0e9c9d4f9","avatarUrl":"/avatars/8a066fc283f208001ce34e9c58673eb3.svg","fullname":"SPARSH TEWATIA","name":"sparsh35","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.9776546359062195},"editors":["sparsh35"],"editorAvatarUrls":["/avatars/8a066fc283f208001ce34e9c58673eb3.svg"],"reactions":[],"isReport":false},"replies":[{"id":"67b3637335cb7d30b46ce343","author":{"_id":"640dd9a5fdeaae13908208a7","avatarUrl":"/avatars/61f8f1d5f6ef1c4af1f47285e9cc0217.svg","fullname":"nieshen","name":"nieshen","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":15,"isUserFollowing":false},"createdAt":"2025-02-17T16:27:31.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Thank you for your attention. We have no plan to open-source the pre-training data of LLaDA. We have previously open-sourced the data, training code, and checkpoint for a work (https://arxiv.org/abs/2410.18514) related to masked diffusion models. However, in this work, the number of model parameters is smaller (just 1B), and the data quality is also relatively poor.","html":"Thank you for your attention. We have no plan to open-source the pre-training data of LLaDA. We have previously open-sourced the data, training code, and checkpoint for a work (https://arxiv.org/abs/2410.18514) related to masked diffusion models. However, in this work, the number of model parameters is smaller (just 1B), and the data quality is also relatively poor.
\n","updatedAt":"2025-02-17T16:27:31.035Z","author":{"_id":"640dd9a5fdeaae13908208a7","avatarUrl":"/avatars/61f8f1d5f6ef1c4af1f47285e9cc0217.svg","fullname":"nieshen","name":"nieshen","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":15,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9360384941101074},"editors":["nieshen"],"editorAvatarUrls":["/avatars/61f8f1d5f6ef1c4af1f47285e9cc0217.svg"],"reactions":[{"reaction":"🤝","users":["sparsh35"],"count":1}],"isReport":false,"parentCommentId":"67b35f5086e153dc10786119"}},{"id":"67b3651d7ab6babf0c0f779b","author":{"_id":"65e01fd5574e5aa0e9c9d4f9","avatarUrl":"/avatars/8a066fc283f208001ce34e9c58673eb3.svg","fullname":"SPARSH TEWATIA","name":"sparsh35","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false},"createdAt":"2025-02-17T16:34:37.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Even a blog related to constructing the dataset as I think this is the major factor for improvement, and is it all from mix of available open source datasets.\nThough I like the comparision you have done in comparing transformer and diffusion model though at 1B scale, it shows the similar result.","html":"Even a blog related to constructing the dataset as I think this is the major factor for improvement, and is it all from mix of available open source datasets.
Though I like the comparision you have done in comparing transformer and diffusion model though at 1B scale, it shows the similar result.
We sincerely apologize. Due to various reasons, we are unable to open - source the training data of LLaDA, similar to the current leading open - source LLMs such as LLaMA, Qwen, and DeepSeek. We have a plan to open - source the model weights and have announced the schedule on the GitHub homepage. We didn't design data specifically for LLaDA. The data we used was prepared for training autoregressive models, and LLaDA directly made use of this data.
\n","updatedAt":"2025-02-17T16:41:45.952Z","author":{"_id":"640dd9a5fdeaae13908208a7","avatarUrl":"/avatars/61f8f1d5f6ef1c4af1f47285e9cc0217.svg","fullname":"nieshen","name":"nieshen","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":15,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9558321237564087},"editors":["nieshen"],"editorAvatarUrls":["/avatars/61f8f1d5f6ef1c4af1f47285e9cc0217.svg"],"reactions":[],"isReport":false,"parentCommentId":"67b35f5086e153dc10786119"}}]},{"id":"67b3e35fcf776a173400e427","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2025-02-18T01:33:19.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Enabling Autoregressive Models to Fill In Masked Tokens](https://huggingface.co/papers/2502.06901) (2025)\n* [MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilities](https://huggingface.co/papers/2501.08648) (2025)\n* [Segment-Based Attention Masking for GPTs](https://huggingface.co/papers/2412.18487) (2024)\n* [From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuning](https://huggingface.co/papers/2501.11877) (2025)\n* [Scalable Language Models with Posterior Inference of Latent Thought Vectors](https://huggingface.co/papers/2502.01567) (2025)\n* [Next Block Prediction: Video Generation via Semi-Autoregressive Modeling](https://huggingface.co/papers/2502.07737) (2025)\n* [ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models](https://huggingface.co/papers/2501.13397) (2025)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
\nThe following papers were recommended by the Semantic Scholar API
\n- \n
- Enabling Autoregressive Models to Fill In Masked Tokens (2025) \n
- MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilities (2025) \n
- Segment-Based Attention Masking for GPTs (2024) \n
- From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuning (2025) \n
- Scalable Language Models with Posterior Inference of Latent Thought Vectors (2025) \n
- Next Block Prediction: Video Generation via Semi-Autoregressive Modeling (2025) \n
- ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models (2025) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
Fascinating paper! A video & written review can be found here - https://aipapersacademy.com/large-language-diffusion-models/
\n","updatedAt":"2025-02-21T21:12:18.344Z","author":{"_id":"665edfcf2b842ec980842bd4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/665edfcf2b842ec980842bd4/GJHNPJ3ULIMEMq6VGxZaI.png","fullname":"AI Papers Academy","name":"aipapersacademy","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7000033855438232},"editors":["aipapersacademy"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/665edfcf2b842ec980842bd4/GJHNPJ3ULIMEMq6VGxZaI.png"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2502.09992","authors":[{"_id":"67b2c31125f77e5fc242f4f8","user":{"_id":"640dd9a5fdeaae13908208a7","avatarUrl":"/avatars/61f8f1d5f6ef1c4af1f47285e9cc0217.svg","isPro":false,"fullname":"nieshen","user":"nieshen","type":"user"},"name":"Shen Nie","status":"admin_assigned","statusLastChangedAt":"2025-02-28T12:48:04.969Z","hidden":false},{"_id":"67b2c31125f77e5fc242f4f9","name":"Fengqi Zhu","hidden":false},{"_id":"67b2c31125f77e5fc242f4fa","user":{"_id":"624f909eac5dd186b01ac3f5","avatarUrl":"/avatars/0aafdb1cbb492fda52a0303031cc6c14.svg","isPro":false,"fullname":"Zebin You","user":"yyyou","type":"user"},"name":"Zebin You","status":"admin_assigned","statusLastChangedAt":"2025-02-28T12:47:21.498Z","hidden":false},{"_id":"67b2c31125f77e5fc242f4fb","user":{"_id":"67513d6d3b8586521cda5d76","avatarUrl":"/avatars/0f95cc5c23a0a1da289aa785bd33b616.svg","isPro":false,"fullname":"Xiaolu Zhang","user":"xiaolu0714","type":"user"},"name":"Xiaolu Zhang","status":"admin_assigned","statusLastChangedAt":"2025-02-28T12:47:27.956Z","hidden":false},{"_id":"67b2c31125f77e5fc242f4fc","user":{"_id":"656949b71d7c2ca7b7aae5f2","avatarUrl":"/avatars/e7b23e260eb348cc26b849aaa601a503.svg","isPro":false,"fullname":"Jingyang Ou","user":"JingyangOu","type":"user"},"name":"Jingyang Ou","status":"admin_assigned","statusLastChangedAt":"2025-02-19T15:51:46.024Z","hidden":false},{"_id":"67b2c31125f77e5fc242f4fd","name":"Jun Hu","hidden":false},{"_id":"67b2c31125f77e5fc242f4fe","name":"Jun Zhou","hidden":false},{"_id":"67b2c31125f77e5fc242f4ff","user":{"_id":"657a651e1433ea7d44de6397","avatarUrl":"/avatars/ccfc76f94595a38ff4a80f77c911eabf.svg","isPro":false,"fullname":"Yankai Lin","user":"lyk423","type":"user"},"name":"Yankai Lin","status":"admin_assigned","statusLastChangedAt":"2025-02-19T15:51:39.766Z","hidden":false},{"_id":"67b2c31125f77e5fc242f500","user":{"_id":"64b8c89052b7353d8c6a1013","avatarUrl":"/avatars/cd59fffe81f6b07b4519540b8ff3d95f.svg","isPro":false,"fullname":"Ji-Rong Wen","user":"jrwen","type":"user"},"name":"Ji-Rong Wen","status":"admin_assigned","statusLastChangedAt":"2025-02-19T15:51:33.428Z","hidden":false},{"_id":"67b2c31125f77e5fc242f501","user":{"_id":"64c07b488e2612254361153b","avatarUrl":"/avatars/ade0f783cc4c2d3e73f402637f595471.svg","isPro":false,"fullname":"chongxuan li","user":"zhenxuan00","type":"user"},"name":"Chongxuan Li","status":"admin_assigned","statusLastChangedAt":"2025-02-19T15:51:26.668Z","hidden":false}],"publishedAt":"2025-02-14T08:23:51.000Z","submittedOnDailyAt":"2025-02-17T02:33:18.228Z","title":"Large Language Diffusion Models","submittedOnDailyBy":{"_id":"60f1abe7544c2adfd699860c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674929746905-60f1abe7544c2adfd699860c.jpeg","isPro":false,"fullname":"AK","user":"akhaliq","type":"user"},"summary":"Autoregressive models (ARMs) are widely regarded as the cornerstone of large\nlanguage models (LLMs). We challenge this notion by introducing LLaDA, a\ndiffusion model trained from scratch under the pre-training and supervised\nfine-tuning (SFT) paradigm. LLaDA models distributions through a forward data\nmasking process and a reverse process, parameterized by a vanilla Transformer\nto predict masked tokens. By optimizing a likelihood bound, it provides a\nprincipled generative approach for probabilistic inference. Across extensive\nbenchmarks, LLaDA demonstrates strong scalability, outperforming our\nself-constructed ARM baselines. Remarkably, LLaDA 8B is competitive with strong\nLLMs like LLaMA3 8B in in-context learning and, after SFT, exhibits impressive\ninstruction-following abilities in case studies such as multi-turn dialogue.\nMoreover, LLaDA addresses the reversal curse, surpassing GPT-4o in a reversal\npoem completion task. Our findings establish diffusion models as a viable and\npromising alternative to ARMs, challenging the assumption that key LLM\ncapabilities discussed above are inherently tied to ARMs.","upvotes":126,"discussionId":"67b2c31225f77e5fc242f527","projectPage":"https://ml-gsai.github.io/LLaDA-demo/","githubRepo":"https://github.com/ml-gsai/llada","githubRepoAddedBy":"auto","ai_summary":"LLaDA, a diffusion model trained from scratch, outperforms autoregressive models in benchmarks and demonstrates strong instruction-following capabilities, challenging the dominance of ARMs in LLMs.","ai_keywords":["autoregressive models","LLaDA","diffusion model","pre-training","supervised fine-tuning","vanilla Transformer","likelihood bound","probabilistic inference","in-context learning","instruction-following","reversal curse","LLaMA3","GPT-4o","reversal poem completion"],"githubStars":3580},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"66f612b934b8ac9ffa44f084","avatarUrl":"/avatars/6836c122e19c66c90f1673f28b30d7f0.svg","isPro":false,"fullname":"Tang","user":"tommysally","type":"user"},{"_id":"6270324ebecab9e2dcf245de","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6270324ebecab9e2dcf245de/cMbtWSasyNlYc9hvsEEzt.jpeg","isPro":false,"fullname":"Kye Gomez","user":"kye","type":"user"},{"_id":"6342796a0875f2c99cfd313b","avatarUrl":"/avatars/98575092404c4197b20c929a6499a015.svg","isPro":false,"fullname":"Yuseung \"Phillip\" Lee","user":"phillipinseoul","type":"user"},{"_id":"63b6f2e752c02ae8acbaa4d8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1672934038280-noauth.jpeg","isPro":false,"fullname":"Habibullah Akbar","user":"ChavyvAkvar","type":"user"},{"_id":"64daecec888b7e9c400f59b5","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64daecec888b7e9c400f59b5/DuSAooSj4sKAp1sT_jHFU.png","isPro":false,"fullname":"Delin Qu","user":"delinqu","type":"user"},{"_id":"63c5d43ae2804cb2407e4d43","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1673909278097-noauth.png","isPro":false,"fullname":"xziayro","user":"xziayro","type":"user"},{"_id":"64747f7e33192631bacd8831","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64747f7e33192631bacd8831/dstkZJ4sHJSeqLesV5cOC.jpeg","isPro":false,"fullname":"Taufiq Dwi Purnomo","user":"taufiqdp","type":"user"},{"_id":"646350107e9025b09bd62bab","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/646350107e9025b09bd62bab/Oou_8-WG72ZbkatdQ1-q6.jpeg","isPro":false,"fullname":"momo","user":"wzc991222","type":"user"},{"_id":"650af93422ce64f22b619549","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/RBybzGRphbiV2MXYHrDoc.png","isPro":false,"fullname":"Moritz Reuss","user":"mbreuss","type":"user"},{"_id":"64b929308b53fb5dbd059ce3","avatarUrl":"/avatars/564c44cf45db794747a96f79f30ecd91.svg","isPro":false,"fullname":"Liu Songhua","user":"Huage001","type":"user"},{"_id":"6697eb4da1c67d388b05deef","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6697eb4da1c67d388b05deef/biaVIbuT63RgvlWDtFI_9.jpeg","isPro":false,"fullname":"Anthonny Olime","user":"Aviv-anthonnyolime","type":"user"},{"_id":"668cd4bbe990292e5f6974d3","avatarUrl":"/avatars/d1747b2372e94500ecb5fb56809b482d.svg","isPro":false,"fullname":"Jinyeong Kim","user":"rubatoyeong","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":1}">Abstract
LLaDA, a diffusion model trained from scratch, outperforms autoregressive models in benchmarks and demonstrates strong instruction-following capabilities, challenging the dominance of ARMs in LLMs.
Autoregressive models (ARMs) are widely regarded as the cornerstone of large language models (LLMs). We challenge this notion by introducing LLaDA, a diffusion model trained from scratch under the pre-training and supervised fine-tuning (SFT) paradigm. LLaDA models distributions through a forward data masking process and a reverse process, parameterized by a vanilla Transformer to predict masked tokens. By optimizing a likelihood bound, it provides a principled generative approach for probabilistic inference. Across extensive benchmarks, LLaDA demonstrates strong scalability, outperforming our self-constructed ARM baselines. Remarkably, LLaDA 8B is competitive with strong LLMs like LLaMA3 8B in in-context learning and, after SFT, exhibits impressive instruction-following abilities in case studies such as multi-turn dialogue. Moreover, LLaDA addresses the reversal curse, surpassing GPT-4o in a reversal poem completion task. Our findings establish diffusion models as a viable and promising alternative to ARMs, challenging the assumption that key LLM capabilities discussed above are inherently tied to ARMs.
Community
Question for the authors, how does this technique compare to Discrete Diffusion with Planned Denoising (https://arxiv.org/abs/2410.06264)?
Thank you for your question! The paper you mentioned, DDPO, requires training both the planner and the denoiser simultaneously, while LLaDA only needs to train a denoiser. This leads to differences in both the training and sampling processes of DDPO and LLaDA.
Will you release the pre training dataset in future as it will allow to reproduce and maybe help the community to train different ablations for comparisons ? Anyway thanks for wonderful work 🥳
Thank you for your attention. We have no plan to open-source the pre-training data of LLaDA. We have previously open-sourced the data, training code, and checkpoint for a work (https://arxiv.org/abs/2410.18514) related to masked diffusion models. However, in this work, the number of model parameters is smaller (just 1B), and the data quality is also relatively poor.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Enabling Autoregressive Models to Fill In Masked Tokens (2025)
- MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilities (2025)
- Segment-Based Attention Masking for GPTs (2024)
- From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuning (2025)
- Scalable Language Models with Posterior Inference of Latent Thought Vectors (2025)
- Next Block Prediction: Video Generation via Semi-Autoregressive Modeling (2025)
- ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Fascinating paper! A video & written review can be found here - https://aipapersacademy.com/large-language-diffusion-models/
Models citing this paper 3
Datasets citing this paper 0
No dataset linking this paper