https://github.com/facebookresearch/RAM/tree/main/projects/cocomix

\n","updatedAt":"2025-02-13T04:42:44.346Z","author":{"_id":"60f1abe7544c2adfd699860c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674929746905-60f1abe7544c2adfd699860c.jpeg","fullname":"AK","name":"akhaliq","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":9179,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.5604965686798096},"editors":["akhaliq"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674929746905-60f1abe7544c2adfd699860c.jpeg"],"reactions":[],"isReport":false}},{"id":"67adce21144d557379137a41","author":{"_id":"647e534b770c299e56fb4ff2","avatarUrl":"/avatars/a0dcb0a1b7d55fc1c35876118f01d499.svg","fullname":"Maksim2200","name":"maxamly","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false},"createdAt":"2025-02-13T10:49:05.000Z","type":"comment","data":{"edited":true,"hidden":true,"hiddenBy":"","latest":{"raw":"This comment has been hidden","html":"This comment has been hidden","updatedAt":"2025-02-13T10:50:35.019Z","author":{"_id":"647e534b770c299e56fb4ff2","avatarUrl":"/avatars/a0dcb0a1b7d55fc1c35876118f01d499.svg","fullname":"Maksim2200","name":"maxamly","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":0,"editors":[],"editorAvatarUrls":[],"reactions":[]}},{"id":"67ae9dbae99f4989893e1d66","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2025-02-14T01:34:50.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning](https://huggingface.co/papers/2502.03275) (2025)\n* [Chunk-Distilled Language Modeling](https://huggingface.co/papers/2501.00343) (2024)\n* [Optimizing Pretraining Data Mixtures with LLM-Estimated Utility](https://huggingface.co/papers/2501.11747) (2025)\n* [Efficient Reasoning with Hidden Thinking](https://huggingface.co/papers/2501.19201) (2025)\n* [Deliberation in Latent Space via Differentiable Cache Augmentation](https://huggingface.co/papers/2412.17747) (2024)\n* [Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey](https://huggingface.co/papers/2412.18619) (2024)\n* [The Impact of Token Granularity on the Predictive Power of Language Model Surprisal](https://huggingface.co/papers/2412.11940) (2024)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning (2025)
Chunk-Distilled Language Modeling (2024)
Optimizing Pretraining Data Mixtures with LLM-Estimated Utility (2025)
Efficient Reasoning with Hidden Thinking (2025)
Deliberation in Latent Space via Differentiable Cache Augmentation (2024)
Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey (2024)
The Impact of Token Granularity on the Predictive Power of Language Model Surprisal (2024)

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2025-02-14T01:34:50.223Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7177917957305908},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}},{"id":"67b0ae69479dba25141785f8","author":{"_id":"665edfcf2b842ec980842bd4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/665edfcf2b842ec980842bd4/GJHNPJ3ULIMEMq6VGxZaI.png","fullname":"AI Papers Academy","name":"aipapersacademy","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false},"createdAt":"2025-02-15T15:10:33.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"A video & written breakdown - https://aipapersacademy.com/cocomix/","html":"

A video & written breakdown - https://aipapersacademy.com/cocomix/

\n","updatedAt":"2025-02-15T15:10:33.735Z","author":{"_id":"665edfcf2b842ec980842bd4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/665edfcf2b842ec980842bd4/GJHNPJ3ULIMEMq6VGxZaI.png","fullname":"AI Papers Academy","name":"aipapersacademy","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.790912389755249},"editors":["aipapersacademy"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/665edfcf2b842ec980842bd4/GJHNPJ3ULIMEMq6VGxZaI.png"],"reactions":[],"isReport":false}},{"id":"682a0a759acd6ca17d37db09","author":{"_id":"6107ff14cf6a45eafb9afd3d","avatarUrl":"/avatars/77579ec7a4c3cfe88fe1a4c7150ce4c2.svg","fullname":"Gábor Berend","name":"berendg","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false},"createdAt":"2025-05-18T16:27:33.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This paper also exploits a similar idea, but for encoder based models: https://aclanthology.org/2023.findings-acl.876/","html":"

This paper also exploits a similar idea, but for encoder based models: https://aclanthology.org/2023.findings-acl.876/

\n","updatedAt":"2025-05-18T16:27:33.506Z","author":{"_id":"6107ff14cf6a45eafb9afd3d","avatarUrl":"/avatars/77579ec7a4c3cfe88fe1a4c7150ce4c2.svg","fullname":"Gábor Berend","name":"berendg","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8254205584526062},"editors":["berendg"],"editorAvatarUrls":["/avatars/77579ec7a4c3cfe88fe1a4c7150ce4c2.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2502.08524","authors":[{"_id":"67ad783da2808b57a3cd3316","user":{"_id":"65b15a471566a60bda376e02","avatarUrl":"/avatars/413340cb2d44c765accb5b3fc450ccc4.svg","isPro":false,"fullname":"Jihoon Tack","user":"jihoontack","type":"user"},"name":"Jihoon Tack","status":"admin_assigned","statusLastChangedAt":"2025-02-14T15:31:46.173Z","hidden":false},{"_id":"67ad783da2808b57a3cd3317","user":{"_id":"65dd30a6ea1a202d3cb469f7","avatarUrl":"/avatars/14029def026380e6e20822d916b25a72.svg","isPro":false,"fullname":"Jack Lanchantin","user":"jcklcn","type":"user"},"name":"Jack Lanchantin","status":"admin_assigned","statusLastChangedAt":"2025-02-14T15:31:36.784Z","hidden":false},{"_id":"67ad783da2808b57a3cd3318","name":"Jane Yu","hidden":false},{"_id":"67ad783da2808b57a3cd3319","name":"Andrew Cohen","hidden":false},{"_id":"67ad783da2808b57a3cd331a","name":"Ilia Kulikov","hidden":false},{"_id":"67ad783da2808b57a3cd331b","name":"Janice Lan","hidden":false},{"_id":"67ad783da2808b57a3cd331c","name":"Shibo Hao","hidden":false},{"_id":"67ad783da2808b57a3cd331d","name":"Yuandong Tian","hidden":false},{"_id":"67ad783da2808b57a3cd331e","name":"Jason Weston","hidden":false},{"_id":"67ad783da2808b57a3cd331f","user":{"_id":"659a395421a7431643caedda","avatarUrl":"/avatars/c1e0bbcedce68fe3b4fe39e0cf01c65c.svg","isPro":false,"fullname":"Xian Li","user":"xlxxl","type":"user"},"name":"Xian Li","status":"extracted_confirmed","statusLastChangedAt":"2025-05-22T23:45:54.667Z","hidden":false}],"publishedAt":"2025-02-12T16:00:11.000Z","submittedOnDailyAt":"2025-02-13T02:12:44.287Z","title":"LLM Pretraining with Continuous Concepts","submittedOnDailyBy":{"_id":"60f1abe7544c2adfd699860c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674929746905-60f1abe7544c2adfd699860c.jpeg","isPro":false,"fullname":"AK","user":"akhaliq","type":"user"},"summary":"Next token prediction has been the standard training objective used in large\nlanguage model pretraining. Representations are learned as a result of\noptimizing for token-level perplexity. We propose Continuous Concept Mixing\n(CoCoMix), a novel pretraining framework that combines discrete next token\nprediction with continuous concepts. Specifically, CoCoMix predicts continuous\nconcepts learned from a pretrained sparse autoencoder and mixes them into the\nmodel's hidden state by interleaving with token hidden representations. Through\nexperiments on multiple benchmarks, including language modeling and downstream\nreasoning tasks, we show that CoCoMix is more sample efficient and consistently\noutperforms standard next token prediction, knowledge distillation and\ninserting pause tokens. We find that combining both concept learning and\ninterleaving in an end-to-end framework is critical to performance gains.\nFurthermore, CoCoMix enhances interpretability and steerability by allowing\ndirect inspection and modification of the predicted concept, offering a\ntransparent way to guide the model's internal reasoning process.","upvotes":30,"discussionId":"67ad783ea2808b57a3cd3361","githubRepo":"https://github.com/facebookresearch/RAM","githubRepoAddedBy":"auto","ai_summary":"A novel pretraining framework, Continuous Concept Mixing (CoCoMix), integrates continuous concepts from a sparse autoencoder into language models, improving sample efficiency and interpretability.","ai_keywords":["Continuous Concept Mixing","CoCoMix","sparse autoencoder","next token prediction","knowledge distillation","pause tokens","concept learning","interleaving","interpretability","steerability"],"githubStars":343},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6479f8335f3450e1ded40774","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/iOjE2dcSUS-XYE0WJe6U8.jpeg","isPro":false,"fullname":"Andrei Semenov","user":"Andron00e","type":"user"},{"_id":"66f612b934b8ac9ffa44f084","avatarUrl":"/avatars/6836c122e19c66c90f1673f28b30d7f0.svg","isPro":false,"fullname":"Tang","user":"tommysally","type":"user"},{"_id":"63f5993afcf95ecac2b419b5","avatarUrl":"/avatars/a8c020080a84d9a663789c4fb19270e9.svg","isPro":false,"fullname":"Mengde Xu","user":"Mendel192","type":"user"},{"_id":"6365a7e2f31ef76df4028612","avatarUrl":"/avatars/e2748304724b94b398216557fa0237c1.svg","isPro":false,"fullname":"Ber666","user":"SDSB","type":"user"},{"_id":"660ee5df35d092e3fc2a3685","avatarUrl":"/avatars/a7e0472fb7ea49973f74e3eea13dc964.svg","isPro":false,"fullname":"Shibo Hao","user":"Shibo-UCSD","type":"user"},{"_id":"648eb1eb59c4e5c87dc116e0","avatarUrl":"/avatars/c636cea39c2c0937f01398c94ead5dad.svg","isPro":false,"fullname":"fdsqefsgergd","user":"T-representer","type":"user"},{"_id":"648e72a866dcba8b5aaecbdc","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/UKR1KF7s0OWCFQzsTrcuX.jpeg","isPro":false,"fullname":"Sergey Bratchikov","user":"hivaze","type":"user"},{"_id":"633af7c4c9b44f5c6ac218ee","avatarUrl":"/avatars/d9881844b399efa8ea9b81f86bb60bad.svg","isPro":false,"fullname":"Opit Opit","user":"Opit","type":"user"},{"_id":"63b6f2e752c02ae8acbaa4d8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1672934038280-noauth.jpeg","isPro":false,"fullname":"Habibullah Akbar","user":"ChavyvAkvar","type":"user"},{"_id":"64c1c77c245c55a21c6f5a13","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64c1c77c245c55a21c6f5a13/d9zlSksf3TxWpBbb-r0fd.jpeg","isPro":false,"fullname":"Reza Sayar","user":"Reza2kn","type":"user"},{"_id":"6342796a0875f2c99cfd313b","avatarUrl":"/avatars/98575092404c4197b20c929a6499a015.svg","isPro":false,"fullname":"Yuseung \"Phillip\" Lee","user":"phillipinseoul","type":"user"},{"_id":"668cd4bbe990292e5f6974d3","avatarUrl":"/avatars/d1747b2372e94500ecb5fb56809b482d.svg","isPro":false,"fullname":"Jinyeong Kim","user":"rubatoyeong","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0}">

Papers

arxiv:2502.08524

LLM Pretraining with Continuous Concepts

Published on Feb 12, 2025

· Submitted by

AK on Feb 13, 2025

Upvote

Authors:

Jihoon Tack ,

Jack Lanchantin ,

Xian Li

Abstract

A novel pretraining framework, Continuous Concept Mixing (CoCoMix), integrates continuous concepts from a sparse autoencoder into language models, improving sample efficiency and interpretability.

AI-generated summary

Next token prediction has been the standard training objective used in large language model pretraining. Representations are learned as a result of optimizing for token-level perplexity. We propose Continuous Concept Mixing (CoCoMix), a novel pretraining framework that combines discrete next token prediction with continuous concepts. Specifically, CoCoMix predicts continuous concepts learned from a pretrained sparse autoencoder and mixes them into the model's hidden state by interleaving with token hidden representations. Through experiments on multiple benchmarks, including language modeling and downstream reasoning tasks, we show that CoCoMix is more sample efficient and consistently outperforms standard next token prediction, knowledge distillation and inserting pause tokens. We find that combining both concept learning and interleaving in an end-to-end framework is critical to performance gains. Furthermore, CoCoMix enhances interpretability and steerability by allowing direct inspection and modification of the predicted concept, offering a transparent way to guide the model's internal reasoning process.