https://www.datacomp.ai/dclm/\n","updatedAt":"2024-06-19T01:45:50.999Z","author":{"_id":"6302e010b1c0b88509812263","avatarUrl":"/avatars/80b7da76045c742bdbc794198c9ef780.svg","fullname":"Merlin Li","name":"MerlinLi","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.8167706727981567},"editors":["MerlinLi"],"editorAvatarUrls":["/avatars/80b7da76045c742bdbc794198c9ef780.svg"],"reactions":[],"isReport":false}},{"id":"667b6a0aa0e1bc1406631245","author":{"_id":"646b8e6f31968a60a0201a12","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/646b8e6f31968a60a0201a12/SU2Gs1NPuk1zoXHwFHl0U.jpeg","fullname":")))?!?(((","name":"stereoplegic","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3925,"isUserFollowing":false},"createdAt":"2024-06-26T01:08:26.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"@librarian-bot recommend","html":"
\n\n@librarian-bot\n\t recommend
\n","updatedAt":"2024-06-26T01:08:26.456Z","author":{"_id":"646b8e6f31968a60a0201a12","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/646b8e6f31968a60a0201a12/SU2Gs1NPuk1zoXHwFHl0U.jpeg","fullname":")))?!?(((","name":"stereoplegic","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3925,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7918877601623535},"editors":["stereoplegic"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/646b8e6f31968a60a0201a12/SU2Gs1NPuk1zoXHwFHl0U.jpeg"],"reactions":[],"isReport":false},"replies":[{"id":"667b6a14eeef441d206121a3","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":317,"isUserFollowing":false},"createdAt":"2024-06-26T01:08:36.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Does your data spark joy? Performance gains from domain upsampling at the end of training](https://huggingface.co/papers/2406.03476) (2024)\n* [Large Language Model-guided Document Selection](https://huggingface.co/papers/2406.04638) (2024)\n* [UCCIX: Irish-eXcellence Large Language Model](https://huggingface.co/papers/2405.13010) (2024)\n* [GECKO: Generative Language Model for English, Code and Korean](https://huggingface.co/papers/2405.15640) (2024)\n* [Large Scale Transfer Learning for Tabular Data via Language Modeling](https://huggingface.co/papers/2406.12031) (2024)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
\nThe following papers were recommended by the Semantic Scholar API
\n- \n
- Does your data spark joy? Performance gains from domain upsampling at the end of training (2024) \n
- Large Language Model-guided Document Selection (2024) \n
- UCCIX: Irish-eXcellence Large Language Model (2024) \n
- GECKO: Generative Language Model for English, Code and Korean (2024) \n
- Large Scale Transfer Learning for Tabular Data via Language Modeling (2024) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
Papers
arxiv:2406.11794
DataComp-LM: In search of the next generation of training sets for language models
Authors:
Abstract
DataComp for Language Models (DCLM) provides a benchmark with a standardized corpus and recipes to improve language models through controlled dataset experiments, showing the importance of data curation for performance and compute efficiency.
AI-generated summary
We introduce DataComp for Language Models (DCLM), a testbed for controlled dataset experiments with the goal of improving language models. As part of DCLM, we provide a standardized corpus of 240T tokens extracted from Common Crawl, effective pretraining recipes based on the OpenLM framework, and a broad suite of 53 downstream evaluations. Participants in the DCLM benchmark can experiment with data curation strategies such as deduplication, filtering, and data mixing at model scales ranging from 412M to 7B parameters. As a baseline for DCLM, we conduct extensive experiments and find that model-based filtering is key to assembling a high-quality training set. The resulting dataset, DCLM-Baseline enables training a 7B parameter language model from scratch to 64% 5-shot accuracy on MMLU with 2.6T training tokens. Compared to MAP-Neo, the previous state-of-the-art in open-data language models, DCLM-Baseline represents a 6.6 percentage point improvement on MMLU while being trained with 40% less compute. Our baseline model is also comparable to Mistral-7B-v0.3 and Llama 3 8B on MMLU (63% & 66%), and performs similarly on an average of 53 natural language understanding tasks while being trained with 6.6x less compute than Llama 3 8B. Our results highlight the importance of dataset design for training language models and offer a starting point for further research on data curation.
Community
SOTA training set for Language Models, beats Mistral, approaches LLAMA-3/Gemma with 2.6T tokens total. Open Dataset, Open Model.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Does your data spark joy? Performance gains from domain upsampling at the end of training (2024)
- Large Language Model-guided Document Selection (2024)
- UCCIX: Irish-eXcellence Large Language Model (2024)
- GECKO: Generative Language Model for English, Code and Korean (2024)
- Large Scale Transfer Learning for Tabular Data via Language Modeling (2024)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend