We introduce a simple and scalable approach to improve the data efficiency of large language model (LLM) training by augmenting existing text data with thinking trajectories.
\n","updatedAt":"2025-09-26T02:38:58.370Z","author":{"_id":"62343594c63d91cec1ca4a8d","avatarUrl":"/avatars/5a1ee74c2dbe349a6ec9843a1599d281.svg","fullname":"Liang Wang","name":"intfloat","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":441,"isUserFollowing":false}},"numEdits":2,"identifiedLanguage":{"language":"en","probability":0.7812336683273315},"editors":["intfloat"],"editorAvatarUrls":["/avatars/5a1ee74c2dbe349a6ec9843a1599d281.svg"],"reactions":[],"isReport":false}},{"id":"68d73f14a68a37a7d830c8e5","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2025-09-27T01:34:12.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Reinforcement Learning on Pre-Training Data](https://huggingface.co/papers/2509.19249) (2025)\n* [PromptCoT 2.0: Scaling Prompt Synthesis for Large Language Model Reasoning](https://huggingface.co/papers/2509.19894) (2025)\n* [Can Structured Templates Facilitate LLMs in Tackling Harder Tasks? : An Exploration of Scaling Laws by Difficulty](https://huggingface.co/papers/2508.19069) (2025)\n* [Large-Scale Diverse Synthesis for Mid-Training](https://huggingface.co/papers/2508.01326) (2025)\n* [Apriel-Nemotron-15B-Thinker](https://huggingface.co/papers/2508.10948) (2025)\n* [Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks](https://huggingface.co/papers/2508.18672) (2025)\n* [CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning tasks](https://huggingface.co/papers/2507.23751) (2025)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
\nThe following papers were recommended by the Semantic Scholar API
\n- \n
- Reinforcement Learning on Pre-Training Data (2025) \n
- PromptCoT 2.0: Scaling Prompt Synthesis for Large Language Model Reasoning (2025) \n
- Can Structured Templates Facilitate LLMs in Tackling Harder Tasks? : An Exploration of Scaling Laws by Difficulty (2025) \n
- Large-Scale Diverse Synthesis for Mid-Training (2025) \n
- Apriel-Nemotron-15B-Thinker (2025) \n
- Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks (2025) \n
- CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning tasks (2025) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
Abstract
Thinking augmented pre-training improves data efficiency and performance of large language models by augmenting text with automatically generated thinking trajectories.
This paper introduces a simple and scalable approach to improve the data efficiency of large language model (LLM) training by augmenting existing text data with thinking trajectories. The compute for pre-training LLMs has been growing at an unprecedented rate, while the availability of high-quality data remains limited. Consequently, maximizing the utility of available data constitutes a significant research challenge. A primary impediment is that certain high-quality tokens are difficult to learn given a fixed model capacity, as the underlying rationale for a single token can be exceptionally complex and deep. To address this issue, we propose Thinking augmented Pre-Training (TPT), a universal methodology that augments text with automatically generated thinking trajectories. Such augmentation effectively increases the volume of the training data and makes high-quality tokens more learnable through step-by-step reasoning and decomposition. We apply TPT across diverse training configurations up to 100B tokens, encompassing pre-training with both constrained and abundant data, as well as mid-training from strong open-source checkpoints. Experimental results indicate that our method substantially improves the performance of LLMs across various model sizes and families. Notably, TPT enhances the data efficiency of LLM pre-training by a factor of 3. For a 3B parameter model, it improves the post-training performance by over 10% on several challenging reasoning benchmarks.
Community
We introduce a simple and scalable approach to improve the data efficiency of large language model (LLM) training by augmenting existing text data with thinking trajectories.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Reinforcement Learning on Pre-Training Data (2025)
- PromptCoT 2.0: Scaling Prompt Synthesis for Large Language Model Reasoning (2025)
- Can Structured Templates Facilitate LLMs in Tackling Harder Tasks? : An Exploration of Scaling Laws by Difficulty (2025)
- Large-Scale Diverse Synthesis for Mid-Training (2025)
- Apriel-Nemotron-15B-Thinker (2025)
- Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks (2025)
- CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning tasks (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 0
No model linking this paper
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper