Pico Decoder models (10M-500M)
pico-train: Minimalist training framework for language models. \npico-analyze: Tools for measuring and visualizing model learning dynamics across checkpoints. \n\n
\n
\n
View all Papers
This HuggingFace organization hosts our pre-trained models and datasets, while the GitHub repository provides the code to train and analyze your own model suites from scratch.
\nAll code and artifacts are licensed under a permissive Apache-2.0 license.
\n\n\nPro Tip 🚀 : \nTo learn more about these libraries and explore detailed tutorials, visit our official website picolm.io and get fully acquainted with the Pico ecosystem.
\n
\n
🤗 HuggingFace Resources (You Are Here)
\n1. Pre-trained Model Suite
\nOur complete suite of models from 11M to 570M parameters trained with Pico:
\n- \n
- pico-decoder-tiny (11M parameters) \n
- pico-decoder-small (65M parameters) \n
- pico-decoder-medium (181M parameters) \n
- pico-decoder-large (570M parameters) \n
\n\n🚧 Disclaimer These models are still under construction. The models released in this repository have been trained for 125,000 steps (corresponding to ~250B tokens). Training will finalize after 200,000 steps.
\n🚧 Coming Soon! pico-decoder-xl (1B+ parameters) Watch this space or star our GitHub repository for updates!
\n
All models are on the pretokenized-dolma dataset. They all see the same training data at each training step, use the same optimizatation process, and share the same model architecture; the only difference between models is the size of their hidden dimension.
\nIn each model repository, we version control checkpoints every 1000 steps that contain:
\n- \n
- Weights and optimizer states (HuggingFace and Lightning Fabric-compatible versions) \n
- Model activations and gradients \n
- The batch of training data observed at the given training step \n
We visualize the learning process in our Wandb.
\nModel Details:
\n\n\t\n\t\t
\n
\n| Aspect | \nDetails | \n
|---|---|
| Architecture | \n- Llama-style transformer (decoder-only) - RMSNorm normalization - RoPE (Rotary Positional Embeddings) - Multi-head attention with KV-cache - SwiGLU activation function | \n
| Sequence Length | \n2048 | \n
| Batch Size | \n1024 | \n
| Optimizer | \nAdamW | \n
| Learning Rate | \n3e-4 (one-cycle warmup) | \n
| Gradient Clipping | \n1.0 | \n
| Precision | \nMixed precision training | \n
| Vocabulary Size | \n50,280 | \n
2. Datasets
\n- \n
- \n
- \n
- 420B tokens of pre-processed, tokenized and shuffled text extraced from the DOLMA corpus \n
- We use this dataset to train our model suite \n
\n - \n
- \n
- A smaller version of the pretokenized-dolma corpus for quick experiments \n
\n - \n
- \n
- A tokenized and shuffled version of the Paloma evaluation corpus \n
- The Paloma corpus was carefully curated to be disjoint from the Dolma corpus and provides \n
- We use this corpus to evaluate the perplexity of our models \n
\n - \n
- \n
- A sub-sampled version of the pretokenized-dolma corpus \n
\n
All datasets are tokenized using the OLMo Tokenizer
\n\n
🔍 Citation
\nIf you use Pico in academic or professional work, please cite it:
\n@inproceedings{diehl-martinez-etal-2025-pico,\n title = \"Pico: A Modular Framework for Hypothesis-Driven Small Language Model Research\",\n author = \"Diehl Martinez, Richard and\n Africa, David Demitri and\n Weiss, Yuval and\n Salhan, Suchir and\n Daniels, Ryan and\n Buttery, Paula\",\n editor = {Habernal, Ivan and\n Schulam, Peter and\n Tiedemann, J{\\\"o}rg},\n booktitle = \"Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations\",\n month = nov,\n year = \"2025\",\n address = \"Suzhou, China\",\n publisher = \"Association for Computational Linguistics\",\n}\nThanks for checking out Pico!
Star our GitHub repositories or join our community discussions to stay updated. If you find a bug or have questions, open an issue—contributions are welcome!
AI & ML interests
None defined yet.
Recent Activity
View all activity
Papers
View all Papers
Organization Card
Pico: A Modular Framework for Hypothesis-Driven Small Language Model Research
Welcome to Pico LM 👋, a research initiative dedicated to demystifying language model learning.
We create two complementary frameworks (pico-train and pico-analyze) for training and analyzing small to mid-scale language models (1M–1B parameters). Our mission is to provide a transparent, research-oriented workflow that illuminates how these models learn.
For full documentation and code, visit our two main repositories:
- pico-train: Minimalist training framework for language models.
- pico-analyze: Tools for measuring and visualizing model learning dynamics across checkpoints.
This HuggingFace organization hosts our pre-trained models and datasets, while the GitHub repository provides the code to train and analyze your own model suites from scratch.
All code and artifacts are licensed under a permissive Apache-2.0 license.
Pro Tip 🚀 : To learn more about these libraries and explore detailed tutorials, visit our official website picolm.io and get fully acquainted with the Pico ecosystem.
🤗 HuggingFace Resources (You Are Here)
1. Pre-trained Model Suite
Our complete suite of models from 11M to 570M parameters trained with Pico:
- pico-decoder-tiny (11M parameters)
- pico-decoder-small (65M parameters)
- pico-decoder-medium (181M parameters)
- pico-decoder-large (570M parameters)
🚧 Disclaimer These models are still under construction. The models released in this repository have been trained for 125,000 steps (corresponding to ~250B tokens). Training will finalize after 200,000 steps.
🚧 Coming Soon! pico-decoder-xl (1B+ parameters) Watch this space or star our GitHub repository for updates!
All models are on the pretokenized-dolma dataset. They all see the same training data at each training step, use the same optimizatation process, and share the same model architecture; the only difference between models is the size of their hidden dimension.
In each model repository, we version control checkpoints every 1000 steps that contain:
- Weights and optimizer states (HuggingFace and Lightning Fabric-compatible versions)
- Model activations and gradients
- The batch of training data observed at the given training step
We visualize the learning process in our Wandb.
Model Details:
| Aspect | Details |
|---|---|
| Architecture | - Llama-style transformer (decoder-only) - RMSNorm normalization - RoPE (Rotary Positional Embeddings) - Multi-head attention with KV-cache - SwiGLU activation function |
| Sequence Length | 2048 |
| Batch Size | 1024 |
| Optimizer | AdamW |
| Learning Rate | 3e-4 (one-cycle warmup) |
| Gradient Clipping | 1.0 |
| Precision | Mixed precision training |
| Vocabulary Size | 50,280 |
2. Datasets
-
- 420B tokens of pre-processed, tokenized and shuffled text extraced from the DOLMA corpus
- We use this dataset to train our model suite
-
- A smaller version of the pretokenized-dolma corpus for quick experiments
-
- A tokenized and shuffled version of the Paloma evaluation corpus
- The Paloma corpus was carefully curated to be disjoint from the Dolma corpus and provides
- We use this corpus to evaluate the perplexity of our models
-
- A sub-sampled version of the pretokenized-dolma corpus
All datasets are tokenized using the OLMo Tokenizer
🔍 Citation
If you use Pico in academic or professional work, please cite it:
@inproceedings{diehl-martinez-etal-2025-pico,
title = "Pico: A Modular Framework for Hypothesis-Driven Small Language Model Research",
author = "Diehl Martinez, Richard and
Africa, David Demitri and
Weiss, Yuval and
Salhan, Suchir and
Daniels, Ryan and
Buttery, Paula",
editor = {Habernal, Ivan and
Schulam, Peter and
Tiedemann, J{\"o}rg},
booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",
month = nov,
year = "2025",
address = "Suzhou, China",
publisher = "Association for Computational Linguistics",
}
Thanks for checking out Pico!
Star our GitHub repositories or join our community discussions to stay updated. If you find a bug or have questions, open an issue—contributions are welcome!