Note CC: https://commoncrawl.org/ en: 305GB en.noclean: 2.3TB en.noblocklist: 380GB realnewslike: 15GB multilingual (mC4): 9.7TB (108 subsets, one per language)

Note From CC 2013-20 ~ 2025-26 (continue updating) 18.5T tokens, English only Blog: https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1

Note From CC 2013-20 ~ 2025-26 (continue updating) 1.3T tokens Paper: https://arxiv.org/abs/2406.17557

Note From CC 2013-20 ~ 2024-18 20TB disk (3T words), 1000+ languages Blog: https://huggingface.co/spaces/HuggingFaceFW/blogpost-fine-tasks

Note From CC, C4, GitHub, Wikipedia, Gutenberg and Books3, ArXiv, Stackexchange 1.2T tokens

Note From CC 2014-15 ~ 2023-14 30T tokens, English, French, Spanish, German, and Italian Blog: https://www.together.ai/blog/redpajama-data-v2

Note From CC 2013-20 ~ 2022-49 2.6T tokens (baseline), 240T tokens (pool) https://data.commoncrawl.org/contrib/datacomp/DCLM-pool/index.html https://data.commoncrawl.org/contrib/datacomp/DCLM-refinedweb/index.html https://data.commoncrawl.org/contrib/datacomp/DCLM-baseline/index.html

Note From CC 2013-20 ~ 2025-13 v1 (2013-20 ~ 2024-30, 3.3T tokens actual & 1.9T synthetic): https://data.commoncrawl.org/contrib/Nemotron/Nemotron-CC/index.html