$\"Screenshot$

\n","updatedAt":"2025-01-17T14:21:33.925Z","author":{"_id":"60107b385ac3e86b3ea4fc34","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1627505688463-60107b385ac3e86b3ea4fc34.jpeg","fullname":"Daniel van Strien","name":"davanstrien","type":"user","isPro":true,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":838,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.5530889630317688},"editors":["davanstrien"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1627505688463-60107b385ac3e86b3ea4fc34.jpeg"],"reactions":[],"isReport":false}},{"id":"678b04c04e8561497c68d2b0","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2025-01-18T01:32:48.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [The Roles of English in Evaluating Multilingual Language Models](https://huggingface.co/papers/2412.08392) (2024)\n* [FullStack Bench: Evaluating LLMs as Full Stack Coders](https://huggingface.co/papers/2412.00535) (2024)\n* [RedPajama: an Open Dataset for Training Large Language Models](https://huggingface.co/papers/2411.12372) (2024)\n* [Maya: An Instruction Finetuned Multilingual Multimodal Model](https://huggingface.co/papers/2412.07112) (2024)\n* [CodeRepoQA: A Large-scale Benchmark for Software Engineering Question Answering](https://huggingface.co/papers/2412.14764) (2024)\n* [Training and Evaluating Language Models with Template-based Data Generation](https://huggingface.co/papers/2411.18104) (2024)\n* [Localizing AI: Evaluating Open-Weight Language Models for Languages of Baltic States](https://huggingface.co/papers/2501.03952) (2025)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2025-01-18T01:32:48.351Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7092196941375732},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[{"reaction":"👍","users":["davanstrien"],"count":1}],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2501.09653","authors":[{"_id":"678a632dd03f325c5a5ad954","user":{"_id":"643d13b04816d7cb420966dc","avatarUrl":"/avatars/4f4875e35fa57c2b958cdb5dd6b70c27.svg","isPro":false,"fullname":"Jonathan katzy","user":"Jkatzy","type":"user"},"name":"Jonathan Katzy","status":"claimed_verified","statusLastChangedAt":"2025-01-17T17:05:55.731Z","hidden":false},{"_id":"678a632dd03f325c5a5ad955","user":{"_id":"6446fea4bd9e0e0edf521d49","avatarUrl":"/avatars/493c16b52e1a9a940b4e02b6cd6c6cc3.svg","isPro":false,"fullname":"Razvan Popescu","user":"Razvan27","type":"user"},"name":"Razvan Mihai Popescu","status":"claimed_verified","statusLastChangedAt":"2025-01-17T17:05:53.530Z","hidden":false},{"_id":"678a632dd03f325c5a5ad956","name":"Arie van Deursen","hidden":false},{"_id":"678a632dd03f325c5a5ad957","user":{"_id":"63dd4b2af37111482526e0e9","avatarUrl":"/avatars/ffd2a07c61ea6272009df6184cb9dcef.svg","isPro":false,"fullname":"Maliheh Izadi","user":"MalihehIzadi","type":"user"},"name":"Maliheh Izadi","status":"admin_assigned","statusLastChangedAt":"2025-01-17T14:25:21.390Z","hidden":false}],"publishedAt":"2025-01-16T16:48:41.000Z","submittedOnDailyAt":"2025-01-17T11:51:33.913Z","title":"The Heap: A Contamination-Free Multilingual Code Dataset for Evaluating\n Large Language Models","submittedOnDailyBy":{"_id":"60107b385ac3e86b3ea4fc34","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1627505688463-60107b385ac3e86b3ea4fc34.jpeg","isPro":true,"fullname":"Daniel van Strien","user":"davanstrien","type":"user"},"summary":"The recent rise in the popularity of large language models has spurred the\ndevelopment of extensive code datasets needed to train them. This has left\nlimited code available for collection and use in the downstream investigation\nof specific behaviors, or evaluation of large language models without suffering\nfrom data contamination. To address this problem, we release The Heap, a large\nmultilingual dataset covering 57 programming languages that has been\ndeduplicated with respect to other open datasets of code, enabling researchers\nto conduct fair evaluations of large language models without significant data\ncleaning overhead.","upvotes":12,"discussionId":"678a632dd03f325c5a5ad997","ai_summary":"A large multilingual code dataset, The Heap, enables fair evaluations of language models by minimizing data contamination and cleaning overhead.","ai_keywords":["large language models","code datasets","deduplicated","multilingual","programming languages","fair evaluations","data contamination","data cleaning"]},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"60107b385ac3e86b3ea4fc34","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1627505688463-60107b385ac3e86b3ea4fc34.jpeg","isPro":true,"fullname":"Daniel van Strien","user":"davanstrien","type":"user"},{"_id":"6446fea4bd9e0e0edf521d49","avatarUrl":"/avatars/493c16b52e1a9a940b4e02b6cd6c6cc3.svg","isPro":false,"fullname":"Razvan Popescu","user":"Razvan27","type":"user"},{"_id":"63dd4b2af37111482526e0e9","avatarUrl":"/avatars/ffd2a07c61ea6272009df6184cb9dcef.svg","isPro":false,"fullname":"Maliheh Izadi","user":"MalihehIzadi","type":"user"},{"_id":"643d13b04816d7cb420966dc","avatarUrl":"/avatars/4f4875e35fa57c2b958cdb5dd6b70c27.svg","isPro":false,"fullname":"Jonathan katzy","user":"Jkatzy","type":"user"},{"_id":"63248b111ed511c0c5256144","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63248b111ed511c0c5256144/Fn265uESH8HPqrymB59xu.jpeg","isPro":false,"fullname":"Ali Al-Kaswan","user":"aalkaswan","type":"user"},{"_id":"655a627aab0644b531a02eb1","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/655a627aab0644b531a02eb1/CVutvAMBNlqcjDT7k2SUV.jpeg","isPro":false,"fullname":"Roham Koohestani","user":"RohamKoohestani","type":"user"},{"_id":"67892fec24072719efc14030","avatarUrl":"/avatars/c5c8d5b87b5ba589b90a2a68c6228779.svg","isPro":false,"fullname":"Ziyou Li","user":"tilos","type":"user"},{"_id":"648eb1eb59c4e5c87dc116e0","avatarUrl":"/avatars/c636cea39c2c0937f01398c94ead5dad.svg","isPro":false,"fullname":"fdsqefsgergd","user":"T-representer","type":"user"},{"_id":"645b91991c24cd669dd7618a","avatarUrl":"/avatars/7c8a7b7b0668239e96bc160f142e8de9.svg","isPro":false,"fullname":"z","user":"Huye2023","type":"user"},{"_id":"64224614eaad1bcb28b3d46d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64224614eaad1bcb28b3d46d/7oQCVOjcbiNedDABovvXG.png","isPro":false,"fullname":"Elijah Wilt","user":"ooj","type":"user"},{"_id":"675d024d828281871f777425","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/675d024d828281871f777425/1JnFQoStOVxbqXEb2D9UI.jpeg","isPro":false,"fullname":"Benjamin Moore","user":"Mooreee","type":"user"},{"_id":"63082bb7bc0a2a5ee2253523","avatarUrl":"/avatars/6cf8d12d16d15db1070fbea89b5b3967.svg","isPro":false,"fullname":"Kuo-Hsin Tu","user":"dapumptu","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0}">

Papers

arxiv:2501.09653

The Heap: A Contamination-Free Multilingual Code Dataset for Evaluating Large Language Models

Published on Jan 16, 2025

· Submitted by

Daniel van Strien on Jan 17, 2025

Upvote

Authors:

Jonathan Katzy ,

Razvan Mihai Popescu ,

Maliheh Izadi

Abstract

A large multilingual code dataset, The Heap, enables fair evaluations of language models by minimizing data contamination and cleaning overhead.

AI-generated summary

The recent rise in the popularity of large language models has spurred the development of extensive code datasets needed to train them. This has left limited code available for collection and use in the downstream investigation of specific behaviors, or evaluation of large language models without suffering from data contamination. To address this problem, we release The Heap, a large multilingual dataset covering 57 programming languages that has been deduplicated with respect to other open datasets of code, enabling researchers to conduct fair evaluations of large language models without significant data cleaning overhead.