here

\n","updatedAt":"2024-11-12T14:07:05.536Z","author":{"_id":"64b04320e5000ae8a57570a7","avatarUrl":"/avatars/226f57ea23b9dd861e71b7ba7821adce.svg","fullname":"Massimo Caccia","name":"optimass","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.5833666324615479},"editors":["optimass"],"editorAvatarUrls":["/avatars/226f57ea23b9dd861e71b7ba7821adce.svg"],"reactions":[],"isReport":false}},{"id":"673401eb81bcaf1774a4769f","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2024-11-13T01:33:31.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Can Language Models Replace Programmers? REPOCOD Says 'Not Yet'](https://huggingface.co/papers/2410.21647) (2024)\n* [CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding Capabilities of CodeLLMs](https://huggingface.co/papers/2410.01999) (2024)\n* [CodeDPO: Aligning Code Models with Self Generated and Verified Source Code](https://huggingface.co/papers/2410.05605) (2024)\n* [Effi-Code: Unleashing Code Efficiency in Language Models](https://huggingface.co/papers/2410.10209) (2024)\n* [CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repair](https://huggingface.co/papers/2409.12993) (2024)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2024-11-13T01:33:31.113Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7453088164329529},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2411.05830","authors":[{"_id":"673360a07c79bb4d069dcc57","name":"Nizar Islah","hidden":false},{"_id":"673360a07c79bb4d069dcc58","user":{"_id":"6421f59be61513ec1c66f277","avatarUrl":"/avatars/7792f5bad78dcae548125e6d6d6c22e7.svg","isPro":false,"fullname":"Justine Gehring","user":"justine-gehring","type":"user"},"name":"Justine Gehring","status":"claimed_verified","statusLastChangedAt":"2024-11-15T09:25:48.365Z","hidden":false},{"_id":"673360a07c79bb4d069dcc59","name":"Diganta Misra","hidden":false},{"_id":"673360a07c79bb4d069dcc5a","name":"Eilif Muller","hidden":false},{"_id":"673360a07c79bb4d069dcc5b","name":"Irina Rish","hidden":false},{"_id":"673360a07c79bb4d069dcc5c","user":{"_id":"62b7fb545233925f253531c8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62b7fb545233925f253531c8/W50u2G1HK3EtUKHRU189V.jpeg","isPro":false,"fullname":"Terry Yue Zhuo","user":"terryyz","type":"user"},"name":"Terry Yue Zhuo","status":"claimed_verified","statusLastChangedAt":"2025-08-27T07:12:32.236Z","hidden":false},{"_id":"673360a07c79bb4d069dcc5d","user":{"_id":"64b04320e5000ae8a57570a7","avatarUrl":"/avatars/226f57ea23b9dd861e71b7ba7821adce.svg","isPro":false,"fullname":"Massimo Caccia","user":"optimass","type":"user"},"name":"Massimo Caccia","status":"claimed_verified","statusLastChangedAt":"2024-12-13T15:23:25.890Z","hidden":false}],"publishedAt":"2024-11-05T23:34:06.000Z","submittedOnDailyAt":"2024-11-12T11:37:05.498Z","title":"GitChameleon: Unmasking the Version-Switching Capabilities of Code\n Generation Models","submittedOnDailyBy":{"_id":"64b04320e5000ae8a57570a7","avatarUrl":"/avatars/226f57ea23b9dd861e71b7ba7821adce.svg","isPro":false,"fullname":"Massimo Caccia","user":"optimass","type":"user"},"summary":"The rapid evolution of software libraries presents a significant challenge\nfor code generation models, which must adapt to frequent version updates while\nmaintaining compatibility with previous versions. Existing code completion\nbenchmarks often overlook this dynamic aspect, and the one that does consider\nit relies on static code prediction tasks without execution-based evaluation,\noffering a limited perspective on a model's practical usability. To address\nthis gap, we introduce \\GitChameleon{}, a novel, manually curated\ndataset comprising 116 Python code completion problems, each conditioned on\nspecific library versions and accompanied by executable unit tests.\n is designed to rigorously assess the ability of modern large\nlanguage models (LLMs) to generate version-specific code that is not only\nsyntactically correct but also functionally accurate upon execution. Our\ncomprehensive evaluations reveal that state-of-the-art LLMs struggle with this\ntask; for instance, GPT-4o achieves a pass@10 of only 39.9\\% (43.7\\%\nwhen provided with error feedback), highlighting the complexity of the problem\nand the limitations of current models. By providing an execution-based\nbenchmark that emphasizes the dynamic nature of code libraries, \nserves as a critical tool to advance the development of more adaptable and\nreliable code generation models. For facilitation for further exploration of\nversion-conditioned code generation, we make our code repository publicly\naccessible at https://github.com/NizarIslah/GitChameleon.","upvotes":21,"discussionId":"673360a07c79bb4d069dcca5","githubRepo":"https://github.com/nizarislah/gitchameleon","githubRepoAddedBy":"auto","ai_summary":"A new dataset, GitChameleon, evaluates large language models' ability to generate version-specific and executable code, highlighting their limitations.","ai_keywords":["code generation models","large language models","code completion","GitChameleon","version-specific code","executable unit tests","pass@10","GPT-4o"],"githubStars":14},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"64b04320e5000ae8a57570a7","avatarUrl":"/avatars/226f57ea23b9dd861e71b7ba7821adce.svg","isPro":false,"fullname":"Massimo Caccia","user":"optimass","type":"user"},{"_id":"6481fafd7046ce730082d772","avatarUrl":"/avatars/5acab1efd358e1ebdd4283b9b0ff31ea.svg","isPro":false,"fullname":"Sahar Shayegan","user":"saharshayegan","type":"user"},{"_id":"64467562f27198480f196aa6","avatarUrl":"/avatars/7c1ab898671056bde4af4b84c5e9a08e.svg","isPro":false,"fullname":"Alexandre Lacoste","user":"recursix","type":"user"},{"_id":"643006f01572f43a481766a9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/643006f01572f43a481766a9/egLlzmsWxHovmuvqAPnBO.jpeg","isPro":false,"fullname":"_","user":"Xa9aX","type":"user"},{"_id":"627a03b48bcc29b8edf5bc65","avatarUrl":"/avatars/ff633d7323d0da133b06be784be5e887.svg","isPro":false,"fullname":"Megh Thakkar","user":"megh1211","type":"user"},{"_id":"61bcebe9351aea330acb5b1f","avatarUrl":"/avatars/d054e80e3b1635c0f5664c75e52d5490.svg","isPro":false,"fullname":"Bahdanau","user":"Dzmitry","type":"user"},{"_id":"6442a3e53610a28a4ad49075","avatarUrl":"/avatars/bf2149a2520d89ecd379aff05cdd8eeb.svg","isPro":false,"fullname":"Maxime Gasse","user":"gasse","type":"user"},{"_id":"6304061c0547362a22a76a17","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1661339692442-6304061c0547362a22a76a17.jpeg","isPro":false,"fullname":"Jean Kaddour","user":"JeanKaddour","type":"user"},{"_id":"607f060442beb4da0f990182","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/607f060442beb4da0f990182/j5W2tLyU6JqkaTf3kv66s.jpeg","isPro":false,"fullname":"Patrice Bechard","user":"patricebechard","type":"user"},{"_id":"62cee0be26c94b14317066ef","avatarUrl":"/avatars/68cd573b9d5bb7d2467c4c5ae21570d6.svg","isPro":false,"fullname":"Christian Hudon","user":"chrish42","type":"user"},{"_id":"65b25e23e312613c60ef4f84","avatarUrl":"/avatars/6a2b29ea4d96def0b8ffa52e7bf3b86c.svg","isPro":false,"fullname":"Arjun Ashok","user":"arjunashok","type":"user"},{"_id":"6555125a4f361968f0e3aad7","avatarUrl":"/avatars/e7692d82804338f21ecdc6e731f5c5ea.svg","isPro":false,"fullname":"marinaretikof","user":"marinaretik","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0}">

Papers

arxiv:2411.05830

GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Models

Published on Nov 5, 2024

· Submitted by

Massimo Caccia on Nov 12, 2024

Upvote

Authors:

Justine Gehring ,

Terry Yue Zhuo ,

Massimo Caccia

Abstract

A new dataset, GitChameleon, evaluates large language models' ability to generate version-specific and executable code, highlighting their limitations.

AI-generated summary

The rapid evolution of software libraries presents a significant challenge for code generation models, which must adapt to frequent version updates while maintaining compatibility with previous versions. Existing code completion benchmarks often overlook this dynamic aspect, and the one that does consider it relies on static code prediction tasks without execution-based evaluation, offering a limited perspective on a model's practical usability. To address this gap, we introduce \GitChameleon{}, a novel, manually curated dataset comprising 116 Python code completion problems, each conditioned on specific library versions and accompanied by executable unit tests. is designed to rigorously assess the ability of modern large language models (LLMs) to generate version-specific code that is not only syntactically correct but also functionally accurate upon execution. Our comprehensive evaluations reveal that state-of-the-art LLMs struggle with this task; for instance, GPT-4o achieves a pass@10 of only 39.9\% (43.7\% when provided with error feedback), highlighting the complexity of the problem and the limitations of current models. By providing an execution-based benchmark that emphasizes the dynamic nature of code libraries, serves as a critical tool to advance the development of more adaptable and reliable code generation models. For facilitation for further exploration of version-conditioned code generation, we make our code repository publicly accessible at https://github.com/NizarIslah/GitChameleon.