PapersWithCode

\n\n","updatedAt":"2023-10-30T16:59:14.574Z","author":{"_id":"639f61c02e13e54dcbbd7fc0","avatarUrl":"/avatars/58a0b486fd0bec90887388ec9bf26b17.svg","fullname":"Avneesh Mishra","name":"TheProjectsGuy","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.780333936214447},"editors":["TheProjectsGuy"],"editorAvatarUrls":["/avatars/58a0b486fd0bec90887388ec9bf26b17.svg"],"reactions":[],"isReport":false}},{"id":"653fe103e8a36680111f2740","author":{"_id":"647e1d277f9ad5e44ba9dec3","avatarUrl":"/avatars/ba0e338de50f4a271352eb7996200e53.svg","fullname":"Matthew Ratzke","name":"degree9","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false},"createdAt":"2023-10-30T16:59:47.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Has anyone made a model like this available?","html":"

Has anyone made a model like this available?

\n","updatedAt":"2023-10-30T16:59:47.944Z","author":{"_id":"647e1d277f9ad5e44ba9dec3","avatarUrl":"/avatars/ba0e338de50f4a271352eb7996200e53.svg","fullname":"Matthew Ratzke","name":"degree9","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9751766324043274},"editors":["degree9"],"editorAvatarUrls":["/avatars/ba0e338de50f4a271352eb7996200e53.svg"],"reactions":[{"reaction":"👍","users":["mrfakename","MichaelBarryUK","Irrelative"],"count":3}],"isReport":false}},{"id":"65407ebd32dcbb86630a42d8","author":{"_id":"646c1855ec9a61e8717f51a9","avatarUrl":"/avatars/f40f94c75cd27625c4420d55c18cb6a3.svg","fullname":"connor power","name":"connnor","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false},"createdAt":"2023-10-31T04:12:45.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"gimme","html":"

gimme

\n","updatedAt":"2023-10-31T04:12:45.312Z","author":{"_id":"646c1855ec9a61e8717f51a9","avatarUrl":"/avatars/f40f94c75cd27625c4420d55c18cb6a3.svg","fullname":"connor power","name":"connnor","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"fr","probability":0.346319317817688},"editors":["connnor"],"editorAvatarUrls":["/avatars/f40f94c75cd27625c4420d55c18cb6a3.svg"],"reactions":[],"isReport":false}},{"id":"654250bb916d9f8b55edb814","author":{"_id":"6142b22afef98da69fbe3092","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1631760874860-noauth.jpeg","fullname":"beyond","name":"beyond","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":6,"isUserFollowing":false},"createdAt":"2023-11-01T13:20:59.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"I wondering if the 20B is a typo.\nIf its true, it should be a BIG news.","html":"

I wondering if the 20B is a typo.
If its true, it should be a BIG news.

\n","updatedAt":"2023-11-01T13:20:59.323Z","author":{"_id":"6142b22afef98da69fbe3092","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1631760874860-noauth.jpeg","fullname":"beyond","name":"beyond","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":6,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8120687007904053},"editors":["beyond"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1631760874860-noauth.jpeg"],"reactions":[{"reaction":"👍","users":["MichaelBarryUK"],"count":1}],"isReport":false}},{"id":"65427dcd4f89f533be6922b6","author":{"_id":"644e1b1d9b4e87c31bab0a14","avatarUrl":"/avatars/88bb4c4a67dc8958069e9014f5e73a0b.svg","fullname":"Michael Barry","name":"MichaelBarryUK","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":7,"isUserFollowing":false},"createdAt":"2023-11-01T16:33:17.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"> I wondering if the 20B is a typo.\n> If its true, it should be a BIG news.\n\nMaybe it's three open source 7B models stuck together and Microsoft is trolling them 😂","html":"

\n
I wondering if the 20B is a typo.
If its true, it should be a BIG news.
\n

Maybe it's three open source 7B models stuck together and Microsoft is trolling them 😂

\n","updatedAt":"2023-11-01T16:33:17.625Z","author":{"_id":"644e1b1d9b4e87c31bab0a14","avatarUrl":"/avatars/88bb4c4a67dc8958069e9014f5e73a0b.svg","fullname":"Michael Barry","name":"MichaelBarryUK","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":7,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9134376645088196},"editors":["MichaelBarryUK"],"editorAvatarUrls":["/avatars/88bb4c4a67dc8958069e9014f5e73a0b.svg"],"reactions":[],"isReport":false}},{"id":"6542fefe9698c3b984c18af0","author":{"_id":"62e54f0eae9d3f10acb95cb9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62e54f0eae9d3f10acb95cb9/VAyk05hqB3OZWXEZW-B0q.png","fullname":"mrfakename","name":"mrfakename","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3477,"isUserFollowing":false},"createdAt":"2023-11-02T01:44:30.000Z","type":"comment","data":{"edited":true,"hidden":false,"latest":{"raw":"The paper just got withdrawn... Anyone have an archive?\n\nEDIT: Found it on [archive.org](https://web.archive.org/web/20231030073415/https://arxiv.org/pdf/2310.17680.pdf).","html":"

The paper just got withdrawn... Anyone have an archive?

EDIT: Found it on archive.org.

\n","updatedAt":"2023-11-02T01:46:26.555Z","author":{"_id":"62e54f0eae9d3f10acb95cb9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62e54f0eae9d3f10acb95cb9/VAyk05hqB3OZWXEZW-B0q.png","fullname":"mrfakename","name":"mrfakename","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3477,"isUserFollowing":false}},"numEdits":2,"identifiedLanguage":{"language":"en","probability":0.9868218302726746},"editors":["mrfakename"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/62e54f0eae9d3f10acb95cb9/VAyk05hqB3OZWXEZW-B0q.png"],"reactions":[],"isReport":false}},{"id":"65432eba5a36a8774d09c8f3","author":{"_id":"631707a496add455ce172755","avatarUrl":"/avatars/57ff04c8a8f8ba9e785059a414a4cb74.svg","fullname":"zeng xian","name":"themez","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false},"createdAt":"2023-11-02T05:08:10.000Z","type":"comment","data":{"edited":true,"hidden":false,"latest":{"raw":"> The paper just got withdrawn... Anyone have an archive?\n> \n> EDIT: Found it on [archive.org](https://web.archive.org/web/20231030073415/https://arxiv.org/pdf/2310.17680.pdf).\n\nYou can also check previous version: https://arxiv.org/abs/2310.17680v1","html":"

\n
The paper just got withdrawn... Anyone have an archive?
\n
EDIT: Found it on archive.org.
\n

You can also check previous version: https://arxiv.org/abs/2310.17680v1

\n","updatedAt":"2023-11-02T05:08:36.316Z","author":{"_id":"631707a496add455ce172755","avatarUrl":"/avatars/57ff04c8a8f8ba9e785059a414a4cb74.svg","fullname":"zeng xian","name":"themez","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.8867107629776001},"editors":["themez"],"editorAvatarUrls":["/avatars/57ff04c8a8f8ba9e785059a414a4cb74.svg"],"reactions":[{"reaction":"👍","users":["Citaman","mrfakename"],"count":2}],"isReport":false}},{"id":"654523793be12fed568d86b5","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2023-11-03T16:44:41.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [SteloCoder: a Decoder-Only LLM for Multi-Language to Python Code Translation](https://huggingface.co/papers/2310.15539) (2023)\n* [T5APR: Empowering Automated Program Repair across Languages through Checkpoint Ensemble](https://huggingface.co/papers/2309.15742) (2023)\n* [CAT-LM: Training Language Models on Aligned Code And Tests](https://huggingface.co/papers/2310.01602) (2023)\n* [InstructCoder: Empowering Language Models for Code Editing](https://huggingface.co/papers/2310.20329) (2023)\n* [InstructExcel: A Benchmark for Natural Language Instruction in Excel](https://huggingface.co/papers/2310.14495) (2023)\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space","html":"

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

\n","updatedAt":"2023-11-03T16:44:41.355Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7192511558532715},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2310.17680","authors":[{"_id":"653f192eec992c5c209bab70","user":{"_id":"63ea298fb1d9c4ef71eab282","avatarUrl":"/avatars/6a0d3b1678e2cf6b0356d0495530e045.svg","isPro":false,"fullname":"Mukul Singh","user":"MukulSingh105","type":"user"},"name":"Mukul Singh","status":"claimed_verified","statusLastChangedAt":"2023-10-30T13:12:07.622Z","hidden":false},{"_id":"653f192eec992c5c209bab71","name":"José Cambronero","hidden":false},{"_id":"653f192eec992c5c209bab72","name":"Sumit Gulwani","hidden":false},{"_id":"653f192eec992c5c209bab73","name":"Vu Le","hidden":false},{"_id":"653f192eec992c5c209bab74","name":"Carina Negreanu","hidden":false},{"_id":"653f192eec992c5c209bab75","name":"Gust Verbruggen","hidden":false}],"publishedAt":"2023-10-26T11:06:15.000Z","submittedOnDailyAt":"2023-10-30T01:17:10.682Z","title":"CodeFusion: A Pre-trained Diffusion Model for Code Generation","submittedOnDailyBy":{"_id":"60f1abe7544c2adfd699860c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674929746905-60f1abe7544c2adfd699860c.jpeg","isPro":false,"fullname":"AK","user":"akhaliq","type":"user"},"summary":"Imagine a developer who can only change their last line of code, how often\nwould they have to start writing a function from scratch before it is correct?\nAuto-regressive models for code generation from natural language have a similar\nlimitation: they do not easily allow reconsidering earlier tokens generated. We\nintroduce CodeFusion, a pre-trained diffusion code generation model that\naddresses this limitation by iteratively denoising a complete program\nconditioned on the encoded natural language. We evaluate CodeFusion on the task\nof natural language to code generation for Bash, Python, and Microsoft Excel\nconditional formatting (CF) rules. Experiments show that CodeFusion (75M\nparameters) performs on par with state-of-the-art auto-regressive systems\n(350M-175B parameters) in top-1 accuracy and outperforms them in top-3 and\ntop-5 accuracy due to its better balance in diversity versus quality.","upvotes":74,"discussionId":"653f192eec992c5c209bab7e","ai_summary":"CodeFusion, a diffusion model for code generation, outperforms auto-regressive models in natural language to code tasks by iteratively refining the entire program.","ai_keywords":["diffusion code generation model","iteratively denoising","natural language to code generation","Bash","Python","Microsoft Excel conditional formatting (CF) rules","top-1 accuracy","top-3 accuracy","top-5 accuracy","diversity versus quality"]},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"64ac17875870653ec8e42cfe","avatarUrl":"/avatars/5346146aca9add2e6c02b01bf4c0df2e.svg","isPro":false,"fullname":"ryuhei horiuchi","user":"horyujey","type":"user"},{"_id":"6448d002e87a77e872e4489e","avatarUrl":"/avatars/c280eb1b59bb2198e44db93f45adad6d.svg","isPro":false,"fullname":"JIANG Xiaoyun","user":"Shay1216","type":"user"},{"_id":"644e1b1d9b4e87c31bab0a14","avatarUrl":"/avatars/88bb4c4a67dc8958069e9014f5e73a0b.svg","isPro":false,"fullname":"Michael Barry","user":"MichaelBarryUK","type":"user"},{"_id":"631e40a4b07bce5f2a7faa38","avatarUrl":"/avatars/999e8f6a820d7145b60a99eb16bd7f35.svg","isPro":false,"fullname":"Kalu","user":"ogkalu","type":"user"},{"_id":"60c8d264224e250fb0178f77","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/60c8d264224e250fb0178f77/i8fbkBVcoFeJRmkQ9kYAE.png","isPro":false,"fullname":"Adam Lee","user":"Abecid","type":"user"},{"_id":"6454d12b9d37c3fb332637bd","avatarUrl":"/avatars/f9134941b077c8c5e282558158404d7a.svg","isPro":false,"fullname":"VS","user":"Sumanthvs","type":"user"},{"_id":"63c4bab3a361002ba0c1ab5d","avatarUrl":"/avatars/62240854e6ed051ca86cbd24c4153b8e.svg","isPro":false,"fullname":"Mehmet Ali Peker","user":"peker","type":"user"},{"_id":"646b452d1da1b6d027fa17ba","avatarUrl":"/avatars/38b521ab18c16eaa23541054cb8d2f8e.svg","isPro":false,"fullname":"dongsokim","user":"dongso","type":"user"},{"_id":"630920925a5c889aaedc7f33","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/630920925a5c889aaedc7f33/w00N19M21l2FXe6ZasSYc.jpeg","isPro":false,"fullname":"Kristaller486","user":"kristaller486","type":"user"},{"_id":"6471ce7f021b8f492f062181","avatarUrl":"/avatars/d9ac851fd6810dd7862eaea47a8a9477.svg","isPro":false,"fullname":"Falcon","user":"FalconLlamalpaca","type":"user"},{"_id":"63e5f6f9f2e9a8f22c51a0d3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1676015280017-noauth.jpeg","isPro":true,"fullname":"Charles Koutcheme","user":"koutch","type":"user"},{"_id":"6311bca0ae8896941da24e66","avatarUrl":"/avatars/48de64894fc3c9397e26e4d6da3ff537.svg","isPro":false,"fullname":"Fynn Kröger","user":"fynnkroeger","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":1}">

Papers

arxiv:2310.17680

CodeFusion: A Pre-trained Diffusion Model for Code Generation

Published on Oct 26, 2023

· Submitted by

AK on Oct 30, 2023

#1 Paper of the day

Upvote

Authors:

Mukul Singh ,

Abstract

CodeFusion, a diffusion model for code generation, outperforms auto-regressive models in natural language to code tasks by iteratively refining the entire program.

AI-generated summary

Imagine a developer who can only change their last line of code, how often would they have to start writing a function from scratch before it is correct? Auto-regressive models for code generation from natural language have a similar limitation: they do not easily allow reconsidering earlier tokens generated. We introduce CodeFusion, a pre-trained diffusion code generation model that addresses this limitation by iteratively denoising a complete program conditioned on the encoded natural language. We evaluate CodeFusion on the task of natural language to code generation for Bash, Python, and Microsoft Excel conditional formatting (CF) rules. Experiments show that CodeFusion (75M parameters) performs on par with state-of-the-art auto-regressive systems (350M-175B parameters) in top-1 accuracy and outperforms them in top-3 and top-5 accuracy due to its better balance in diversity versus quality.

View arXiv page View PDF Add to collection

Community

MichaelBarryUK

Oct 30, 2023

•

edited Oct 30, 2023

Amazing. Also, it would be intriguing to see the code being non-linearly inferred, would make for some interesting UI effects. Imagine the green rain matrix effect morphing into real source code

Also, according to this paper, gpt3.5 has 20B parameters.

Drstone1

Oct 30, 2023

The code in python for deleting unwanted cookie

TheProjectsGuy

Oct 30, 2023

Proposes CodeFusion: code generation model from diffusion (combined with an encoder-decoder model), conditioned on natural language. Diffusion for text: embedding layer to convert discrete tokens to continuous embeddings, then denoise, then retrieve closest discrete enbedding. Architecture has encoder, diffusion, decoder, and classification head (for code tokens). Two stage training: unsupervised pretraining of denoiser and decoder, and supervised (utterance, code) pairs fine-tuning for encoder, denoiser, and decoder. Loss adapted from GENIE. Benchmarked on Python (CoNaLa), Bash, and conditional rules in MS Excel. Encoder initialized from CodeT5. Better performance than StarCoder, CodeT5+, and GPT-3.5 (Python CodeBERT, bash template, and CF rule execution); also generates more diverse outputs. Appendix has implementation and training details, baseline details, visualization of diffusion process (code with time step), and background (auto regression and diffusion). From Microsoft.