https://github.com/bigai-ai/ICE.\n","updatedAt":"2024-07-19T04:08:53.749Z","author":{"_id":"6483ec5fe2f6f33be175c5c3","avatarUrl":"/avatars/6372d1139c65ed976e5e558654821649.svg","fullname":"Siyuan Qi","name":"syqi","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.894311249256134},"editors":["syqi"],"editorAvatarUrls":["/avatars/6372d1139c65ed976e5e558654821649.svg"],"reactions":[],"isReport":false},"replies":[{"id":"66722c930f6ec76f33351101","author":{"_id":"5ead1b914e876668a0c37772","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/5ead1b914e876668a0c37772/ftW3bs6hy2Q_J63_OUKKW.png","fullname":"PenutChen","name":"penut85420","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false},"createdAt":"2024-06-19T00:55:47.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Looks great! Any plans for the code release?","html":"
Looks great! Any plans for the code release?
\n","updatedAt":"2024-06-19T00:55:47.474Z","author":{"_id":"5ead1b914e876668a0c37772","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/5ead1b914e876668a0c37772/ftW3bs6hy2Q_J63_OUKKW.png","fullname":"PenutChen","name":"penut85420","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8984746932983398},"editors":["penut85420"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/5ead1b914e876668a0c37772/ftW3bs6hy2Q_J63_OUKKW.png"],"reactions":[],"isReport":false,"parentCommentId":"667192c328cd551050adc281"}},{"id":"6673797d7cdc8da5ae7d1e12","author":{"_id":"6483ec5fe2f6f33be175c5c3","avatarUrl":"/avatars/6372d1139c65ed976e5e558654821649.svg","fullname":"Siyuan Qi","name":"syqi","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false},"createdAt":"2024-06-20T00:36:13.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Yes, we will be releasing the code and update the paper with link afterwards!","html":"Yes, we will be releasing the code and update the paper with link afterwards!
\n","updatedAt":"2024-06-20T00:36:13.879Z","author":{"_id":"6483ec5fe2f6f33be175c5c3","avatarUrl":"/avatars/6372d1139c65ed976e5e558654821649.svg","fullname":"Siyuan Qi","name":"syqi","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8968603610992432},"editors":["syqi"],"editorAvatarUrls":["/avatars/6372d1139c65ed976e5e558654821649.svg"],"reactions":[{"reaction":"🤗","users":["penut85420","Yofuria"],"count":2}],"isReport":false,"parentCommentId":"667192c328cd551050adc281"}},{"id":"669936a0caf454d074dd5bfb","author":{"_id":"6483ec5fe2f6f33be175c5c3","avatarUrl":"/avatars/6372d1139c65ed976e5e558654821649.svg","fullname":"Siyuan Qi","name":"syqi","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false},"createdAt":"2024-07-18T15:37:04.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"The code has been released at: https://github.com/bigai-ai/ICE. Thanks for your interest!","html":"The code has been released at: https://github.com/bigai-ai/ICE. Thanks for your interest!
\n","updatedAt":"2024-07-18T15:37:04.092Z","author":{"_id":"6483ec5fe2f6f33be175c5c3","avatarUrl":"/avatars/6372d1139c65ed976e5e558654821649.svg","fullname":"Siyuan Qi","name":"syqi","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9371434450149536},"editors":["syqi"],"editorAvatarUrls":["/avatars/6372d1139c65ed976e5e558654821649.svg"],"reactions":[{"reaction":"🤗","users":["penut85420"],"count":1}],"isReport":false,"parentCommentId":"667192c328cd551050adc281"}},{"id":"6699b019004e418e7134d964","author":{"_id":"5ead1b914e876668a0c37772","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/5ead1b914e876668a0c37772/ftW3bs6hy2Q_J63_OUKKW.png","fullname":"PenutChen","name":"penut85420","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false},"createdAt":"2024-07-19T00:15:21.000Z","type":"comment","data":{"edited":true,"hidden":false,"latest":{"raw":"@syqi fantastic! I think the link should be edited into the first comment so it won't be hidden.","html":"\n\n@syqi\n\t fantastic! I think the link should be edited into the first comment so it won't be hidden.
\n","updatedAt":"2024-07-19T00:15:40.392Z","author":{"_id":"5ead1b914e876668a0c37772","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/5ead1b914e876668a0c37772/ftW3bs6hy2Q_J63_OUKKW.png","fullname":"PenutChen","name":"penut85420","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.8933727741241455},"editors":["penut85420"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/5ead1b914e876668a0c37772/ftW3bs6hy2Q_J63_OUKKW.png"],"reactions":[],"isReport":false,"parentCommentId":"667192c328cd551050adc281"}}]}],"primaryEmailConfirmed":false,"paper":{"id":"2406.11194","authors":[{"_id":"66719193202196a178f8d501","user":{"_id":"6483ec5fe2f6f33be175c5c3","avatarUrl":"/avatars/6372d1139c65ed976e5e558654821649.svg","isPro":false,"fullname":"Siyuan Qi","user":"syqi","type":"user"},"name":"Siyuan Qi","status":"claimed_verified","statusLastChangedAt":"2024-06-18T14:13:55.628Z","hidden":false},{"_id":"66719193202196a178f8d502","user":{"_id":"63ece843c8827dd0f0f7b8c4","avatarUrl":"/avatars/e6c5840510ab88b007183cf79ed1bdf9.svg","isPro":false,"fullname":"Bangcheng Yang","user":"dumbmouse","type":"user"},"name":"Bangcheng Yang","status":"claimed_verified","statusLastChangedAt":"2024-07-02T07:39:42.252Z","hidden":false},{"_id":"66719193202196a178f8d503","user":{"_id":"65745569839aa08899ea5d27","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4X8waDwiphbfKZySrYlFy.jpeg","isPro":false,"fullname":"Kailin Jiang, 蒋凯林","user":"kailinjiang","type":"user"},"name":"Kailin Jiang","status":"claimed_verified","statusLastChangedAt":"2024-06-19T20:21:01.784Z","hidden":false},{"_id":"66719193202196a178f8d504","user":{"_id":"65e933fcacb29a596cd6eb6e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65e933fcacb29a596cd6eb6e/dlH14rO2I4EUMVyjEjdRK.jpeg","isPro":false,"fullname":"Xiaobo Wang","user":"Yofuria","type":"user"},"name":"Xiaobo Wang","status":"claimed_verified","statusLastChangedAt":"2024-06-19T20:20:59.856Z","hidden":false},{"_id":"66719193202196a178f8d505","name":"Jiaqi Li","hidden":false},{"_id":"66719193202196a178f8d506","name":"Yifan Zhong","hidden":false},{"_id":"66719193202196a178f8d507","name":"Yaodong Yang","hidden":false},{"_id":"66719193202196a178f8d508","user":{"_id":"63a95a6a7930fa8c7dd63d4e","avatarUrl":"/avatars/d9d0420f7ddfe2f3a7e029fb05f1c89f.svg","isPro":false,"fullname":"Zilong Zheng","user":"zlzheng","type":"user"},"name":"Zilong Zheng","status":"claimed_verified","statusLastChangedAt":"2024-06-19T07:39:17.582Z","hidden":false}],"publishedAt":"2024-06-17T04:00:04.000Z","submittedOnDailyAt":"2024-06-18T12:29:31.813Z","title":"In-Context Editing: Learning Knowledge from Self-Induced Distributions","submittedOnDailyBy":{"_id":"6483ec5fe2f6f33be175c5c3","avatarUrl":"/avatars/6372d1139c65ed976e5e558654821649.svg","isPro":false,"fullname":"Siyuan Qi","user":"syqi","type":"user"},"summary":"The existing fine-tuning paradigm for language models is brittle in knowledge\nediting scenarios, where the model must incorporate new information without\nextensive retraining. This brittleness often results in overfitting, reduced\nperformance, and unnatural language generation. To address this, we propose\nConsistent In-Context Editing (ICE), a novel approach that leverages the\nmodel's in-context learning capability to tune toward a contextual distribution\nrather than a one-hot target. ICE introduces a straightforward optimization\nframework that includes both a target and a procedure, enhancing the robustness\nand effectiveness of gradient-based tuning methods. We provide analytical\ninsights into ICE across four critical aspects of knowledge editing: accuracy,\nlocality, generalization, and linguistic quality, showing its advantages.\nExperimental results across four datasets confirm the effectiveness of ICE and\ndemonstrate its potential for continual editing, ensuring that updated\ninformation is incorporated while preserving the integrity of the model.","upvotes":20,"discussionId":"66719194202196a178f8d53a","githubRepo":"https://github.com/bigai-ai/ICE","githubRepoAddedBy":"user","ai_summary":"Consistent In-Context Editing leverages in-context learning to robustly fine-tune language models, improving knowledge editing by maintaining accuracy, locality, generalization, and linguistic quality.","ai_keywords":["fine-tuning","knowledge editing","Consistent In-Context Editing","in-context learning","gradient-based tuning"],"githubStars":48},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6483ec5fe2f6f33be175c5c3","avatarUrl":"/avatars/6372d1139c65ed976e5e558654821649.svg","isPro":false,"fullname":"Siyuan Qi","user":"syqi","type":"user"},{"_id":"61344ab1d19e49a35751462e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1630816930903-noauth.jpeg","isPro":false,"fullname":"Puffy Bird","user":"puffy310","type":"user"},{"_id":"655ac762cb17ec19ef82719b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/655ac762cb17ec19ef82719b/1kDncYrGLYS_2SR8cNdAL.png","isPro":false,"fullname":"Welcome to matlok","user":"matlok","type":"user"},{"_id":"5ead1b914e876668a0c37772","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/5ead1b914e876668a0c37772/ftW3bs6hy2Q_J63_OUKKW.png","isPro":false,"fullname":"PenutChen","user":"penut85420","type":"user"},{"_id":"63a95a6a7930fa8c7dd63d4e","avatarUrl":"/avatars/d9d0420f7ddfe2f3a7e029fb05f1c89f.svg","isPro":false,"fullname":"Zilong Zheng","user":"zlzheng","type":"user"},{"_id":"64d98ef7a4839890b25eb78b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64d98ef7a4839890b25eb78b/215-CSVLl81z6CAq0ECWU.jpeg","isPro":true,"fullname":"Fangyuan Yu","user":"Ksgk-fy","type":"user"},{"_id":"65745569839aa08899ea5d27","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4X8waDwiphbfKZySrYlFy.jpeg","isPro":false,"fullname":"Kailin Jiang, 蒋凯林","user":"kailinjiang","type":"user"},{"_id":"65e933fcacb29a596cd6eb6e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65e933fcacb29a596cd6eb6e/dlH14rO2I4EUMVyjEjdRK.jpeg","isPro":false,"fullname":"Xiaobo Wang","user":"Yofuria","type":"user"},{"_id":"655d9f43b5da99edaf3f2f81","avatarUrl":"/avatars/c7225b3ed54d099a4fd87682427fb5bf.svg","isPro":false,"fullname":"Yifan Zhong","user":"Yifan-Zhong","type":"user"},{"_id":"6672ff2c00f5c86be95459b5","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6672ff2c00f5c86be95459b5/NQBNyTZOk27wL2lZwz19w.jpeg","isPro":false,"fullname":"Ning Jiang","user":"flozou","type":"user"},{"_id":"65331f72b3852ed1ce9c5c06","avatarUrl":"/avatars/b2704f0820ca1f2a561742c978ce75e4.svg","isPro":false,"fullname":"bigainlco","user":"bigainlco","type":"user"},{"_id":"6538119803519fddb4a17e10","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6538119803519fddb4a17e10/ffJMkdx-rM7VvLTCM6ri_.jpeg","isPro":false,"fullname":"samusenps","user":"samusenps","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0}">
Papers
arxiv:2406.11194
In-Context Editing: Learning Knowledge from Self-Induced Distributions
Published on Jun 17, 2024
Abstract
Consistent In-Context Editing leverages in-context learning to robustly fine-tune language models, improving knowledge editing by maintaining accuracy, locality, generalization, and linguistic quality.
AI-generated summary
The existing fine-tuning paradigm for language models is brittle in knowledge editing scenarios, where the model must incorporate new information without extensive retraining. This brittleness often results in overfitting, reduced performance, and unnatural language generation. To address this, we propose Consistent In-Context Editing (ICE), a novel approach that leverages the model's in-context learning capability to tune toward a contextual distribution rather than a one-hot target. ICE introduces a straightforward optimization framework that includes both a target and a procedure, enhancing the robustness and effectiveness of gradient-based tuning methods. We provide analytical insights into ICE across four critical aspects of knowledge editing: accuracy, locality, generalization, and linguistic quality, showing its advantages. Experimental results across four datasets confirm the effectiveness of ICE and demonstrate its potential for continual editing, ensuring that updated information is incorporated while preserving the integrity of the model.
Community
This paper proposes Consistent In-Context Editing (ICE), an approach for tuning language models through contextual distributions, overcoming the limitations of traditional fine-tuning methods that learn towards one-hot targets.
The code is available at: https://github.com/bigai-ai/ICE.
Looks great! Any plans for the code release?
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2406.11194 in a model README.md to link it from this page.
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2406.11194 in a Space README.md to link it from this page.