$\"1715413558560.jpg\"$

\n","updatedAt":"2024-05-12T05:31:09.069Z","author":{"_id":"64348395546e16f17a1667f0","avatarUrl":"/avatars/439c3504c964c6196f207d8b26755881.svg","fullname":"Hosen ahmad","name":"Hosen236","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.3206782937049866},"editors":["Hosen236"],"editorAvatarUrls":["/avatars/439c3504c964c6196f207d8b26755881.svg"],"reactions":[],"isReport":false}},{"id":"6664c5c3d8e1e34bc65bd284","author":{"_id":"6186ddf6a7717cb375090c01","avatarUrl":"/avatars/716b6a7d1094c8036b2a8a7b9063e8aa.svg","fullname":"Julien BLANCHON","name":"blanchon","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":176,"isUserFollowing":false},"createdAt":"2024-06-08T20:57:39.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"# Unlocking Precision: ControlNet in Text-to-Image Models\n\nhttps://cdn-uploads.huggingface.co/production/uploads/6186ddf6a7717cb375090c01/7Ei1uj4Ud-hUnzE0CPHA9.mp4 \n\n## Links 🔗:\n👉 Subscribe: https://www.youtube.com/@Arxflix\n👉 Twitter: https://x.com/arxflix\n👉 LMNT (Partner): https://lmnt.com/\n\n\nBy Arxflix\n![9t4iCUHx_400x400-1.jpg](https://cdn-uploads.huggingface.co/production/uploads/6186ddf6a7717cb375090c01/v4S5zBurs0ouGNwYj1GEd.jpeg)","html":"

\n\t\n\t\t\n\t\n\t\n\t\tUnlocking Precision: ControlNet in Text-to-Image Models\n\t\n

\n\n

\n\t\n\t\t\n\t\n\t\n\t\tLinks 🔗:\n\t\n

👉 Subscribe: https://www.youtube.com/@Arxflix
👉 Twitter: https://x.com/arxflix
👉 LMNT (Partner): https://lmnt.com/

By Arxflix
$\"9t4iCUHx_400x400-1.jpg\"$

\n","updatedAt":"2024-06-08T20:57:39.478Z","author":{"_id":"6186ddf6a7717cb375090c01","avatarUrl":"/avatars/716b6a7d1094c8036b2a8a7b9063e8aa.svg","fullname":"Julien BLANCHON","name":"blanchon","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":176,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.5026009678840637},"editors":["blanchon"],"editorAvatarUrls":["/avatars/716b6a7d1094c8036b2a8a7b9063e8aa.svg"],"reactions":[],"isReport":false}},{"id":"67c4f4a3661af6f84694c0cb","author":{"_id":"67c4597ed942609d9b599c43","avatarUrl":"/avatars/d08e9fdc875d1d90ad614390621c01ae.svg","fullname":"Aminjon","name":"Ameifbnydb","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false},"createdAt":"2025-03-03T00:15:31.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"(https://cdn-uploads.huggingface.co/production/uploads/67c4597ed942609d9b599c43/3Y4Ei5hvoxRj7n8QOVkk3.jpeg)\n","html":"

(https://cdn-uploads.huggingface.co/production/uploads/67c4597ed942609d9b599c43/3Y4Ei5hvoxRj7n8QOVkk3.jpeg)

\n","updatedAt":"2025-03-03T00:15:31.096Z","author":{"_id":"67c4597ed942609d9b599c43","avatarUrl":"/avatars/d08e9fdc875d1d90ad614390621c01ae.svg","fullname":"Aminjon","name":"Ameifbnydb","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.3395450711250305},"editors":["Ameifbnydb"],"editorAvatarUrls":["/avatars/d08e9fdc875d1d90ad614390621c01ae.svg"],"reactions":[],"isReport":false},"replies":[{"id":"67c4f53e7682522a1628079b","author":{"_id":"67c4597ed942609d9b599c43","avatarUrl":"/avatars/d08e9fdc875d1d90ad614390621c01ae.svg","fullname":"Aminjon","name":"Ameifbnydb","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false},"createdAt":"2025-03-03T00:18:06.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Please change the black coat to white t-shirt","html":"

Please change the black coat to white t-shirt

\n","updatedAt":"2025-03-03T00:18:06.816Z","author":{"_id":"67c4597ed942609d9b599c43","avatarUrl":"/avatars/d08e9fdc875d1d90ad614390621c01ae.svg","fullname":"Aminjon","name":"Ameifbnydb","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.708346962928772},"editors":["Ameifbnydb"],"editorAvatarUrls":["/avatars/d08e9fdc875d1d90ad614390621c01ae.svg"],"reactions":[],"isReport":false,"parentCommentId":"67c4f4a3661af6f84694c0cb"}},{"id":"67c9407e2b8eb29312e987bf","author":{"_id":"67b609f8eb9d024c862663d5","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/AokxufXaF70p3-KSj5yS9.png","fullname":"ALi Haider","name":"alihaider372","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false},"createdAt":"2025-03-06T06:28:14.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Api key","html":"

Api key

\n","updatedAt":"2025-03-06T06:28:14.691Z","author":{"_id":"67b609f8eb9d024c862663d5","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/AokxufXaF70p3-KSj5yS9.png","fullname":"ALi Haider","name":"alihaider372","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"hr","probability":0.283184677362442},"editors":["alihaider372"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/AokxufXaF70p3-KSj5yS9.png"],"reactions":[],"isReport":false,"parentCommentId":"67c4f4a3661af6f84694c0cb"}}]},{"id":"68a36b7d3173634f9472abd5","author":{"_id":"65d9fc2a0e6ad24551d87a1e","avatarUrl":"/avatars/3aedb9522cc3cd08349d654f523fd792.svg","fullname":"Grant Singleton","name":"grantsing","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":4,"isUserFollowing":false},"createdAt":"2025-08-18T18:05:49.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"arXiv explained breakdown of this paper 👉 https://arxivexplained.com/papers/adding-conditional-control-to-text-to-image-diffusion-models","html":"

arXiv explained breakdown of this paper 👉 https://arxivexplained.com/papers/adding-conditional-control-to-text-to-image-diffusion-models

\n","updatedAt":"2025-08-18T18:05:49.754Z","author":{"_id":"65d9fc2a0e6ad24551d87a1e","avatarUrl":"/avatars/3aedb9522cc3cd08349d654f523fd792.svg","fullname":"Grant Singleton","name":"grantsing","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":4,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.6734464764595032},"editors":["grantsing"],"editorAvatarUrls":["/avatars/3aedb9522cc3cd08349d654f523fd792.svg"],"reactions":[],"isReport":false}},{"id":"68be545e0b54503c33c939c7","author":{"_id":"63325388fece0490c3d2fc27","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63325388fece0490c3d2fc27/vTuByof5vuac7bJKhTAz0.png","fullname":"shuxun","name":"shuxunoo","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false},"createdAt":"2025-09-08T03:58:22.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"nice work！","html":"

nice work！

\n","updatedAt":"2025-09-08T03:58:22.730Z","author":{"_id":"63325388fece0490c3d2fc27","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63325388fece0490c3d2fc27/vTuByof5vuac7bJKhTAz0.png","fullname":"shuxun","name":"shuxunoo","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.5807498693466187},"editors":["shuxunoo"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/63325388fece0490c3d2fc27/vTuByof5vuac7bJKhTAz0.png"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2302.05543","authors":[{"_id":"6411c7826b75ddced3891c12","user":{"_id":"639c1572445b133a4e9b3a3f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1671173450971-noauth.jpeg","isPro":true,"fullname":"Lvmin Zhang","user":"lllyasviel","type":"user"},"name":"Lvmin Zhang","status":"admin_assigned","statusLastChangedAt":"2023-10-16T15:03:17.455Z","hidden":false},{"_id":"652cf01962b6582c237f3349","user":{"_id":"63f8130749569335b679af62","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63f8130749569335b679af62/vgTu23-y0UKocwAGqNMwT.jpeg","isPro":false,"fullname":"Anyi Rao","user":"anyirao","type":"user"},"name":"Anyi Rao","status":"admin_assigned","statusLastChangedAt":"2023-10-16T14:49:24.580Z","hidden":false},{"_id":"6411c7826b75ddced3891c13","user":{"_id":"642bb3e44f379cdc3deb26cd","avatarUrl":"/avatars/556e792f013bcbe3748d7c9c719070d8.svg","isPro":false,"fullname":"Maneesh Agrawala","user":"magrawala","type":"user"},"name":"Maneesh Agrawala","status":"admin_assigned","statusLastChangedAt":"2023-10-16T14:49:15.910Z","hidden":false}],"publishedAt":"2023-02-10T23:12:37.000Z","title":"Adding Conditional Control to Text-to-Image Diffusion Models","summary":"We present ControlNet, a neural network architecture to add spatial\nconditioning controls to large, pretrained text-to-image diffusion models.\nControlNet locks the production-ready large diffusion models, and reuses their\ndeep and robust encoding layers pretrained with billions of images as a strong\nbackbone to learn a diverse set of conditional controls. The neural\narchitecture is connected with \"zero convolutions\" (zero-initialized\nconvolution layers) that progressively grow the parameters from zero and ensure\nthat no harmful noise could affect the finetuning. We test various conditioning\ncontrols, eg, edges, depth, segmentation, human pose, etc, with Stable\nDiffusion, using single or multiple conditions, with or without prompts. We\nshow that the training of ControlNets is robust with small (<50k) and large\n(>1m) datasets. Extensive results show that ControlNet may facilitate wider\napplications to control image diffusion models.","upvotes":58,"discussionId":"641192393ea54b1aa7e2f918","githubRepo":"https://github.com/explainingai-code/ControlNet-PyTorch","githubRepoAddedBy":"auto","ai_summary":"ControlNet adds spatial conditioning to pretrained text-to-image diffusion models using zero-initialized convolutions, enabling diverse applications with small or large datasets.","ai_keywords":["neural network architecture","spatial conditioning controls","large diffusion models","zero convolutions","Stable Diffusion"],"githubStars":32},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"651bac4e0e6b7fa4292e9115","avatarUrl":"/avatars/6a3d30a735a90cd5d17c01aef71e941e.svg","isPro":false,"fullname":"JIANG","user":"Crow12138","type":"user"},{"_id":"5dd96eb166059660ed1ee413","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/NQtzmrDdbG0H8qkZvRyGk.jpeg","isPro":true,"fullname":"Julien Chaumond","user":"julien-c","type":"user"},{"_id":"63f8130749569335b679af62","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63f8130749569335b679af62/vgTu23-y0UKocwAGqNMwT.jpeg","isPro":false,"fullname":"Anyi Rao","user":"anyirao","type":"user"},{"_id":"634ceb6488c6b621d3589d3e","avatarUrl":"/avatars/22ade33ad4dc57ca9048d1927e684b20.svg","isPro":false,"fullname":"seongmi woo","user":"SSEONG","type":"user"},{"_id":"63f41abf4745321de3587dcd","avatarUrl":"/avatars/ce6b0052cd16cccc88c038a7ef297246.svg","isPro":false,"fullname":"hyeonwoojin","user":"maizer2","type":"user"},{"_id":"60aef0fbee40717d1a8fa6a5","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1624676266012-60aef0fbee40717d1a8fa6a5.png","isPro":false,"fullname":"Mayank Bhaskar","user":"cataluna84","type":"user"},{"_id":"63b5f506bda8d44adf369df5","avatarUrl":"/avatars/0e4655efb8a38c255d908ccfc87101f9.svg","isPro":false,"fullname":"Pawel Solyga","user":"solydzajs","type":"user"},{"_id":"659fa266697a41751bb3dca9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/uTFpjgxDaKvZvHCevoUh7.jpeg","isPro":false,"fullname":"Yang Xue","user":"AdriannaYang","type":"user"},{"_id":"64a720c5b9f90ed1ecc8acc9","avatarUrl":"/avatars/d740ecd8d23835eaa3891deb8c26878c.svg","isPro":false,"fullname":"John Dickens","user":"Chillowt","type":"user"},{"_id":"6435173f4ff6a19a90ca67da","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/TeFEjWX9bvcsmn0nJyafO.png","isPro":false,"fullname":"Artem Hvostov","user":"Sovenok-Hacker","type":"user"},{"_id":"65803d5ba2284a018e294fa0","avatarUrl":"/avatars/231bc98fd381700c8ca7875c3131e8ab.svg","isPro":false,"fullname":"Khattabka","user":"khattabka","type":"user"},{"_id":"63eeeb126c2c7c702bf95ef8","avatarUrl":"/avatars/ea08daea6c5c52f21bca03353f8b507b.svg","isPro":false,"fullname":"Lingbo Yang","user":"Lotayou","type":"user"}],"acceptLanguages":["*"]}">

Papers

arxiv:2302.05543

Adding Conditional Control to Text-to-Image Diffusion Models

Published on Feb 10, 2023

Upvote

Authors:

Lvmin Zhang ,

Anyi Rao ,

Maneesh Agrawala

Abstract

ControlNet adds spatial conditioning to pretrained text-to-image diffusion models using zero-initialized convolutions, enabling diverse applications with small or large datasets.

AI-generated summary

We present ControlNet, a neural network architecture to add spatial conditioning controls to large, pretrained text-to-image diffusion models. ControlNet locks the production-ready large diffusion models, and reuses their deep and robust encoding layers pretrained with billions of images as a strong backbone to learn a diverse set of conditional controls. The neural architecture is connected with "zero convolutions" (zero-initialized convolution layers) that progressively grow the parameters from zero and ensure that no harmful noise could affect the finetuning. We test various conditioning controls, eg, edges, depth, segmentation, human pose, etc, with Stable Diffusion, using single or multiple conditions, with or without prompts. We show that the training of ControlNets is robust with small (<50k) and large (>1m) datasets. Extensive results show that ControlNet may facilitate wider applications to control image diffusion models.