Just wanted to let you know that I've sent you an email regarding potential collaboration. It's from my address: gsq23@mails.tsinghua.edu.cn.
\n","updatedAt":"2025-06-23T07:12:34.992Z","author":{"_id":"630cf3388e3ff0c7232c8f36","avatarUrl":"/avatars/4d3fafcb1b77b344ec763f7dff6bb93f.svg","fullname":"Senquan Gao","name":"Maxwell-Gao314","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9379909038543701},"editors":["Maxwell-Gao314"],"editorAvatarUrls":["/avatars/4d3fafcb1b77b344ec763f7dff6bb93f.svg"],"reactions":[],"isReport":false,"parentCommentId":"685269c1ca5082ada25f0cc6"}}]}],"primaryEmailConfirmed":false,"paper":{"id":"2505.21189","authors":[{"_id":"6836babd75a4c5486bac4149","user":{"_id":"672e0638ee49faac3ad53af7","avatarUrl":"/avatars/3a273b4beba8286309296a1e25bc34a9.svg","isPro":false,"fullname":"Gleb Mezentsev","user":"glebzok","type":"user"},"name":"Gleb Mezentsev","status":"extracted_confirmed","statusLastChangedAt":"2025-05-28T07:43:05.521Z","hidden":false},{"_id":"6836babd75a4c5486bac414a","user":{"_id":"6169a581d05945bfd8718dfa","avatarUrl":"/avatars/1892ab06a7ddb557232777de3cbec470.svg","isPro":false,"fullname":"Ivan Oseledets","user":"oseledets","type":"user"},"name":"Ivan Oseledets","status":"admin_assigned","statusLastChangedAt":"2025-05-28T12:15:02.162Z","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/672e0638ee49faac3ad53af7/M2gWNMdYYUjkCuamqmHJ4.png","https://cdn-uploads.huggingface.co/production/uploads/672e0638ee49faac3ad53af7/ZAl4A9HCN24-VRZwER0H1.png","https://cdn-uploads.huggingface.co/production/uploads/672e0638ee49faac3ad53af7/zFXr1JmOc8jHoNtVJoiSM.png"],"publishedAt":"2025-05-27T13:39:24.000Z","submittedOnDailyAt":"2025-05-28T06:03:21.363Z","title":"Exploring the Latent Capacity of LLMs for One-Step Text Generation","submittedOnDailyBy":{"_id":"672e0638ee49faac3ad53af7","avatarUrl":"/avatars/3a273b4beba8286309296a1e25bc34a9.svg","isPro":false,"fullname":"Gleb Mezentsev","user":"glebzok","type":"user"},"summary":"A recent study showed that large language models (LLMs) can reconstruct\nsurprisingly long texts - up to thousands of tokens - via autoregressive\ngeneration from just one specially trained input embedding. In this work, we\nexplore whether such reconstruction is possible without autoregression. We show\nthat frozen LLMs can generate hundreds of accurate tokens in just one forward\npass, when provided with only two learned embeddings. This reveals a surprising\nand underexplored capability of LLMs - multi-token generation without iterative\ndecoding. We investigate the behaviour of these embeddings and provide insight\ninto the type of information they encode. We also empirically show that\nalthough these representations are not unique for a given text, they form\nconnected and local regions in embedding space - a property that suggests the\npotential of learning a dedicated encoder into that space.","upvotes":61,"discussionId":"6836babe75a4c5486bac4170","githubRepo":"https://github.com/Glebzok/OneStepLLMGeneration","githubRepoAddedBy":"user","ai_summary":"LLMs can generate long text segments in a single forward pass using learned embeddings, revealing a capability for multi-token generation without iterative decoding.","ai_keywords":["large language models","autoregressive generation","input embedding","frozen LLMs","multi-token generation","iterative decoding","learned embeddings","embedding space","dedicated encoder"],"githubStars":11},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"61ade264f602880813dbe10b","avatarUrl":"/avatars/a92dea7d853bbabbf60b351c207b6875.svg","isPro":false,"fullname":"Daniil Moskovskiy","user":"etomoscow","type":"user"},{"_id":"652ced57756a15d750266362","avatarUrl":"/avatars/5ef33e4935ccc6e930ceb2475c270bb1.svg","isPro":false,"fullname":"Alexey Skrynnik","user":"tviskaron","type":"user"},{"_id":"634c72e6fe1bfa967d6c2b5c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/634c72e6fe1bfa967d6c2b5c/WFWIAlWl-FsiJRyGxQTTx.jpeg","isPro":false,"fullname":"Nikita Sushko","user":"chameleon-lizard","type":"user"},{"_id":"6169a581d05945bfd8718dfa","avatarUrl":"/avatars/1892ab06a7ddb557232777de3cbec470.svg","isPro":false,"fullname":"Ivan Oseledets","user":"oseledets","type":"user"},{"_id":"667541b4d40714f7ccdc87b4","avatarUrl":"/avatars/c0c3bda8a8d1ec10d8f59316d9580247.svg","isPro":false,"fullname":"Olga Popova","user":"Pautike","type":"user"},{"_id":"6818a905900d924261251034","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/r8pO-yLXmJDo6gx1OLzxr.jpeg","isPro":false,"fullname":"Aleksander Karpov","user":"aleksanderkarpov","type":"user"},{"_id":"65c129a3444ff44cb6e3637d","avatarUrl":"/avatars/0de6dbd25faff1b452fb56cfe42fb579.svg","isPro":false,"fullname":"Vitaliy Pozdnyakov","user":"vpozdnyakov","type":"user"},{"_id":"667003f4df8cad479334e49a","avatarUrl":"/avatars/4109914df706e0acdeca89604c624d7b.svg","isPro":false,"fullname":"Artem","user":"wernon","type":"user"},{"_id":"6655ac89fdf911031cd0e16a","avatarUrl":"/avatars/816fcc3ae3808e5b4b00f7e89b594c2f.svg","isPro":false,"fullname":"Daniil Shlenskii","user":"d-shlenskii","type":"user"},{"_id":"66db434b8636a92bcdf1c298","avatarUrl":"/avatars/21c3d6ec44f39548c431de3d6f92e98c.svg","isPro":false,"fullname":"Yulia Trekhletova","user":"HR-Julie","type":"user"},{"_id":"634c084c4abe84057588fd63","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/634c084c4abe84057588fd63/-3doVHYwCqbkPQtwVX9aM.jpeg","isPro":false,"fullname":"Maxim Kurkin","user":"dondosss","type":"user"},{"_id":"64f734b08e7fa6529fb693f1","avatarUrl":"/avatars/a50c7e1d22174f06fbaf99e8caca7c37.svg","isPro":false,"fullname":"Petr Anokhin","user":"petranokhin","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0}">Exploring the Latent Capacity of LLMs for One-Step Text Generation
Abstract
LLMs can generate long text segments in a single forward pass using learned embeddings, revealing a capability for multi-token generation without iterative decoding.
A recent study showed that large language models (LLMs) can reconstruct surprisingly long texts - up to thousands of tokens - via autoregressive generation from just one specially trained input embedding. In this work, we explore whether such reconstruction is possible without autoregression. We show that frozen LLMs can generate hundreds of accurate tokens in just one forward pass, when provided with only two learned embeddings. This reveals a surprising and underexplored capability of LLMs - multi-token generation without iterative decoding. We investigate the behaviour of these embeddings and provide insight into the type of information they encode. We also empirically show that although these representations are not unique for a given text, they form connected and local regions in embedding space - a property that suggests the potential of learning a dedicated encoder into that space.
Community
A recent study showed that large language models (LLMs) can reconstruct surprisingly long texts - up to thousands of tokens - via autoregressive generation from just one specially trained input embedding. In this work, we explore whether such reconstruction is possible without autoregression. We show that frozen LLMs can generate hundreds of accurate tokens in just one forward pass, when provided with only two learned embeddings. This reveals a surprising and underexplored capability of LLMs - multi-token generation without iterative decoding. We investigate the behaviour of these embeddings and provide insight into the type of information they encode. We also empirically show that although these representations are not unique for a given text, they form connected and local regions in embedding space - a property that suggests the potential of learning a dedicated encoder into that space.
Good try! I'm wondering if you have attempted to use all different [m] sequences( like [m1,m2,...m_N-1]). I conducted similar experiments before and found that if all the different m sequences were used, it seemed that only the initial 1-2 tokens could be reconstructed.
Do you mean a separate m-vector for each target token?
One-step text generation using proto-tokens is truly groundbreaking!
I'm particularly interested in the feasibility of training an end-to-end proto-token encoder. I'm curious if you have any preliminary insights into the computational resources (e.g., GPU hours, specific hardware) that might be required for such an endeavor.
(Perhaps there's an opportunity for us to collaborate on exploring this exciting next step? I've just got access to some A100 GPUs)
Yes, sure, it would be really great to collaborate on this topic, please write to mezentsev@airi.net, so we can stay in touch
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper