Deprecated: The each() function is deprecated. This message will be suppressed on further calls in /home/zhenxiangba/zhenxiangba.com/public_html/phproxy-improved-master/index.php on line 456 Paper page - On Teacher Hacking in Language Model Distillation
\n","updatedAt":"2025-02-06T11:15:51.168Z","author":{"_id":"6262880c5eb4fa93219f0064","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6262880c5eb4fa93219f0064/6yyBvRK4Oh7OhjaaweaVN.jpeg","fullname":"Daniil Tiapkin","name":"dtiapkin","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":8,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8842591047286987},"editors":["dtiapkin"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/6262880c5eb4fa93219f0064/6yyBvRK4Oh7OhjaaweaVN.jpeg"],"reactions":[{"reaction":"đź‘Ť","users":["sugatoray"],"count":1}],"isReport":false}},{"id":"67a563084b741937347365c4","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2025-02-07T01:34:00.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Self-Evolution Knowledge Distillation for LLM-based Machine Translation](https://huggingface.co/papers/2412.15303) (2024)\n* [Enhancing Knowledge Distillation for LLMs with Response-Priming Prompting](https://huggingface.co/papers/2412.17846) (2024)\n* [Knowledge Injection via Prompt Distillation](https://huggingface.co/papers/2412.14964) (2024)\n* [Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision](https://huggingface.co/papers/2501.07886) (2025)\n* [Chunk-Distilled Language Modeling](https://huggingface.co/papers/2501.00343) (2024)\n* [MoPD: Mixture-of-Prompts Distillation for Vision-Language Models](https://huggingface.co/papers/2412.19087) (2024)\n* [Online Preference Alignment for Language Models via Count-based Exploration](https://huggingface.co/papers/2501.12735) (2025)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
\n
The following papers were recommended by the Semantic Scholar API
Experiments reveal that knowledge distillation can lead to teacher hacking, an issue akin to reward hacking, but it can be mitigated by using online data generation techniques that enhance data diversity.
AI-generated summary
Post-training of language models (LMs) increasingly relies on the following
two stages: (i) knowledge distillation, where the LM is trained to imitate a
larger teacher LM, and (ii) reinforcement learning from human feedback (RLHF),
where the LM is aligned by optimizing a reward model. In the second RLHF stage,
a well-known challenge is reward hacking, where the LM over-optimizes the
reward model. Such phenomenon is in line with Goodhart's law and can lead to
degraded performance on the true objective. In this paper, we investigate
whether a similar phenomenon, that we call teacher hacking, can occur during
knowledge distillation. This could arise because the teacher LM is itself an
imperfect approximation of the true distribution. To study this, we propose a
controlled experimental setup involving: (i) an oracle LM representing the
ground-truth distribution, (ii) a teacher LM distilled from the oracle, and
(iii) a student LM distilled from the teacher. Our experiments reveal the
following insights. When using a fixed offline dataset for distillation,
teacher hacking occurs; moreover, we can detect it by observing when the
optimization process deviates from polynomial convergence laws. In contrast,
employing online data generation techniques effectively mitigates teacher
hacking. More precisely, we identify data diversity as the key factor in
preventing hacking. Overall, our findings provide a deeper understanding of the
benefits and limitations of distillation for building robust and efficient LMs.
Post-training of language models (LMs) increasingly relies on the following two stages: (i) knowledge distillation, where the LM is trained to imitate a larger teacher LM, and (ii) reinforcement learning from human feedback (RLHF), where the LM is aligned by optimizing a reward model. In the second RLHF stage, a well-known challenge is reward hacking, where the LM over-optimizes the reward model. Such phenomenon is in line with Goodhart's law and can lead to degraded performance on the true objective. In this paper, we investigate whether a similar phenomenon, that we call teacher hacking, can occur during knowledge distillation. This could arise because the teacher LM is itself an imperfect approximation of the true distribution. To study this, we propose a controlled experimental setup involving: (i) an oracle LM representing the ground-truth distribution, (ii) a teacher LM distilled from the oracle, and (iii) a student LM distilled from the teacher. Our experiments reveal the following insights. When using a fixed offline dataset for distillation, teacher hacking occurs; moreover, we can detect it by observing when the optimization process deviates from polynomial convergence laws. In contrast, employing online data generation techniques effectively mitigates teacher hacking. More precisely, we identify data diversity as the key factor in preventing hacking. Overall, our findings provide a deeper understanding of the benefits and limitations of distillation for building robust and efficient LMs.