Your intuition is correct. I think about it as trying to perturb all the directions in the model that are going to be important. I've been pleasantly surprised by how few samples are needed to get a representative set. Just 2k samples from Alpaca seems to be enough to preserve llm-eval metrics.
\nWe have not experimented with steering! Feel free to check out the code and pull out the Principal components to see what they each do. I'd be interested to hear what happens.
\n","updatedAt":"2024-01-29T17:05:41.128Z","author":{"_id":"65af77603db2280ece8a6644","avatarUrl":"/avatars/37fac7c73e0190f933201398c74bcc50.svg","fullname":"James Hensman","name":"Jameshensman","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9759540557861328},"editors":["Jameshensman"],"editorAvatarUrls":["/avatars/37fac7c73e0190f933201398c74bcc50.svg"],"reactions":[{"reaction":"β€οΈ","users":["imryanxu"],"count":1},{"reaction":"π","users":["leegao19"],"count":1}],"isReport":false}},{"id":"65b84f42e0bde92c175b5dfb","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2024-01-30T01:22:10.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [SPT: Fine-Tuning Transformer-based Language Models Efficiently with Sparsification](https://huggingface.co/papers/2312.10365) (2023)\n* [Rethinking Compression: Reduced Order Modelling of Latent Features in Large Language Models](https://huggingface.co/papers/2312.07046) (2023)\n* [The LLM Surgeon](https://huggingface.co/papers/2312.17244) (2023)\n* [PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation](https://huggingface.co/papers/2401.11316) (2024)\n* [DSFormer: Effective Compression of Text-Transformers by Dense-Sparse Weight Factorization](https://huggingface.co/papers/2312.13211) (2023)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
\nThe following papers were recommended by the Semantic Scholar API
\n- \n
- SPT: Fine-Tuning Transformer-based Language Models Efficiently with Sparsification (2023) \n
- Rethinking Compression: Reduced Order Modelling of Latent Features in Large Language Models (2023) \n
- The LLM Surgeon (2023) \n
- PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation (2024) \n
- DSFormer: Effective Compression of Text-Transformers by Dense-Sparse Weight Factorization (2023) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
SliceGPT: Compress Large Language Models by Deleting Rows and Columns
Abstract
SliceGPT, a novel post-training sparsification method, reduces model parameters by up to 25% without significant performance decrease, optimizing inference speed and GPU usage on large language models.
Large language models have become the cornerstone of natural language processing, but their use comes with substantial costs in terms of compute and memory resources. Sparsification provides a solution to alleviate these resource constraints, and recent works have shown that trained models can be sparsified post-hoc. Existing sparsification techniques face challenges as they need additional data structures and offer constrained speedup with current hardware. In this paper we present SliceGPT, a new post-training sparsification scheme which replaces each weight matrix with a smaller (dense) matrix, reducing the embedding dimension of the network. Through extensive experimentation, we show that SliceGPT can remove up to 25% of the model parameters (including embeddings) for LLAMA2-70B, OPT 66B and Phi-2 models while maintaining 99%, 99% and 90% zero-shot task performance of the dense model respectively. Our sliced models run on fewer GPUs and run faster without any additional code optimization: on 24GB consumer GPUs we reduce the total compute for inference on LLAMA2-70B to 64% of that of the dense model; on 40GB A100 GPUs we reduce it to 66%. We offer a new insight, computational invariance in transformer networks, which enables SliceGPT and we hope it will inspire and enable future avenues to reduce memory and computation demands for pre-trained models. Code is available at: https://github.com/microsoft/TransformerCompression
Community
Not APril FOols, Rad!
Code is 404
Code is 404
it's there now!
On the calibration set - it looks like this is running a bunch of samples of Wiki2 and Alpaca truncated to the maximum length of the LLM.
I'm guessing the intuition here is to have a representative set of samples that's either seen during pretraining or are likely to be encountered in day-to-day use.
Have you guys explored the idea of using this for feature activation steering? E.g. prime the model (maybe even using contrastive pairs to get positive and negatives) using calibration prompts that steer behavior to, for e.g., avoid sycophancy (vs emulates typical interactions).
Admittedly, it's really heavy for what can be done, but it'd be interesting to see if the PCA picks up anything interesting (e.g. do they decompose into basic feature directions? can they be interpreted? can they be manipulated to amplify/suppress certain behaviors? do these components become sparser as steering becomes more directed?)
Hi @leegao19 ,
Your intuition is correct. I think about it as trying to perturb all the directions in the model that are going to be important. I've been pleasantly surprised by how few samples are needed to get a representative set. Just 2k samples from Alpaca seems to be enough to preserve llm-eval metrics.
We have not experimented with steering! Feel free to check out the code and pull out the Principal components to see what they each do. I'd be interested to hear what happens.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- SPT: Fine-Tuning Transformer-based Language Models Efficiently with Sparsification (2023)
- Rethinking Compression: Reduced Order Modelling of Latent Features in Large Language Models (2023)
- The LLM Surgeon (2023)
- PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation (2024)
- DSFormer: Effective Compression of Text-Transformers by Dense-Sparse Weight Factorization (2023)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper