The Hidden Power of Scaling Factor in LoRA Optimization

Maybe the first systematic study (empirical + theoretical) of LoRA's scaling factor $\alpha$ from an optimization perspective!

Recent studies highlight that a large learning rate ($\eta$) is crucial for LoRA optimization. However, this paper points out that such a conclusion was drawn while leaving the scaling factor $\alpha$ systematically underexplored. Through a joint empirical and theoretical lens, the authors reveal a shifting paradigm: a significantly large scaling factor $\alpha$ is what actually matters most, delivering optimization gains that learning rate scaling alone cannot replicate.

Key takeaways:

1. LoRA's low-rank nature smooths the optimization landscape (spectral suppression), making standard hyperparameters overly conservative and causing an optimization gap.

2. $\alpha$ vs $\eta$: Increasing $\alpha$ acts as a "purity-preserving accelerator", which amplifies the task signal without increasing the bilinear drift, outperforming learning rate scaling.

3. Under standard, small learning rates, an optimal $\alpha$ must be sufficiently large and follow a sublinear relationship with rank ($r$). This reveals that popular rank-tied heuristics (like $\alpha = r$ or $2r$) leave LoRA severely under-scaled due to their insufficient magnitudes.

Based on these insights, the authors propose LoRA-$\alpha$, which scales $\alpha$ based on a principled square-root law (e.g., using a large base coefficient like $256\sqrt{r}$). This minimalist shift allows LoRA to directly inherit standard, small Full Fine-Tuning (FFT) learning rates while matching or even exceeding FFT performance across NLP, multimodal, and RL tasks.

Bye-bye expensive hyperparameter tuning! 👋