GaLore：在消費級硬體上訓練大型語言模型的突破性技術

As the parameter counts of large language models (LLMs) have skyrocketed, the hardware requirements for training and fine-tuning these models have risen accordingly. Traditionally, when performing full-parameter training with optimizers like Adam, the memory consumed by optimizer states is typically two to three times that of the model weights themselves. This has made training large models on consumer-grade hardware nearly impossible. While parameter-efficient fine-tuning (PEFT) techniques such as LoRA (Low-Rank Adaptation) have alleviated memory pressure during fine-tuning, they cannot be directly applied to pre-training from scratch and limit the expressive capacity of learnable parameters.