In this study, we introduce an optimization-centric approach to enhance the efficiency of large-scale model training. Our primary contribution lies in the development of novel optimization algorithms that exhibit superior generalization performance, reduce hyperparameter tuning requirements, and lower computational and memory costs. Our approach not only achieves comparable test accuracy in less training time but also demonstrates improved model performance under similar training costs. Memory efficiency is achieved through innovative schemes that factorize first and second-order momentum. By leveraging the parallel computing capabilities of GPUs and crafting custom optimizers for specific model architectures, we optimize training processes. Moreover, we prioritize optimization objectives that ensure robust generalization concerning diverse data distributions and model architectures. This research paves the way for more efficient and practical large-scale model applications across various domains.

Lion (Evolved Sign Momentum), a new optimizer discovered through program search, has shown promising results in training large AI models. It performs comparably or favorably to AdamW but with greater memory efficiency. This work aims to demystify Lion. Based on both continuous-time and discrete-time analysis, we demonstrate that Lion is a theoretically novel and principled approach for minimizing a general loss function f(x) while enforcing a bound constraint ‖x‖∞ ≤ 1/λ. Our findings provide valuable insights into the dynamics of Lion and pave the way for further improvements and extensions of Lion-related algorithms.