I received my Ph.D. in computer science from the University of Texas at Austin (UT Austin), where I was fortunate to be advised by Prof. Qiang Liu. I earned my bachelor's degree in mathematics in 2022.
I am interested in developing fundamental yet computationally feasible algorithms for the basic learning, inference,
and optimization problems that underpin the cutting-edge AI/ML/statistical technologies.
These days, I am mostly drawn by the training efficiency of large-scale models.
If you share common interests, are interested in potential collaboration, or simply want to connect for a chat, feel free to contact me. I'm always open to conversation :)
Core contributor to DeMo, the optimizer foundation behind Nous Research's distributed-training stack, used to train the Hermes model series.
Set a world record on Slowrun, the leaderboard for language modeling in the fixed-data, unlimited-compute regime; my submission has held the best validation loss on the benchmark for almost a month.
Andrej Karpathy on using Cautious Weight Decay in nanochat:
"Worked great out of the box on nanochat too, beat standard weight decay in a solid sweep."
AK
Andrej Karpathy@karpathy
Worked great out of the box on nanochat too, beat standard weight decay in a solid sweep.
Hugging Face timm adopted Cautious Optimizers. Ross Wightman:
"One of the last minute papers I added support for that delayed this release was 'Cautious Optimizers'... Consider me impressed, this boost appears more consistent than some of the new optimizers."
RW
Ross Wightman@wightmanr
One of the last minute papers I added support for that delayed this release was 'Cautious Optimizers'... Consider me impressed, this boost appears more consistent than some of the new optimizers.
Contributing to google-deepmind/simply, DeepMind's minimal and scalable JAX research codebase for frontier LLM research.
Invited Talks & Lectures
Meta AI
“Communication Efficient Distributed Training with Distributed Lion”
Host: Raghu Krishnamoorthi
