I am an Applied Scientist at Amazon Rufus. I study how large language models learn and how they can learn better.
At Amazon, I have worked on pretraining 100B–800B MoE models, studying expert learning dynamics and training stability. More recently, I have been extending this work to post-training and reinforcement learning, with a focus on SFT/RL infrastructure, data curation, and how training dynamics change beyond pretraining.
Experience
- B.E. in Computer Science · Peking University2015–2020
- Ph.D. in Computer Science · Texas A&M University2020–2025
- Applied Scientist · Amazon Rufus2025–present
News
Our follow-up blog on LLAL is out: How did LLAL change the model? — Similar to Engram, but not the same.
Our blog on preventing expert collapse in ultra-sparse MoE (8/768) training is out: Mitigate Silent Expert Death in Ultra-Sparse MoE. LLAL is proposed to help train ultra-sparse MoE models with negligible overhead. A small step towards more sparse models!
Check out our first blog about MoE training: The death of z-loss in modern LLMs: a story of expert collapse and specialization.
LLM Maybe LongLM (SelfExtend) has been selected as Spotlight (3.5%) at ICML 2024!
Show more newsShow less news
Started an internship at Amazon Rufus. Happy to chat if you’re in Seattle!
KIVI (2-bit, tuning-free KV cache quantization) has been accepted by ICML 2024! Congrats to Zirui Liu and Jiayi Yuan!
Our paper on extending LLM context windows is now available on arXiv: LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning. SelfExtend elicits LLMs’ inherent long-context capabilities without fine-tuning.
One paper accepted by NeurIPS 2023.
One paper accepted by TMLR: Retiring ∆DP!
New preprint: Survey on LLMs!
Selected Works
Pretraining
-
Ultra-sparse MoE training
Mitigate Silent Expert Death in Ultra-Sparse MoE
Understanding expert collapse in lower MoE layers and mitigating it with early auxiliary LM supervision (LLAL).
-
Long-context modeling · SelfExtend
LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning
Extending LLM context windows at inference time without fine-tuning.
Post-training
-
Stepwise Penalization for Length-Efficient Chain-of-Thought Reasoning
Step-level length penalties that reduce redundant reasoning while preserving useful steps.
-
Process Reward Informed Tree Rollout for Effective Multi-Turn RL
Using process feedback to guide tree rollouts and explore promising intermediate states in multi-turn agent RL.
-
CoMem: Context Management with A Decoupled Long-Context Model
Decoupling memory management from agent reasoning so context summarization and inference can run in parallel.
Feel free to reach out for collaborations, discussions, or opportunities.
Last updated on September 29, 2026.