About Me

Welcome to my website! :) I am a postdoc research fellow at the Kempner Institute at Harvard University, working closely with the ML Foundations group. My research focuses on scientific understanding of machine learning, especially towards understanding and realizing methods and opportunities at small scales. My work uses a combination of theory and controlled experiments, often drawing insights from synthetic "sandbox" settings (write-up) that isolate particular aspects of complex phenomena. Most recently, I'm particularly interested in understanding how various inductive biases (from data usage, training algorithms, and architecture) affect training efficiency and inference cost.

Before Kempner, I was a postdoc fellow at the Simons Institute, participating in Special Year of Language Models and Modern Paradigm of Generalization. I got my PhD from the Machine Learning Department of Carnegie Mellon University, where I was very fortunate to be co-advised by Prof. Andrej Risteski and Prof. Pradeep Ravikumar.

I am currently on the academic job market for positions starting in Fall 2027.

Research

/

Model performance improves with scale, but the scaling frontier keeps shifting as our understanding and methods advance. Small models can now achieve performance that once requires much larger ones.
A natural question is: How much further can we push the small-scale frontier?
Simply scaling down frontier-scale recipes likely won't suffice, and we need deeper understanding on the fundamental limits of representation and learning efficiency to guide algorithm design.

My research pursues this small-scale agenda in three directions.

1. Understanding representational limits.

The first step is to ensure that the capabilities we seek are theoretically possible. We show that shallow, parallel models can be powerful at sequential reasoning (project page), and that some problems admit infinitely many feasible solutions (which also highlights challenges for interpretability). There are encouraging empirical signals too: models trained on proper synthetic data can match the performance of much larger models. Together, these findings suggest that models far smaller than today's may be possible and worth pursuing.

2. Improving learning efficiency.

Even when small models have sufficient representational capacity, learning remains a practical bottleneck.

One promising approach is knowledge distillation, which allows small "student" model to benefit from the compute already invested in training large "teacher" models. However, effective distillation requires proper teacher choice. We show that learning from the right teachers can improve sample efficiency, and propose a lightweight GRACE score for teacher selection in LLM finetuning.

There are many other ways to improve learning efficiency beyond distillation. On the data side, we find that sometimes training with less data can improve compute efficiency. When considering the optimization landscape, even small changes to the training objective can drastically affect the convergence rate, and optimizer design can make a substantial difference.

3. Closing the long-tail reliability gap.

As models become stronger, errors at the tail can be more revealing than benchmark performance. We find that Transformers can exhibit nontrivial tail errors on a simple task representable by a single-layer model, despite achieving over 99% accuracy. We trace these errors to architectural factors and identify ways to mitigate them (coming soon!). Understanding and systematically eliminating such failures will be crucial for building reliable models.

opd_misspec

When Does Online Imitation Learning Help in LLM Post-Training? The Role of (Non-)Realizability Beyond Horizon

Huaqing Zhang, Jingchu Gai, Juno Kim, Bingbin Liu, Andrej Risteski,

NeurIPS26.

[arXiv]
two-clock

The two clocks and the innovation window: When and how generative models learn rules

Binxu Wang, Emma Lucia Byrnes Finn, Bingbin Liu

NeurIPS26. Contributed talk at SPIGM.

[arXiv]
small_vs_large_gap

Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases

Jingwen Liu, Ezra Edelman, Surbhi Goel, Bingbin Liu

ICML26. Contributed talk at Sci4DL.

[arXiv]
GRACE

In Good GRACES: Principled Teacher Selection for Knowledge Distillation

Abhishek Panigrahi, Bingbin Liu, Sadhika Malladi, Sham M. Kakade, Surbhi Goel

ICLR26

[arXiv]
Adam

Adam or Gauss-Newton? A Comparative Study In Terms of Basis Alignment and SGD Noise

Bingbin Liu, Rachit Bansal, Depen Morwani, Nikhil Vyas, David Alvarez-Melis, Sham M. Kakade

Preprint

[arXiv]
Progressive

Progressive distillation induces an implicit curriculum

Abhishek Panigrahi*, Bingbin Liu*, Sadhika Malladi, Andrej Risteski, Surbhi Goel

ICLR25 (Oral)

[arXiv] [OpenReview]
TinyGSM

Tinygsm: achieving> 80% on gsm8k with small language models

Bingbin Liu, Sebastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, Yi Zhang,

[arXiv]
Dyck

(Un)interpretability of Transformers: a case study with bounded Dyck grammars

Kaiyue Wen, Yuchen Li, Bingbin Liu, Andrej Risteski

NeurIPS 2023

[arXiv]
Flipflop

Exposing Attention Glitches with Flip-Flop Language Modeling

Bingbin Liu, Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, Cyril Zhang

NeurIPS 2023 (Spotlight)

[arXiv]
Automata

Transformers Learn Shortcuts to Automata

Bingbin Liu, Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, Cyril Zhang

ICLR 2023 (Oral)

[arXiv] [OpenReview] [Project Page] [Code]
HMM

Masked prediction tasks: a parameter identifiability view

Bingbin Liu, Daniel Hsu, Pradeep Ravikumar, Andrej Risteski

NeurIPS 2022

[arXiv]
eNCE

Analyzing and improving the optimization landscape of noise-contrastive estimation

Bingbin Liu, Elan Rosenfeld, Pradeep Ravikumar, Andrej Risteski

ICLR 2022 (Spotlight)

[arXiv]
SDE

Contrastive learning of strong-mixing continuous-time stochastic processes

Bingbin Liu, Pradeep Ravikumar, Andrej Risteski

AISTATS 2021

[arXiv]
Boosting

Generalized Boosting

Arun Sai Suggala, Bingbin Liu, Pradeep Ravikumar

NeurIPS 2020

[Paper]
STIP

Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction

Bingbin Liu, Ehsan Adeli, Zhangjie Cao, Kuan-Hui Lee, Abhijeet Shenoi, Adrien Gaidon, Juan Carlos Niebles

IEEE-RAL & ICRA 2020

[arXiv]
Medicine

A Computer Vision System to Detect Bedside Patient Mobilization

Serena Yeung*, Francesca Rinaldo*, Jeffrey Jopling, Bingbin Liu, Rishab Mehra, Lance Downing, Michelle Guo, Gabriel Bianconi, Alexandre Alahi, Julia Lee, Brandi Campbell, Kayla Deru, William Beninati, Li Fei-Fei, Arnold Milstein.

Nature Digital Medicine, 2019

[Paper]
TMN

Temporal Modular Networks for Retrieving Complex Compositional Activities in Videos

Bingbin Liu, Serena Yeung, Edward Chou, De-An Huang, Li Fei-Fei, Juan Carlos Niebles

ECCV 2018

[Paper] [Project Page] [Video]
DDEAP

Learning to Decompose and Disentangle Representations for Video Prediction

Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li Fei-Fei, Juan Carlos Niebles

NeurIPS 2018

[Paper] [Github] [Video]
MLHC

3D Point Cloud-Based Visual Prediction of ICU Mobility Care Activities

Bingbin Liu*, Michelle Guo*, Edward Chou, Rishab Mehra, Serena Yeung, N. Lance Downing, Francesca Salipur, Jeffrey Jopling, Brandi Campbell, Kayla Deru, William Beninati, Arnold Milstein

MLHC 2018

[Paper]

Activities

Talks

Lifting the small-scale frontier

Learning from "sandboxes" Learning with data repetition Knowledge distillation Transformer reasoning

Organized Events

Misc

I want to keep a list of good advice I've received from various friends and mentors. Please let me know if you have recommendations (especially since I'm doing a horrible job at gender balance here)!

Random. Some photos of places I've lived and furry friends I met along the way. A Connections game for our wedding.
(More to be continued :))