Yuchen Li

I am a Ph.D. student in the Machine Learning Department at Carnegie Mellon University, where I am fortunate to be advised by Professor Andrej Risteski.

Previously, I spent four wonderful years at the University of Illinois at Urbana-Champaign, completing my bachelor's degree in Statistics & Computer Science, as well as Mathematics. I am grateful for the mentorship of Professor Jiawei Han, Professor AJ Hildebrand, and Professor Pramod Viswanath.

In the industry, I spent some exciting time at Microsoft (summer 2024), Google (summer 2023), ByteDance (TikTok) (summer 2022), Quora (2019-2020), and Facebook (Meta) (summer 2018).

In fall 2024, I am a visiting graduate student for the Modern Paradigms in Generalization program at Simons Institute for the Theory of Computing in Berkeley, California, USA.

Email  /  CV (resume)  /  LinkedIn  /  Google Scholar  /  Twitter (X)  /  GitHub  /  Quora

I study machine learning, natural language processing, and data mining. I currently work on (1) improving mathematical understanding of language models (training dynamics, efficient sampling, mechanistic interpretability), and (2) developing principled approaches to self-supervised learning.

In the following list, the asterisk symbol (*) in the author list means equal contribution or alphabetical order.

We establish a mathematical framework for understanding the potentials of applying masked language models (MLMs) to generative tasks. We theoretically analyze the training sample efficiency and the inference efficiency, and empirically characterize the critical components and failure modes.

The highlight of our theory is a "dictionary" between the sample efficiency of MLM losses and the mixing times of Markov Chains. As a side product, we show that masking more is always statistically better (but algorithmically harder). Our theory also implies that "multimodal" distributions cause problems for both learning and inference. Finally, we theoretically identify a limitation of Transformers for parallel decoding: they enforce conditional product distributions, which can prevent fast mixing at inference time. We empirically validate our theory on synthetic and real data.

We show that interpretability claims based on isolated attention patterns or weight components can be (provably) misleading.

Our results are based on next-token prediction on Dyck (i.e. balanced brackets). We show that Transformers (exactly or approximately) solving this task satisfy a structural characterization, “balance condition“, derived from ideas in formal languages (the pumping lemma).

We analyze the optimization process of transformers trained on data involving "semantic structure", e.g. topic modeling. Through theoretical analysis and experiments, we show that between same-topic words, the embeddings should be more similar, and the average pairwise attention should be larger.

In particular, we observe that with carefully chosen initialization and learning rate, the optimization process of self-attention can be approximately broken down into two stages: in stage 1, only the value matrix changes significantly; in stage 2, the key and query matrices catch up much later, even though all components are jointly optimized through standard SGD or Adam. This observation might be of independent interest, for future works on understanding the learning dynamics of transformers as well.

We show through theoretical results and controlled experiments that even on simple data models, non-contrastive losses have a preponderance of non-collapsed bad minima (which do not exist for their contrastive loss counterpart). Moreover, we show that the training process does not avoid these minima.

We prove that, if the context for the parsing decision at each word is unbounded & bidirectional, then the parsing model has full representational power.

On the other hand, even if the context is bounded in one direction (which is the case for various leading approaches to constituency parsing), the parsing power is quite limited for hard grammars.

Consider the sequence of leading digits of 2^n: 1, 2, 4, 8, 1, 3, 6, ...... What is the complexity of such sequences? They are neither periodic nor random. How to quantify and distinguish the middle ground?

We prove that, the block complexity is linear for the sequence of leading digits of a^n in base b (except for some special cases). In fact, we give explicit formula for the linear coefficients in terms of a and b.

This work discovers hypernym-hyponym ("is-a" relation) pairs by an innovative combination of information from natural language and graph structure. Our system outperforms existing methods that utilize signals from text, graph, or both.

Given a misspelled word, our algorithm generates candidates based on the surface form, represents the context as a linear space, and selects the candidate whose embedding is closest to the context subspace. The spelling correction accuracy is better than the state-of-the-art methods, improving the performance in downstream applications.

This work shows the effectiveness of temporal motifs (frequent substructures) in quantifying the relationship between nodes in heterogeneous information networks, and proposes efficient algorithms for counting a family of useful motifs.

Template of this page: credit to: link.