Universality Phenomenon in Random Feature and Kernel-based Learning

Abstract: Universality corresponds to the high-dimensional phenomenon that the macroscopic properties of a large-scale system do not depend too much on its microscopic structure. Such phenomenon has been explored and exploited in many different fields such as statistical physics, random matrix theory and signal processing.

In this talk, I will present recent works on understanding the exact asymptotics of random feature and kernel-based learning. One major challenge in the analysis stems from the non-linearity of underlying machine learning models. It turns out that in this context, there is a universality phenomenon called Gaussian equivalence: in terms of macroscopic performance such as training or generalization errors, the non-linear models can be equivalent to some microscopically different Gaussian models, which are much easier to analyze. This universality phenomenon enables us to obtain sharp characterization which reveals how the scalings of model and sample sizes, regularization, activation and target functions jointly affect the learning performance.