Vidya Muthukumar (Georgia Tech)

Apr 13, 2022

Title and Abstract

Classification versus regression in overparameterized regimes: Does the loss function matter?

While the initial mathematical explanations for “benign overfitting” were provided for regression, almost all success stories of modern machine learning have occurred in classification tasks. In this talk, I will compare classification and regression tasks in the overparameterized linear model (in both noiseless and noisy settings) and present the following results: On the side of optimization, the minimum-norm interpolating solution is identical to the hard-margin SVM solution either under sufficient effective overparameterization or the “neural collapse” conditions. For multiclass classification, we show an equivalence between two different SVM formulations (the multiclass SVM and the one-versus-all SVM) and interpolation. Coupled with characterizations of the implicit bias of gradient descent, our results imply that training with the cross-entropy loss and squared loss yield exactly identical solutions. On the side of generalization, we uncover high-dimensional regimes where the minimum-norm interpolating solution generalizes well for a classification task, but does not generalize in a corresponding regression task. We show that classification generalization is possible despite adverse signal recovery and in regimes where margin-based bounds are not predictive of generalization. These results show the contrasting roles of training and test loss functions in the overparameterized regime. Time permitting, I will conclude the talk with partial extensions of these results to kernel interpolation, and a brief discussion of consequences for adversarial robustness.

Bio

Vidya Muthukumar is an Assistant Professor in the Schools of Electrical and Computer Engineering and Industrial and Systems Engineering at Georgia Institute of Technology. Her broad interests are in game theory, online and statistical learning. She is particularly interested in designing online learning algorithms that provably adapt in strategic environments, fundamental properties of overparameterized models, and the intersection of game theory and reinforcement learning.

Vidya received the PhD degree in Electrical Engineering and Computer Sciences from University of California, Berkeley. She is the recipient of an Adobe Data Science Research Award, Simons-Berkeley Research Fellowship (for the Fall 2020 program on "Theory of Reinforcement Learning”), and a Georgia Tech Class of 1969 Teaching Fellowship for the academic year 2021-2022.