Ramya Vinayak (UW-Madison)

1:00pm - 2:00pm PT Friday, November 8, 2024

Title and Abstract

Towards Pluralistic Alignment: Foundations for Learning Diverse Human Preferences

Large pre-trained models trained on internet-scale data are often not ready for deployment out- of-the-box. They are heavily fine-tuned or aligned using large quantities of human preference data, usually elicited using pairwise comparisons. While aligning an AI/ML model to human preferences or values, it is important to ask whose preference and values we are aligning it to? The current approaches of preference alignment are severely limited due to inherent assumption of uniformity by the preference models. We aim to overcome this limitation by building mathematical foundations for learning diverse human preferences. In this talk, I will present, PAL, a personalize- able reward modelling framework for pluralistic alignment. PAL has modular design that leverages commonalities across users while catering to individual personalization, enabling efficient few-shot generalization. PAL is versatile to be applied to various domains and matches or outperforms state-of-the-art methods on both text-to-text and text-to-image tasks with 100x fewer parameters in practice. I will also present theoretical results on per user sample complexity for generalization and fundamental limitations when there are limited pairwise comparisons. Based on work with Daiwei Chen, Yi Chen, Aniket Rege, Zhi Wang, Geelon So, Greg Canal, Blake Mason, Gokcan Tatli, and Rob Nowak.

References:

1. PAL: Pluralistic Alignment Framework for learning from heterogeneous preferences (preprint, 2024)

2. One-for-all: Simultaneous metric and preference learning (NeurIPS 2022)

3. Metric learning via limited pairwise comparisons (UAI 2024), and

4. Learning Populations of Preferences via pairwise comparisons (AISTATS 2024).

Bio

Ramya Korlakai Vinayak is the Dugald C. Jackson assistant professor in the Dept. of ECE and affiliated faculty in the Dept. of Computer Science and the Dept. of Statistics at the UW-Madison. Her research interests span the areas of machine learning, statistical inference, and crowdsourcing, with a focus on preference learning and alignment under heterogeneity, reliable and efficient dataset creation, and human-in-the-loop systems. Her works aim to address theoretical and practical challenges that arise when learning from heterogeneous societal data. Prior to joining UW-Madison, Ramya was a postdoctoral researcher in the Paul G. Allen School of Computer Science and Engineering at the University of Washington. She received her Ph.D. in Electrical Engineering from Caltech. She obtained her Masters from Caltech and Bachelors from IIT Madras. She is a recipient of the Schlumberger Foundation Faculty of the Future fellowship from 2013-15, and an invited participant at the Rising Stars in EECS workshop in 2019. She is the recipient of NSF CAREER Award in 2023