Berivan Isik (Google)

Apr 18, 2025
Cory 400

Title and Abstract

Beyond Pretraining Loss: Evaluating Value of Pretraining Data for Large Language Models at Scale

The performance of Large Language Models (LLMs) is fundamentally dependent on the quality and quantity of their pretraining data. Understanding the impact of data choices and scale is crucial for optimizing model development and deployment. This talk will begin by highlighting the importance of accurately estimating the effect of data characteristics on LLM performance. We will then delve into scaling laws, which provide valuable insights into predicting pretraining loss as a function of pretraining data size and model parameters. However, pretraining loss alone does not fully capture the complex interplay between pretraining data and downstream task performance. To address this, we will explore our recent work extending beyond pretraining loss to directly predict downstream metric performance at scale. This approach provides a more comprehensive evaluation of value of pretraining data, enabling a nuanced understanding of how data influences real-world applications.

This talk will draw upon findings in our recent ICLR 2025 paper: Scaling Laws for Downstream Task Performance in Machine Translation.

Bio

Berivan Isik is a research scientist at Google, working on efficient and trustworthy AI. Her current interests are efficient training of large models, LLM data and evaluation metrics, and unlearning. She completed her PhD at Stanford University in 2024. Her research was supported by Stanford Graduate Fellowship, Google Ph.D. Fellowship, and a Meta research grant