Language model scaling laws and zero-sum learning
Investigating the relationship between language model size, training dynamics and the phenomenon of zero-sum learning.
Topics:
This work aims to understand how, in terms of training dynamics, scaling up language model size yields predictable loss improvements. We find that these improvements can be tied back to loss deceleration, an abrupt transition in the rate of loss improvement, characterized by piece-wise linear behavior in log-log space. Notably, improvements from increased model size appear to be a result of (1) improving the loss at which this transition occurs; and (2) improving the rate of loss improvement after this transition. As an explanation for the mechanism underlying this transition (and the effect of model size on loss it mediates), we propose the zero-sum learning (ZSL) hypothesis. In ZSL, per-token gradients become systematically opposed, leading to degenerate training dynamics where the model can't improve loss on one token without harming it on another; bottlenecking the overall rate at which loss can improve. We find compelling evidence of ZSL, as well as unexpected results which shed light on other factors contributing to ZSL.
Latest publications
BEDTime: A unified benchmark for automatically describing time series
The first benchmark dataset to assess models on each task, comprising four datasets reformatted for these tasks.
NeurIPSSearching for efficient linear layers over a continuous space of structured matrices
Searching for efficient linear operators with optimal scaling laws leading to the development of the BTT-MoE architecture.
NeurIPSScaling-laws for large time-series models
Discovering power-law scaling relationships in large time-series transformer models, analogous to those found in language models.
NeurIPS