Scaling-laws for large time-series models
Discovering power-law scaling relationships in large time-series transformer models, analogous to those found in language models.
Scaling laws for large language models (LLMs) have provided useful guidance in training ever larger models for predictable performance gains. Time series forecasting shares a similar sequential structure to language, and is amenable to large-scale transformer architectures. Here we show that foundational decoder-only time series transformer models exhibit analogous scaling-behavior to LLMs, with architectural details (aspect ratio and number of heads) having a minimal effect over broad ranges. We assemble a large corpus of heterogenous time series data on which to train and establish for the first time power-law scaling with parameter count, dataset size and training compute that spans five orders of magnitude.
Latest publications
BEDTime: A unified benchmark for automatically describing time series
The first benchmark dataset to assess models on each task, comprising four datasets reformatted for these tasks.
NeurIPSInfluence functions for efficient data selection in reasoning
A proposal to define reasoning data quality using influence functions.
NeurIPSContinual pre-training of MoEs: how robust is your router?
A systematic study of Mixture of Experts (MoE) continual pre-training.
NeurIPS