SAINT: improved neural networks for tabular data via row attention and contrastive pre-training
A novel deep learning model with row attention and contrastive pre-training for enhanced performance on tabular data tasks.
Topics:
Tabular data underpins numerous high-impact applications of machine learning from fraud detection to genomics and healthcare. Classical approaches to solving tabular problems, such as gradient boosting and random forests, are widely used by practitioners. However, recent deep learning methods have achieved a degree of performance competitive with popular techniques. We devise a hybrid deep learning approach to solving tabular data problems. Our method, SAINT, performs attention over both rows and columns, and it includes an enhanced embedding method. We also study a new contrastive self-supervised pre-training method for use when labels are scarce. SAINT consistently improves performance over previous deep learning methods, and it even outperforms gradient boosting methods, including XGBoost, CatBoost, and LightGBM, on average over a variety of benchmark tasks.
Latest publications
Searching for efficient linear layers over a continuous space of structured matrices
Searching for efficient linear operators with optimal scaling laws leading to the development of the BTT-MoE architecture.
NeurIPSScaling-laws for large time-series models
Discovering power-law scaling relationships in large time-series transformer models, analogous to those found in language models.
NeurIPSSimplifying neural network training under class imbalance
Improving neural network performance on imbalanced datasets by tuning standard training components, without specialized methods.
NeurIPS