A comprehensive bibliography: https://www.gwern.net/notes/Scaling

Explaining Neural Scaling Laws

Learning Curve Theory

A Neural Scaling Law from the Dimension of the Data Manifold

Scaling Laws for Transfer

Scaling Laws for Autoregressive Generative Modeling

Scaling Laws for Neural Language Models

On Power Laws in Deep Ensembles

A constructive prediction of the generalization error across scales (ICLR 2020)

Generalization

Multiple Descent: Design Your Own Generalization Curve

Rethinking Bias-Variance Trade-off for Generalization of Neural Networks

----

Deep Learning Scaling is Predictable, Empirically

Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks

Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigm

Different scaling of linear models and deep learning in UKBiobank brain images versus machine-learning datasets

Generalization bounds for deep learning

Foundation Models

AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing