The scaling hypothesis is the theory, that neural networks will continually increase in cognitive capability, as we make the model bigger. We can scale a model in three ways: 1. increase the training compute 2. have more training data 3. increase the amount of parameters The scaling hypothesis has been true thus far, but every big breakthrough in deep learning have come to the surprise of many, who expected eventual diminishing returns from scaling the model. It still seems likely that increasing scale will bring model improvements, however experts continue to question the continuation of the scaling hypothesis, with many saying that the scaling hypothesis will not take us to [[AGI is a type of AI that would match or surpass human capabilities across virtually all cognitive tasks|AGI]]. We are also reaching limits in both compute and training data, which will probably slow down the scaling of models.