Skip to main content
Added explanation of overfitting, requested in comments
Source Link
Clumsy cat
  • 183
  • 1
  • 8

Taking a slightly different angle on the question the answer that "we don't need more" is definitely correct, but further, I would say we don't need more, and the less the better.

ThisTo answer why this is we need to know about a problem sometimes referred to as overfitting. The idea behind overfitting is this; say I design a predictive model, for example I wish to obtain a relationship between the qualities of oak sap and the age of an oak. I learn the ages of a 100 oaks from records of their planting and I measure the viscosity of their sap, the colour, the density, the water content, and 46 more observables. I start by just looking at the relationship between viscosity and age. Maybe using a quadratic model, with 3 parameters. It's quite a strong relationship, and I can get within about 20 years of the age of each tree using just the viscosity of the sap. But there are 49 more variables! So I construct a complex model for the age of a tree that predicts how the age should effect all 50 variables, and then I fit the 150 parameters of this model. By tuning the complex model I can predict the ages of all 100 trees to within a day, how wonderful. Then someone shows me another oak, they say "I know how old this oak is, let's use it to test your model". The simple, viscosity only, model an the complex model give very different answers. The simple model is much more accurate at predicting the age of the new tree. Why? Because the complex model was not really learning the ages of the 100 trees at all, it was learning to identify the 100 trees using noise in the data that was unrelated to their age, and then I had tuned it's parameters so that it would give me the correct weight for each tree. It wasn't possible to do this for the simple model, because there were not enough parameters available for me to tune, so the simple model was forced to find a genuine correlation if there was one to be found. The complex model is overfitting.

And so the more complexity there is in any model the greater it's tendency to overfit. When a model starts overfitting it loses predictive power. I am quite confident that someone with enough time an patience could derive a model that had 5 forces in it. They might even find a model that was more intuitive, or computationally easier to work with. But that more complex model would be less likely to fit new unseen results, because its complexity would be tuned to the results that we have seen, and also likelyso use the noise in ourto identify the data and give us the answer we tried to fit. This noise is not something we wanted to fit, and a simpler model is less likely to have the capacity to fit noise in the first place.

Taking a slightly different angle on the question the answer that "we don't need more" is definitely correct, but further, I would say we don't need more, and the less the better.

This is because the more complexity there is in any model the greater it's tendency to overfit. When a model starts overfitting it loses predictive power. I am quite confident that someone with enough time an patience could derive a model that had 5 forces in it. They might even find a model that was more intuitive, or computationally easier to work with. But that more complex model would be less likely to fit new unseen results, because its complexity would be tuned to the results that we have seen, and also likely the noise in our data. This noise is not something we wanted to fit, and a simpler model is less likely to have the capacity to fit noise in the first place.

Taking a slightly different angle on the question the answer that "we don't need more" is definitely correct, but further, I would say we don't need more, and the less the better.

To answer why this is we need to know about a problem sometimes referred to as overfitting. The idea behind overfitting is this; say I design a predictive model, for example I wish to obtain a relationship between the qualities of oak sap and the age of an oak. I learn the ages of a 100 oaks from records of their planting and I measure the viscosity of their sap, the colour, the density, the water content, and 46 more observables. I start by just looking at the relationship between viscosity and age. Maybe using a quadratic model, with 3 parameters. It's quite a strong relationship, and I can get within about 20 years of the age of each tree using just the viscosity of the sap. But there are 49 more variables! So I construct a complex model for the age of a tree that predicts how the age should effect all 50 variables, and then I fit the 150 parameters of this model. By tuning the complex model I can predict the ages of all 100 trees to within a day, how wonderful. Then someone shows me another oak, they say "I know how old this oak is, let's use it to test your model". The simple, viscosity only, model an the complex model give very different answers. The simple model is much more accurate at predicting the age of the new tree. Why? Because the complex model was not really learning the ages of the 100 trees at all, it was learning to identify the 100 trees using noise in the data that was unrelated to their age, and then I had tuned it's parameters so that it would give me the correct weight for each tree. It wasn't possible to do this for the simple model, because there were not enough parameters available for me to tune, so the simple model was forced to find a genuine correlation if there was one to be found. The complex model is overfitting.

And so the more complexity there is in any model the greater it's tendency to overfit. When a model starts overfitting it loses predictive power. I am quite confident that someone with enough time an patience could derive a model that had 5 forces in it. They might even find a model that was more intuitive, or computationally easier to work with. But that more complex model would be less likely to fit new unseen results, because its complexity would be tuned to the results that we have seen, and so use the noise to identify the data and give us the answer we tried to fit. This noise is not something we wanted to fit, and a simpler model is less likely to have the capacity to fit noise in the first place.

Source Link
Clumsy cat
  • 183
  • 1
  • 8

Taking a slightly different angle on the question the answer that "we don't need more" is definitely correct, but further, I would say we don't need more, and the less the better.

This is because the more complexity there is in any model the greater it's tendency to overfit. When a model starts overfitting it loses predictive power. I am quite confident that someone with enough time an patience could derive a model that had 5 forces in it. They might even find a model that was more intuitive, or computationally easier to work with. But that more complex model would be less likely to fit new unseen results, because its complexity would be tuned to the results that we have seen, and also likely the noise in our data. This noise is not something we wanted to fit, and a simpler model is less likely to have the capacity to fit noise in the first place.