3
$\begingroup$

Wikipedia describes "effective theories" as follows.

In science, an effective theory is a scientific theory which proposes to describe a certain set of observations, but explicitly without the claim or implication that the mechanism employed in the theory has a direct counterpart in the actual causes of the observed phenomena to which the theory is fitted. That means, the theory proposes to model a certain effect, without proposing to adequately model any of the causes which contribute to the effect.

While Wikipedia starts with the context 'in science', I have only seen this language used in Physics so I am asking here on Physics.SE.

Theorems such as the Universal Function Approximation Theorem show that certain classes of artificial neural networks can approximate Lebesque measurable functions to arbitrary precision (up to measure zero) with sufficient width and depth of the layers of the network. This has led me to tell a joke to people that "Neural networks will never be a Theory of Everything, but they are already a Theory of 'Anything'".

Tying the notion of effective theories together with neural networks, do neural networks actually count as effective theories? Or is there a nuance here in which they do not qualify?


Presumably one could ask a similar question about polynomials in the context of the Stone-Weierstrass theorem, but I suspect the answer will be similar in character.

$\endgroup$
2
  • 1
    $\begingroup$ @ChiralAnomaly Yes, that is the type of clearer understanding that I was hoping to find here. I've been reading some of the counterfactualist causality literature and I am aware of the approach of causal set theory, but in general the term "causality" seems to be used rather loosely by most people. Leaving causality aside for this post, this is a rather interesting point you bring up about effective theories as a type of relationship between pairs of theories. It sounds like a high-quality answer waiting to happen. $\endgroup$
    – Galen
    Oct 24 at 17:46
  • 1
    $\begingroup$ I grew the comment into an answer. $\endgroup$ Oct 24 at 17:58
5
$\begingroup$

I don't think this changes the spirit of the question you're asking, but I'll start with a clarification because I think Wikipedia's definition is unclear (I don't know what "actual causes" means). The way most physicists use the term, "effective theory" doesn't refer to a single theory by itself. It's a relationship between two different theories, one of which may be unknown. One theory can be an "effective theory" of another theory (possibly unknown) by reproducing some of that other theory's predictions using less-detailed postulates.

With that clarification, the answer is pretty clearly yes: a neutal network could indeed be crafted to reproduce many of the predictions of some other theory, say QED, so a neural network could indeed be an "effective theory." Most physicists wouldn't be happy with that kind of theory, not even as an "effective theory," because it's not very efficient: every parameter characterizing every neuron, not to mention the topology of the network, is essentially a separate postulate. That's a lot of postulates. But aesthetic criteria aside, the answer is yes: a neural network can be a (very inefficient) effective theory.

Actually, maybe I'm not being fair by assuming that it would be inefficient. Suppose we trained a neural net to reproduce a jillion different experimental results. In order for the training to succeed, the neural-net architecture that we started with needs to be sufficiently expressive. If we're lucky enough to choose an architecture that is just barely expressive enough, then we would indeed have an efficient theory, whether or not we call it "effective." In practice, though, choosing an architecture that is just barely expressive enough seems just as difficult as coming up with an efficient theory the traditional way — by chipping away at the problem over many decades, one publication at a time.

$\endgroup$
3
  • $\begingroup$ Excellent! Often what machine learning specialists do in practice is they pick neural network structures that have more representational capacity than needed, and they reduce the overfitting of such models with regularization. The point of this explication is that unfortunately I don't think ML experts will find such models in most practices. $\endgroup$
    – Galen
    Oct 24 at 18:04
  • $\begingroup$ I have not heard anything about it in a while, but I have long thought that NEAT was an interesting way of exploring the space of models. $\endgroup$
    – Galen
    Oct 24 at 18:05
  • 1
    $\begingroup$ At some level almost all the models I see in Science are special cases from the space of computation graphs, which include neural networks and are used to explicate automated differentiation. Indeed, I have wondered if some arcane choice of computation graph has yet to be found that is even simpler than artificial neural networks but has comparable representational capacity. Anyway, thank you for the clarifying post. It is appreciated. $\endgroup$
    – Galen
    Oct 24 at 18:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.