3
$\begingroup$

By efficient I mean, that why is the brain able to learn tasks, for example driving, at a much less energy cost than Machine learning models?

From a quick google search it seems like it took about 55 million Kwh to train GPT-4, whereas a human consumes around 3 Kwh in a day (2500 calories, food consumption) with a total of 87 thousand Kwh for a human who lives till the age of 80.

Quite apparently, there is a huge difference in both ability and efficiency.

The reason I am asking this question here and not on the neuroscience stack-exchange is because I am hoping to get a physical perspective.

$\endgroup$
18
  • 1
    $\begingroup$ It is a different technology, we are going to get there or better, eventually $\endgroup$ Dec 26, 2023 at 23:19
  • 1
    $\begingroup$ different how ? $\endgroup$ Dec 26, 2023 at 23:20
  • 1
    $\begingroup$ It would take far more than a human lifetime to read everything that was fed into GPT. $\endgroup$
    – Jon Custer
    Dec 26, 2023 at 23:20
  • $\begingroup$ Do you see any difference between an integrated circuit and biological tissue? $\endgroup$ Dec 26, 2023 at 23:21
  • 3
    $\begingroup$ You mention you put this on Physics.SE rather than Neuroscience.SE. But the reality is that the difference between the brain and a silicon based AI/ML algorithm has little-to-nothing to do with the physics difference, and everything to do with the emergent behaviors that occur on top of the physics. May I recommend moving this to the Artificial Intelligence stack exchange, whose sole purpose is to answer these questions? $\endgroup$
    – Cort Ammon
    Dec 27, 2023 at 16:28

4 Answers 4

4
$\begingroup$

You are observing that training one large computer model took about 1000 times the energy that one human will burn in their lifetime. But you are forgetting that the human brain has been honed for its current skillset over many thousands of generations, and that there are many thousands of humans whose intellectual output went into training this computer model. "Many thousands" is in both cases an offensive understatement — especially when you consider that the problem-solving and pattern-matching abilities of the human brain are also present, to various degrees, in our various mammalian cousins.

I don't think your question will be answered by some physics digression about the relative energy requirements of transistors as compared with neurons. I think the meat of your question is about deciding carefully which energy inputs to include so that you are really doing an apples-to-apples comparison.

$\endgroup$
3
  • $\begingroup$ It needed a human brain to make a neural net; so energy to produce neural net > energy to produce human brain :D $\endgroup$
    – innisfree
    Dec 27, 2023 at 6:39
  • $\begingroup$ I would look at it this way to though: could computer architectures inspired by the human brain improve efficient? Seems we May answers soon if not already purdue.edu/newsroom/releases/2019/Q1/… $\endgroup$
    – innisfree
    Dec 27, 2023 at 6:42
  • $\begingroup$ Well, it takes two adults to make a baby, but that doesn't imply that producing a new human adult takes twice as much energy as producing each of their adult parents. $\endgroup$
    – rob
    Dec 27, 2023 at 11:40
1
$\begingroup$

In my opinion there is no physics angle here.

Compared to humans the current state of machine learning is vastly inefficient.

The difference is at the level of information processing.

I think the following youtube video, published in April 2023, is quite illuminating Why AI art struggles with hands

I think the content of that video illustrates vividly that Machine learning is processing information in a way that is vastly different from how biological intelligence is processing information.

Another vivid example is something that Andrej Karpathy described in the April 2019 Tesla autonomy day, discussing object recognition. The engineers noticed: when the object recognition system was presented with an image of a car with a bicycle strapped to the back the object recognition system reported recognition of a car, and a bicycle, as separate objects.

In order to give the object recognition system the capability to recognize that a bicycle is being transported by strapping it to the back of a car they had to augment the data set with a large number of annotated images of cars with a bicycle strapped to the back.

As we know: for us humans: even a person who has never seen an instance of a bicycle strapped to car will instantly recognize it. You are aware of the general concept of strapping something to a car, and you know bicycles. For a human that is sufficient.


It seems to me that in the human brain information is flowing far more freely than in machine learning.

It seems highly plausible that in the brain information is processed both top-down, and bottom-top.

Example:
In the visual cortex large areas responsible for object recognition receive the signals from the retina in such a way that the color information is not there. That is, large areas of the visual cortex receive as input information comparable to the information of a movie in grayscale, only shades of grey. A comparitively smaller area is dedicated to processing color information. Cross-links between the various areas enable the information flow that allows the brain to assign color to the objects we recognize.

(This explains why watching a black-and-white movie is not particularly straining for us. Much of our image processing does not use color information anyway.)


In any information processing there is a trade-off between reliability and speed.

Supposition:
Mental diseases such as schizophrenia are disorders where information is flowing between specialized areas of the brain too freely.

Supposition: All humans are actually teetering on the brink of mental disorder. Evolution has found a way to make the brain extraordinarily efficient at information processing. It would appear evolution has pushed the brain to the edge of a cliff. You are very high functioning, but if for whatever reason you slip over the edge you will tend to slide down ever further


Contrast:
The way that machine learning processes information is robust. Supposition: the very factors that make the process robust also make it inefficient.

$\endgroup$
2
  • $\begingroup$ Large parts of the brain are hardwired to handle certain types of problems (vision, language, etc.). $\endgroup$
    – Jon Custer
    Dec 27, 2023 at 1:20
  • $\begingroup$ @JonCuster There is that specialization, of course. At the same time there is a strong tendency to grow lateral information flows. For instance, I think synesthesia is a level of lateral information flow that high enough to make it to the level of conscious awareness. Presumably everybody has pretty much that degree of lateral information flow, but arranged to remain just below the level of conscious awareness. $\endgroup$
    – Cleonis
    Dec 27, 2023 at 9:05
1
$\begingroup$

That's a very interesting question. The brain and machine learning models are both capable of learning tasks, but they do so in very different ways. The brain is a biological system that consists of billions of neurons and synapses, which communicate with each other through electrical and chemical signals. The brain learns by adjusting the strength of these connections based on the feedback it receives from the environment. This process is called synaptic plasticity, and it allows the brain to adapt to changing situations and acquire new skills.

Machine learning models, on the other hand, are artificial systems that are designed to learn from data using mathematical algorithms. Machine learning models can be divided into two types: supervised and unsupervised. Supervised models learn from labeled data, which means that each input has a corresponding output or target value. For example, a supervised model can learn to recognize handwritten digits by being trained on images of digits with their correct labels. Unsupervised models learn from unlabeled data, which means that there is no predefined output or target value for each input. For example, an unsupervised model can learn to cluster similar images together by finding patterns in the data.

The main difference between the brain and machine learning models is that the brain learns by predicting its own perceptions, while machine learning models learn by optimizing their own outputs. The brain uses prior knowledge and expectations to make inferences about the causes of sensory inputs, while machine learning models use objective criteria and constraints to minimize errors between their outputs and targets. The brain is more flexible and robust than machine learning models, as it can handle uncertainty and ambiguity better than any fixed algorithm.

However, this does not mean that the brain is superior to machine learning models in every aspect. Machine learning models have some advantages over the brain in terms of speed, scalability, accuracy, and generalization. Machine learning models can process large amounts of data faster than the brain can process information from its limited sensory channels. Machine learning models can also be trained on specialized hardware like graphics processor units (GPUs), which draw more power than traditional CPUs. Machine learning models can achieve higher accuracy than the brain in some tasks that require precise calculations or measurements. Machine learning models can also generalize better than the brain in tasks that involve new or unseen data.

Therefore, the answer to your question depends on what kind of task you are interested in. If you want to learn how to drive a car or speak a foreign language, then you might benefit more from using your own brain than any machine learning model. But if you want to perform complex scientific calculations or analyze large datasets or create realistic images or sounds, then you might benefit more from using a machine learning model than your own brain.

As an example, chess bots like stockfish are highly accurate and formidable than GM's.

$\endgroup$
-7
$\begingroup$

The brain is a color wheel. It just recombines information by laying it on top of each other, rather than processing.

Processing has a thermal limit due to temperature. The rough form of the limit is

C = material heat capacity D = energy per unit information E = total energy or information

E = C/D

Increasing the heat capacity of the material or reducing the energy per work unit increases the total information capacity, which can also be expressed as time. Heat flux is implied as a denominator as it is a rate at which the other variables operate.

The brain avoids this by simply not processing information; its D is zero. It instead overlays information. Humans are parrots and repeat back what they see, and it displays logic by accident due to overlay of information occurring in related concepts. In other words, humans are "good at" word salad.

Because no information is being processed, only retrieved, the constraint falls to the energy required to store information which is much lower than the energy to do logical work with it. As a result humans can be efficient even though they are just regurgitating.

Because computer storage is inefficient, and requires objects much bigger than a neuron to store information, it has nowhere near this IO capacity and fails. A neuron is about a billion atoms whereas a hard drive magneton is on the order of quadrillions of atoms.

$\endgroup$
2
  • $\begingroup$ Can you cite any sources ? $\endgroup$ Dec 26, 2023 at 23:26
  • 3
    $\begingroup$ Very wrong at many levels. $\endgroup$ Dec 26, 2023 at 23:28

Not the answer you're looking for? Browse other questions tagged or ask your own question.