0
$\begingroup$

I have learned that people in the field of machine learning and statistics often use the least squares method. Is there any alternative formulation of the least squares method using entropy? (It might be better to ask if it is related to entropy in physics or information theory.)

Sorry for the very vague question, but when I saw it I got a naive idea that it is kind of similar to what I learned about entropy in some of the Physics courses I took at University.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ They mean Shannon entropy rather than the entropy used in physics (although sometimes the two overalap), see, e.g., here. I think the question is more suitable for cross-validated. $\endgroup$
    – Roger V.
    Commented May 7, 2022 at 6:06

1 Answer 1

Reset to default
1
$\begingroup$

Here's one important sense in which the answer is yes. Least squares is equivalent to maximum likelihood estimation for IID (and hence equal-variance) zero-mean Gaussian errors. Their being Gaussian maximizes continuous Shannon entropy for their mean and variance.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.