If there are enough data and the prior is not completely unreasonable, the frequentist and the Bayesian approach give essentially the same answer. This is related to the central limit theorem.
If data are fairly scarce, the two approaches may differ a lot. In this case the Bayesian approach is far preferable but only if the prior reflects true prior knowledge and not just prejudice. (With a drastically wrong prior and limited data, the Bayesian approach tends to reaffirm the prejudice, and hence will be far worse than the frequentist result.)
To see this, take a prior and assume that just one observation. It is clear that the Bayesian outcome is just a small change of the prior. So if the prior was appropriate (reflected true knowledge), the result is an improvement, while if the prior was bogus (just prejudice), the outcome is as bad. If the number of observations is large, their contribution dominates the outcome, and the result is essentially prior-independent, and is easily seen to agree with the frequentist (maximum likelihood) result.
All this is completely independent of physics. But in statistical mechanics we have a good enough theory that enables us to choose appropriate priors. This is the (only) reason why the maximum entropy principle works there.