I am studying quantum statistical mechanics from the mathematician's perspective. I don't quite understand what the duality formula for entropy is really saying (or why there is a "duality").
If $A$ is an $n\times n$ Hermitian (self-adjoint, diagonalizable) matrix, then $$S(A)=\sup\bigl[{\operatorname{Tr}(AH) - \ln\bigl(\operatorname{Tr}[E^H]\bigr) : H \in H_n}\bigr]$$ where $H_n$ is the set of Hermitean matrices.
What does this even mean?
The entropy of the (density) matrix $A$, usually denoted $\rho$, is evaluated as the supremum (i.e. maximum that may never be realized, just arbitrarily closely approached) of the trace of the product of matrices $AH$ minus the natural logarithm of $\exp(H)$, the exponential of $H$. The supremum is taken over all Hermitean matrices $H$ of the same size as $A$.
In practice, when you try to maximize this expression, you will find out that the best choice is $$ H = \ln(A) + C\cdot {\bf 1} $$ In words, the ideal matrix $H$ that gives you the supremum is the logarithm of the matrix $A$ (there should be a minus sign somewhere to get the right conventions for entropy but I will overlook this detail to agree with the literature below). The choice of $C$ doesn't matter because the piece proportional to the unit matrix gets subtracted.
See a proof of this formula e.g. as theorem 2.13 in
which also provides you with some background, as much as you need.
This is just a comment to point out that it is called a duality because (minus) the entropy formula that OP mentions is the (generalized) Legendre transform$^1$ of the convex function $$H~\mapsto~\ln {\rm Tr}(e^H).$$ If one performs the Legendre transformation, it is possible to recover (minus) the von Neumann entropy
$$ -S(A)~=~\left\{\begin{array}{ccl} {\rm Tr}(A\ln A)& &\text{if} ~ A ~\text{is a (semi)positive operator with}~ {\rm Tr}A=1, \\ \\ \infty && \text{otherwise}. \end{array} \right. $$
--
$^1$ Also known as the convex conjugate.
