It is helpful here to remember that the expectation value $\langle \hat{a} \rangle$ is taken with respect to a quantum state $|\psi \rangle$ (or a density operator, but let's keep it simple). In this case,
$$
\langle \hat{a} \rangle = \langle \psi | \hat{a} | \psi \rangle
$$Recall now that $\langle \phi | \hat{O} | \psi\rangle^* = \langle \psi | \hat{O}^\dagger|\phi\rangle$. An intuitive picture for this relation comes from the matrix element interpretation of these objects - ie., a linear operator $\hat{O}$ can be expressed as an $N \times N$ matrix where $N$ is the dimension of the Hilbert space. In this case, the 'dagger' operator is the conjugate transpose, so the '$\psi$ by $\phi$'-th element of $\hat{O}^\dagger$ is the complex conjugate of the '$\phi$ by $\psi$'-th element of $\hat{O}$.
Given this relation,
$$
\langle \hat{a} \rangle^* = \langle \psi|\hat{a}|\psi\rangle ^* = \langle \psi | \hat{a}^\dagger |\psi\rangle = \langle \hat{a}^\dagger \rangle.
$$
So indeed the expectation values of $\hat{a}$ and $\hat{a}^\dagger$ are complex conjugates.