We have the annihilation and creation operators $a$ and $a^\dagger$, respectively (we know that they are the hermitian conjugates of each other, but we won't assume that fact). So let's rename $a^\dagger=b$.
I'll try to show that $a=a^\dagger$ and $b=b^\dagger$, gives a contradiction. We know:
$$ba\vert n\rangle=n\vert n\rangle \quad, \quad ab\vert n\rangle=(n+1)\vert n\rangle$$We are supposing that $a^\dagger$ is not the hermitian conjugate of $a$.
With this you could prove: $[a,b]=\mathbb{I}$
Then:
$$n^2\langle n \vert n\rangle=\langle n\vert a^\dagger b^\dagger ba\vert n \rangle=\langle n\vert a b ba\vert n \rangle=$$
$$=\langle n\vert[( b^\dagger a^\dagger+\mathbb{I}) ba]\vert n \rangle=n(n+1) \langle n\vert n \rangle+n \langle n\vert n \rangle$$
Where I have used the commutation relation and the hermiticy property. This yields: $n=0$ or $\vert n\rangle =0$.
So unless $\vert n\rangle=\vert 0\rangle$ or $0$, $a$ and $a^\dagger$ won't be hermitian (I don't think it holds if the only state is $\vert 0 \rangle $ or $0$ but I'm not sure).
Is it because they 'create' and 'annihilate' photons
This reasoning would be very sloppy. Ladder operators arise in the context of the harmonic oscillator, angular momentum... But in Quantum Mechanics, particle number is conserved (you could say something like it modifies the energy, hence it emmits a photon or something like that). These operators create and destroy photons in Quantum Field Theory.