Note that : $$\left(\frac{d}{dx}\right)^\dagger=-\frac{d}{dx}$$
$$a=(\cdots )X+(\cdots )\frac{d}{dx}$$ $$a^\dagger=(\cdots )X-(\cdots )\frac{d}{dx}$$ As expected $a\not=a^\dagger$.
Edit: It's not a rigorous proof $$P=P^\dagger $$ $$P\rightarrow -i\hbar \frac{d}{dx}$$ $$-i\hbar \frac{d}{dx}=\left(-i\hbar \frac{d}{dx}\right)^\dagger=+i\hbar\left(\frac{d}{dx}\right)^\dagger$$ $$\Rightarrow \left(\frac{d}{dx}\right)^\dagger=-\frac{d}{dx} $$