The reason that creation and destruction operators don't commute is that, on top of 'moving a state up and down energy levels', they multiply it by a number in the process, and this number depends on where you are in the ladder. More specifically,
$$\begin{cases}
\hat{a}|n\rangle&=\sqrt{n}|n-1\rangle,\text{ while}\\
\hat{a}^\dagger|n\rangle&=\sqrt{n+1}|n+1\rangle.
\end{cases}$$
Thus, if you act on a number state $|m\rangle$ with a creation operator first, you will get the $\sqrt{n+1}$ factor with $m=n$, but the $\sqrt n$ factor with $n=m+1$, so you will get $\hat a \hat a^\dagger|m\rangle=(m+1)|m\rangle$. Conversely, if you act with the destruction operator first, you will get a factor with $\sqrt n$ at $n=m$, and then a factor of $\sqrt{n+1}$ at $n=m-1$, so the result will be $\hat a^\dagger \hat a|m\rangle=m|m\rangle$.
Ultimately, though, there is a more fundamental reason why creation and annihilation operators can't commute, and it is the fact that energy levels are bounded from below: that is, there are no $|n\rangle$ with $n<0$. To see why this is the case, consider the creation and annihilation operators without the prefactors I just talked about, to get
$$\begin{cases}
\hat{E}\,|n\rangle&=|n-1\rangle\text{ and}\\
\hat{E}^\dagger|n\rangle&=|n+1\rangle.
\end{cases}$$
The argument above no longer holds, and both routes will return the same coefficient on $|m\rangle$ after applying $\hat E$ and $\hat E^\dagger$ in both orders. The problem, however, is what happens to the ground state? The formulas above are mostly fine, but they do not specify what $\hat E|0\rangle$ should be, and there is no $|-1\rangle$ state we can put it into. However, if we want both operators to be hermitian conjugates, then we have really no room to play with, because the identity
$$
\langle n|\hat E|0\rangle^\ast=\langle 0|\hat E^\dagger|n\rangle=\langle 0|n+1\rangle=0
$$
implies that $\hat E|0\rangle$ has zero component along $|n\rangle$ for all $n\geq0$; since that is a complete set, it implies that $\hat E|0\rangle=0$.
Finally, then, if you compute $\hat E\hat E^\dagger |0\rangle$ as above, you will get $|1\rangle$ as normal, but $\hat E^\dagger \hat E|0\rangle$ returns 0, which is different, and the operators don't commute:
$$\hat E\hat E^\dagger =1\text{ but }\hat E^\dagger \hat E=1-|0\rangle\langle 0|.$$
The only way around this is to allow for an infinite ladder of negative-energy eigenstates, which are completely unphysical, but do allow for commuting ladder operators. While these states have indeed appeared in the literature they are unwieldy to use as they defy all intuition and they have not caught on.
So, to answer your final question: the asymmetry in the harmonic oscillator that ultimately causes this behaviour is the fact that the ladder of energy eigenstates is bounded from below but not from above, which makes going up and down the ladder not completely equivalent.