For me this is most easily clinched by the fact that the annihilation operator annihilates the vacuum:
$$\hat a|0\rangle=0.$$
If you're having trouble remembering which of $a$ and $a^\dagger$ is the annihilation/creator operator, this form may be useful; a-times-zero-is-zero sounds about as nice a mnemonic as you're likely to get, at least on the symbols side. (Of course, this requires you to beware of misremembering as $a^\dagger|0\rangle\stackrel{\text{!}}{=}0$. This form is more complicated - it's got more symbols -, so one way to remember it is to go for the simpler of the two.)
Further than that, if you know the general form of the relations,
$$
\begin{align}
a^\dagger |n\rangle &= \sqrt{n(+1?)} |n(\pm?)1\rangle,
\\
a|n\rangle& = \sqrt{n(+1?)} |n(\pm?)1\rangle,
\end{align}
$$
but you're having trouble remembering their exact form,
you can settle the ambiguity by noting that since $a|0\rangle$ must vanish, the coefficient must be $\sqrt{n}$; the coefficient of $a^\dagger$ must therefore be $\sqrt{n+1}$. The fact that $a$ 'kills' something also settles it as the annihilation operator, which means that it must take $|n\rangle$ to $|n-1\rangle$, and that leaves simply $|n+1\rangle$ for the final choice.
Of course, as with any mnemonics, it's a completely personal thing whether any particular mnemonic works for you or not. Also, as time goes on you will either use those relations often enough that you don't need a mnemonic, or you will stop using it and stop needing to remember.