My professor told us that in quantum mechanics a transformation is a symmetry transformation if $$ UH(\psi) = HU(\psi) $$
Can you give me an easy explanation for this definition?
In a context like this, a symmetry is a transformation that converts solutions of the equation(s) of motion to other solutions of the equation(s) of motion.
In this case, the equation of motion is the Schrödinger equation $$ i\hbar\frac{d}{dt}\psi=H\psi. \tag{1} $$ We can multiply both sides of equation (1) by $U$ to get $$ Ui\hbar\frac{d}{dt}\psi=UH\psi. \tag{2} $$ If $UH=HU$ and $U$ is independent of time, then equation (2) may be rewritten as $$ i\hbar\frac{d}{dt}U\psi=HU\psi. \tag{3} $$ which says that if $\psi$ solves equation (1), then so does $U\psi$, so $U$ is a symmetry.
For a more general definition of symmetry in QM, see
What you have written there is nothing but the commutator. Consider for example the time evolution operator \begin{align*} U\left(t-t_{0}\right)=e^{-i\left(t-t_{0}\right) H} \end{align*} If $\psi\left(\xi_{1}, \dots, \xi_{N} ; t_{0}\right)$ is the wave function at time $t_0$ and $U(t−t0)$ is the time evolution operator that for all permutations $P$ satisfies $\left[U\left(t-t_{0}\right), P\right]=0$ then also $$\left(P U\left(t-t_{0}\right) \psi\right)\left(\xi_{1}, \ldots, \xi_{N} ; t_{0}\right)=\left(U\left(t-t_{0}\right) P \psi\right)\left(\xi_{1}, \ldots, \xi_{N} ; t_{0}\right)$$ This means that the permuted time evolved wave function is the same as the time evolved permuted wave function.
Another example would be if you consider identical particles. An arbitrary observable $A$ should be the same under the permutation operator $P$ if one has identical particles. This is to say: \begin{align*} [A, P]=0 \end{align*} for all $P\in S_N$ (in permutation group of $N$ particles).