Imagine 2 snooker balls in a table: Every time you shoot one against the other, the faster moving ball will end up moving slower (even if just a little) and the one moving slower will move faster by a corresponding amount. In other words, for 2 isolated objects, the motion always passes from the one that has more to the one that has less$^*$, even though the opposite is absolutely compatible with energy conservation$^*$$^*$.
At a microscopic level, heat transfer is just the process of transferring motion from some particles to the next, and therefore, just as before, the ones with less motion (colder) will gain momentum (heat up) by taking it from the ones with more (hotter). At equilibrium the average quantity of motion at both objects will match, because if the initially colder object gets hotter the process moves in the opposite direction.
All this said, now the caveat: imagine that instead of just 2 balls you have the full set on the table, each one at its own speed. If you just concentrate yourself into a single slowing moving ball, there is a chance that after being stroked by a faster moving one it strike(s) other(s) at its vicinity and ends up moving slower (or even stop). This means that, (at the vicinity of) that ball speed was reduced (it got colder) after being stroked by a faster moving one (a hotter one). However, after sometime, there is a huge chance that the ball will get stroked again and heat up. What this means is that locally at a (very) small part of the spoon, and for a (very) short time interval, its temperature may be reduced by the colder ice cream. However, at the big scale and for a considerable amount of time the overall process will be the spoon cooling down and the ice cream eating up.
In conclusion hotter objects are just much more likely to get colder in overall than the opposite.
$^*$ This is the second law of thermodynamics.
$^*$$^*$ This is the first law of thermodynamics.