The gold objects you are pushing together are not sufficiently flat to stick, at the atomic level. I imagine that the contact at those points that actually touch may well be sufficient for fusion, but at the atomic scale there is so little contact that you can simply separate the objects apart like nothing happened.
Experiment: if you hammer two pieces of rust-free iron together, they stick a bit, because hammering them together improves how much of the metal actually touches at the atomic level. But the contact is still not very good, especially since iron has the oxide layer you mention.
Cold welding makes use of the idea that if you make the metals truly contact each other without any contamination, they will actually stick together. Quote from none other than Richard Feynmann, answering your question literally:
The reason for this unexpected behavior is that when the atoms in contact are all of the same kind, there is no way for the atoms to “know” that they are in different pieces of copper. When there are other atoms, in the oxides and greases and more complicated thin surface layers of contaminants in between, the atoms “know” when they are not on the same part.
In particular, gold does actually stick together exceptionally well in the right conditions, the right conditions being nanoscale flatness, aligned crystalline structures, gold purity and clean surfaces:
single-crystalline ultrathin gold nanowires (diameters less than 10 nm) can be cold-welded together within seconds by mechanical contact alone, and under remarkably low applied pressures