Diffeomorphisms do not involve moving each point to a new point. Instead, what they do is give a new label to the very same point. For example, consider a very non-symmetric manifold, such as the spacetime for the solar system. You don't have any kind of equivalence between different points in the spacetime, so you can't physically move them around (although that would be possible in a sphere, for example). Hence, diffeomorphisms are passive coordinate transformations. When you write the expression for the pullback of a scalar function, notice it simply says that the function keeps its value on a point after relabeling the coordinates. That is because the point is still the same, only the coordinates changed.

Diffeomorphisms do not involve moving each point to a new point. Instead, what they do is give a new label to the very same point. For example, consider a very non-symmetric manifold, such as the spacetime for the solar system. You don't have any kind of equivalence between different points in the spacetime, so you can't physically move them around. Hence, diffeomorphisms are passive coordinate transformations. When you write the expression for the pullback of a scalar function, notice it simply says that the function keeps its value on a point after relabeling the coordinates. That is because the point is still the same, only the coordinates changed.