To quote from Wikipedia on elastic scattering, "In this scattering process, the energy (and therefore the wavelength) of the incident photon is conserved and only its direction is changed."

How does this work? The impacted particle will also start moving to conserve momentum. So it'll also carry some energy. How can the photon not lose any energy during the collision?

Another possibly unrelated question: how exactly can the photon and the particle interact? The photon essentially carries information about a changing electromagnetic field. How can it influence a particle that carries neither charge nor magnetism?

-

Here's a qualitative argument for why this is true. Let $p$ be the momentum of the incoming photon. The momentum transferred to the massive particle is at most of order $p$ (specifically, it's at most $2p$). For a nonrelativistic particle, the kinetic energy is $p^2/(2m)$, so the energy transferred is of this order. So the fractional energy lost by the photon is of order $${\Delta E\over E}\sim{p^2/2m\over pc}={pc\over 2mc^2}={E\over 2mc^2},$$ where $E =pc$ is the original photon energy. So when $E\ll mc^2$, the change in energy is small in comparison to the original energy.
One common way to express this more quantitatively is via the Compton scattering formula, $$\Delta\lambda={h\over mc}(1-\cos\theta),$$ where $m$ is the mass of the massive particle and $\theta$ is the angle through which the photon scattered. This formula is usually used for much higher-energy photons scattering off charged particles, but it is derived just from energy and momentum conservation, so it applies to any situation in which a photon scatters off a massive particle initially at rest.
If I've done the arithmetic right, when $m$ is the mass of an atom or molecule, $\Delta\lambda$ comes out to be of order $10^{-16}$ m, which is certainly small enough to be ignored for visible light.