For simplicity, let us talk about a scalar field $\phi : \mathbb{R}^4 \rightarrow \mathbb{R}$.
The action for a free scalar field is
$$S[\phi] = \frac{1}{2}\int_{\mathbb{R}^4} \partial_\mu\phi\partial^\mu\phi - m^2\phi^2$$
and its classical equations of motion is the Klein-Gordon equation
$$ (\partial_\mu\partial^\mu + m^2) \phi = 0 $$
Now that looks suspiciously like an oscillator or wave equation, doesn't it? This inspires us to do a Fourier transform to obtain the eigenfunctions $\mathrm{e}^{\mathrm{i}px}$ solving this equation. The general solution $\phi(x)$ can then be expanded as
$$\phi(x) = \int \frac{\mathrm{d}^3p}{(2\pi)^3} \frac{1}{\sqrt{2\omega_p}}(a(\vec p)\mathrm{e}^{\mathrm{i}px} + a^\dagger(\vec p)\mathrm{e}^{-\mathrm{i}px}) $$
which is precisely the expansion one could do for any other oscillator. Now, you can talk about the modes $a(\vec p)$ and $a^\dagger(\vec p)$ of the field being excited, and you can imagine $\mathrm{e}^{\mathrm{i}px}$ describing a (basic) oscillation at any point $x$, and talk about the integral representing the field being made out of such oscillators.
This is all nonsense.
This might sound strong, but it has been the source of many annoying misunderstandings in the publicization of quantum theories to laypeople.
Just because something ($\phi$) fulfills a wave/oscillator equation and has a mode expansion (as the above is called), it does not mean that anything oscillates. It's just the same type of equation you encounter in oscillator, not the same physical situation. It's a nice pretty picture to tell ourselves that we understand the quantum field, but ultimately, there is nothing there that would justify the oscillator interpretation. Nothing physical is vibrating or oscillating here.
Furthermore, the above only holds for a free, non-interacting field. When you have a field with arbitrary interactions, its equations of motion may not look at all like the wave/oscillator equation, and it has no modes, so the picture falls apart there completely.
Since the comments show that this is more controversial than I thought, I will elaborate a bit:
Electromagnetic waves also look as if there is a harmonic oscillator at every point in space, by the same logic of mode expansion. The descriptions are formally equivalent. This led people to believe that there is the luminiferous aether, because, how else could empty space carry the wave? But this turned out to be not true, there is no aether, and there is nothing carrying the wave. The formal equivalence is misleading, there is no physical object oscillating when the wave travels through the vacuum.
I am not saying that anything about the formal treatment is false, I am trying to explain why it is not a good idea to believe that the oscillator description is a good physical interpretation of the situation, which is what I believe the OP refers to when he mentions a book saying "This is not true" about the "oscillators at every point in space" idea.