Now if the interaction things with light can be understood in terms of the interactions with the fields, what does the size of the Poynting vector actually mean in practice? Are there any processes that depend on this rather than directly on the amplitude of the electric or magnetic field?
The Poynting vector, a function of total electric and total magnetic field, is interpreted as density of flux of EM energy in macroscopic theory (watts per unit area), due to energy interpretation of the Poynting theorem. The Poynting theorem is the following mathematical relation based solely on Maxwell's equations:
$$
\mathbf j \cdot \mathbf E = -\nabla \cdot \mathbf S - \partial_t w
$$
where $w$ is the well-known Poynting energy density expression in terms of $E^2,B^2$.
The energy interpretation of this equation adopts additional assumption: that $\mathbf j \cdot \mathbf E$ is an accurate expression of total work per unit volume per unit time done by EM field on matter, so the terms on the right-hand side of this equation can be interpreted as convergence of EM energy flux density vector and rate of decrease of energy density at the same point of space. Under this assumption, Poynting's theorem becomes a law of local conservation of energy of matter and EM field. One can say work done on matter particle in a region of space by EM forces equals EM energy that came in through the surface of the region from the outside, plus decrease of EM energy in the region.
Such interpretation of the Poynting theorem equation is incorrect or correct but unfeasible in microscopic EM, where total fields are $\mathbf e, \mathbf b$, for the following reasons:
if the charged particle that is experiencing EM force is a point, the term $\mathbf j \cdot \mathbf e$ is too singular and its meaning cannot be consistently defined; total electric field diverges at the particle. So the assumption about $\mathbf j\cdot \mathbf e$ is invalid and Poynting's theorem while true, is useless. We can derive another similar theorem that can be interpreted as local conservation of energy, if we base it on some regular expression for EM work. There is the Frenkel theory which defines its own energy density and energy flux density expressions based on the idea that point particles experience only external field, and their own field does not affect them. This leads to expressions for EM energy and EM energy flux density that are different from the Poynting ones.
if the charged particles are extended charge distributions, the term $\mathbf j \cdot \mathbf e$ is fine and Poynting's vector formula is fine, but $\mathbf e$ is usually unknown to us, because near and inside the particles, it is affected by the particle's own field in a non-trivial way:
$$
\mathbf e = \mathbf e_{external} + \mathbf e_{due~to~particle}.
$$
People sometimes assume (as you did) they can simply put into the Poynting formula the macroscopic field $\mathbf E$ or total external microscopic field $\mathbf e_{external}$ and obtain the correct energy flux density. But that is not correct. The electric field due to extended charged particle itself is contributing to the total electric field acting on the particle; particle's own field acts and can do work on the very same particle. This energy exchange is not captured by Poynting's vector based on macroscopic field $\mathbf E,\mathbf B$, or based on external microscopic fields $\mathbf e_{external},\mathbf b_{external}$.
In your example, Poynting's vector vanishes at a certain point of space, while electric field does not. But that is possible only if there is no moving charged particle there. If there is some moving charged particle or group of moving charged particles there, these will affect the total microscopic field so that magnetic field does not vanish. Poynting's vector based on microscopic fields won't vanish there anymore.
When electric field accelerates a charged particle with finite charge density, the particle generates magnetic field component proportional to the velocity, and total magnetic field no longer vanishes where the particle is. This way, the particle's own magnetic field makes it possible for electric energy to be transferred into kinetic energy of the particle, in agreement with either Poynting's or Frenkel's expressions for EM energy and EM energy flux density.