Now if the interaction things with light can be understood in terms of the interactions with the fields, what does the size of the Poynting vector actually mean in practice? Are there any processes that depend on this rather than directly on the amplitude of the electric or magnetic field?
The Poynting vector, a function of total electric and total magnetic field, is interpreted as density of flux of EM energy in macroscopic theory (watts per unit area), due to energy interpretation of the Poynting theorem. The Poynting theorem is the following mathematical relation based solely on Maxwell's equations:
$$
\mathbf j \cdot \mathbf E = -\nabla \cdot \mathbf S - \partial_t w
$$
where $w$ is the well-known Poynting energy density expression in terms of $E^2,B^2$.
The energy interpretation of this equation adopts additional assumption: that $\mathbf j \cdot \mathbf E$ is an accurate expression of total work per unit volume per unit time done by EM field on matter, so the terms on the right-hand side of this equation can be interpreted as convergence of EM energy flux density vector and rate of decrease of energy density at the same point of space. Under this assumption, Poynting's theorem becomes a law of local conservation of energy of matter and EM field. One can say work done on matter particle in a region of space by EM forces equals EM energy that came in through the surface of the region from the outside, plus decrease of EM energy in the region.
Such interpretation of the Poynting theorem equation is either incorrect, or correct but more complicated in the microscopic EM theory, where total fields are $\mathbf e, \mathbf b$, for the following reasons:
if the charged particle that is experiencing EM force is a point, the term $\mathbf j \cdot \mathbf e$ is too singular and its meaning cannot be consistently defined; total electric field diverges at the particle. So the assumption about $\mathbf j\cdot \mathbf e$ is invalid and Poynting's theorem while true, is useless. We can derive another similar theorem that can be interpreted as local conservation of energy, if we base it on some regular expression for EM work. There is the Frenkel theory which defines its own energy density and energy flux density expressions based on the idea that point particles experience only external field, and their own field does not affect them. This leads to expressions for EM energy and EM energy flux density that are different from the Poynting ones.
if the charged particle is extended in space with a finite charge density everywhere, the term $\mathbf j \cdot \mathbf e$ is fine and formula for the Poynting vector is fine and can be interpreted in terms of local conservation of energy, but $\mathbf e$ is usually unknown to us, because near and inside the particles, it is affected by the particle's own field in a non-trivial way:
$$
\mathbf e = \mathbf e_{external} + \mathbf e_{due~to~particle}.
$$
People sometimes assume (as you did) they can simply put into the Poynting formula the macroscopic field $\mathbf E$ or total external microscopic field $\mathbf e_{external}$ and obtain the correct microscopic energy flux density. But that is not correct. The electric field due to extended charged particle itself is contributing to the total electric field acting on the particle; particle's own field acts and can do work on the very same particle. This energy exchange is not captured by Poynting's vector based on macroscopic field $\mathbf E,\mathbf B$, or based on external microscopic fields $\mathbf e_{external},\mathbf b_{external}$.
In your example, macroscopic Poynting's vector vanishes at a certain point of space, while macroscopic electric field does not. But that is possible in microscopic theory too only if there is no moving charged particle there. If there is some moving charged particle or group of moving charged particles there, these will affect the total microscopic field so that magnetic field does not vanish. Poynting's vector based on microscopic fields won't vanish there anymore and EM energy flux into or from the charged particle can be non-zero.
When electric field accelerates a charged particle, the particle generates magnetic field component proportional to the velocity everywhere near and inside the particle, and total magnetic field no longer vanishes where the particle is. This way, the particle's own magnetic field due to its motion makes it possible for vector of EM energy flux to not vanish, and for electric energy to be transformed into kinetic energy of the particle. This is the case irrespective of whether the charged particle is a point or extended charge distribution with finite density.