Stated differently, why does EM radiation not "ripple inwards" and collect at some point? These are perfectly possible, by time-reversal.
Well, in most cases observed or measured, EM radiation does not ripple outwards from a point either; usually, the radiation is connected to a body with non-zero spatial dimensions. In fact, radiation of point charges may contain advanced field component but macroscopically look like the retarded solution - see the Feynman-Wheeler theory.
I suppose you are interested in the following question:
since there are well known processes in which EM waves are created in and propagate outwards from physical bodies out to distant space (such as radiating antenna), why are there not also inverse processes, where the EM waves propagate from the distant space towards some physical bodies and collapse on them?
We know that Maxwell's equations in some simple scenarios (such as continuous motion of a point charge) have solutions like that (called advanced fields as opposed to retarded fields), so why aren't we observing such collapsing spherical waves at least somewhere?
Short answer: the reason can be sought either in:
- the state of the EM field in the past (a special initial condition that does not lead to macroscopic advanced waves); past is not a part of physical laws, it must be assumed as a separate assumption;
or
- it can be sought in a separate physical law that restricts the fields of charged particles to be the retarded solutions of field equations with their individual source terms.
Long attempt at an answer:
If we knew the state of the field everywhere at some point of time and if we knew the subsequent motion of the charges, we could predict the field into the future. If we fix the motion of the charges, the field would be determined by the initial condition.
The initial condition could be, in principle, such that field in vicinity of charged bodies would evolve like an ingoing wave, coming from far away and collapsing on those bodies.
Another initial condition could be such that the opposite would happen; the field would evolve like an outgoing wave, getting away from the bodies to infinity.
There are infinity of other initial conditions which are different from the above; in general, the difference between them is a "free field" - a solution of Maxwell's equations without any sources.
The appropriateness of initial condition in macroscopic theory is to be judged based on experience; there is no physical law that would require one or the other.
Absence of ingoing EM waves in our experience means that certain class of initial conditions for fields is to be avoided or even rejected in macroscopic EM theory.
But this does not easily translate into microscopic theory. Let us assume that total EM field is composed of elementary EM fields of very big number of charged particles.
It is possible to have macroscopically retarded fields that are made of microscopic fields which are not purely retarded but contain advanced field. And it is possible to simulate macroscopic advanced field with a special arrangement of microscopic fields that are purely retarded.
An interesting but not the only possible version of EM theory is that the microscopic fields are completely symmetrical combination of retarded and advanced waves (Tetrode, Frenkel or Feynman-Wheeler models and their variations), but when these are used to explain our experience with macroscopic bodies, things are not so simple anymore: if the elementary fields are symmetrical combination of retarded and advanced waves, how come we do not see such symmetrical EM wave around an antenna? (People actually proposed an experiment and verified that the EM field is not such symmetrical one. Experience suggests that EM field near antenna is well given by the retarded solution.)
In the Feynman-Wheeler model, they came up with an interesting idea: they introduced a boundary condition into the theory (so-called absorber condition) which formally allowed them to arrive at realistic description of EM field where the observable field seems to be almost retarded and where the correction supposedly explains radiation reaction. I find the absorber condition very formal and unnatural, and their explanation of radiation reaction (which relies on that condition) as ill motivated and unnecessary for point particles. Still, the idea of symmetrical half-retarded, half-advanced field has some interesting implications, for example, systems of opposite charges are much more stable, because now the waves go both in and out and there is no intense radiation of EM energy to surrounding space from such systems as would be expected based on Larmor's formula (which is not valid here). It is possible that the symmetrical microscopical fields with the right probabilistic assumptions can be consistent with our macroscopic experience with retarded fields, even if we abandon the somehow unnatural absorber condition.
The most simple and natural stance currently is that the elementary fields are retarded and the advanced solutions are unphysical - the second variant from the short answer. This simple choice is quite intuitive - elementary waves only ever propagate outwards from the particles. It explains why macroscopic waves are retarded without further assumptions such as the Feynman-Wheeler absorber condition. True, a collapsing approximately spherical wave could be created if lots of particles danced in a special way, but such correlated motion across great distances is quite improbable, so this poses no challenge for the model.