The scattered wave function fills all of space. It doesn't pick out a preferred direction for the scattering. The electric field mode, and hence any excitation of the field (a quantum) fills all of space. The destruction of the quantum can occur anywhere the wave function is not zero and there is some means to absorb the energy and momentum of the excitation: some kind of detector. That can occur just about anywhere in space, but the spot at which it happens is entirely unpredictable. The field loses one quantum of excitation at a particular place, and transfers a particular amount of energy and momentum to the detector. It behaves for all the world as if a particle collided with the detector at that point, and then vanished.
So each excitation is detected at some particular, unpredictable place. If you repeat this experiment many times, you find that detection events can occur almost everywhere.
In my opinion the picture of a photon as a particle is seriously problematic. You get questions like yours whose answer goes against intuition, and you have to accept the fact that the particle can simply vanish at the detection event. In our intuitive notion of a particle, it doesn't simply disappear. With care, the picture is useful, but safe use of the metaphor requires some understanding of the fuller picture so that the limits of the particle picture are recognized.
Pedantic note: should you be able to launch a single "photon" toward a silver atom, the scattering would not be isotropic. The scattering distribution depends on the polarization. (That's why I haven't used the word isotropic.) But that point has no bearing on the question.