Mie scattering and Raleigh scattering are classical effects. It should not be necessary to talk about photons in any answer.
The directionality of scattering in both cases is determined by the polarisation of the incoming electric field vectors.
In both cases the light can either be polarised in a plane perpendicular to the direction of the incoming transverse electromagnetic wave. This defines two directions in space - the direction of the electric field vector and the wave vector of the electromagnetic wave.
For unpolarised light, the electric field vector is still in the plane perpendicular to the wave vector, so there is still one defined direction associated with the incoming radiation, which is perpendicular to the plane of the electric field.
For Rayleigh scattering, the scattering of linearly polarised light is directional in the sense that it has the classical dipole radiation pattern with no radiation emitted along the axis of oscillation, which is the polarisation direction of the incoming wave. For unpolarised light there is no preferred direction of scattering and the scattering particle doesn't care from which direction the wave is coming.
For Mie scattering there is directionality irrespective of whether the incoming light is polarised or not. The directionality is imposed by the boundary conditions of the problem in the same way that Snell's law for transmitted light through an interface is defined by the angle between the normal to the surface (of a particle in this case) and the incident electromagnetic wave direction. The direction of the incident electromagnetic wave is in turn perpendicular to the plane of the electric field.
At a deeper level, I'm sure it is true that the direction of the momentum vector of the consituent photons, which is coincident with the electromagnetic wave direction and with the Poynting vector of the electromagnetic field sets a fundamental axis and direction for the problem, but the concepts of the photon or indeed the Poynting vector are not needed to predict the scattering properties.