I am familiar with the Faraday Effect, but I am not talking about that. Even though photons do not have any electric charge or magnetic field, they are still composed of oscillating electric and magnetic waves. I do not get why these electric and magnetic components do not align with an electric/magnetic field, and as the result, cause the photon's direction to align with the field. For an analogy of what I'm trying to say, imagine the photon's direction as an electromagnetic arrow that aligns itself with the (imaginary) lines of an electric/magnetic field.
I have looked at this question, and its first answer says that in order for a photon to be able to align, it must first loose information on its original polarization through dissipation (e.g., absorption). I do not understand how the polarization of light has anything to do with this, as direction of a photon and the polarization of a photon are 2 different things. Another answer for that question also talked about photons interacting with electric/magnetic fields through Delbruck scattering by the photon splitting into virtual electron-positron pairs. However, such interactions are very improbable and require high energy photons (e.g., gamma rays).
Forgive me if my question isn't worded correctly or my understanding is screwed up; I'm a bit of an amateur in the field of quantum mechanics. Any answers would be much appreciated.