You don't need a vector field on the sphere - you just need vectors. Vectors don't have any intrinsic location, just a direction and a magnitude.
The polarization of light is independent of the propagation direction of the light. Let's examine this with a simple experiment:
Consider an ideal plane-wave laser beam, beam 1, propagating in the z-direction and striking a screen some distance away. Another beam with exactly the same intensity and wavelength, beam 2, hits the screen at the same point, but from a different angle. Both beams are vertically polarized -- that is to say, their E-vectors point in the x-direction.
At the point where these beams hit the screen, you will observe interference fringes, which will go all the way down to zero intensity at their darkest points. This is because the x-polarized components of the electric field interfere with each other. Besides the x-components, there are no y or z-components to interfere.
Now consider the same situation again, but with both beams horizontally polarized. That is to say, the E-field of each beam points at right angles to the beam's propagation direction, and lies in the yz-plane.
However, the polarizations are not the same, even though they are both known as "p-polarized". (At least I hope so -- I can never remember which is s and which is p.) Beam 1's E-field points purely in the y-direction, but beam 2's E-field has both a y and a z-component, as I've drawn on the beam in the illustration.
At the point where these beams hit the screen, the y-components of the electric fields interfere, once again producing interference fringes. However, even though both beams' amplitudes are the same, the y-components are not equal, so the dark parts of the fringes don't go all the way down to zero intensity.
So you see, polarization vectors are polarization vectors, no matter which direction they're coming from. Equally valid, you could define your coordinate axes according to the propagation direction of beam 2, so that beam 1 came in at an angle instead, and still arrive at the same result. Specifically, this means that you don't need to worry about parallel transport.