How does a photon leave trace of its polarization state in a photon detector but not trace of which direction it came in? Some quantum erasure experiments involve polarization of photons. In one such experiment with a double slit, a horizontal polarizer is used in front of one slit, and a vertical polarizer is used for the second slit and the idea is that then horizontal or vertical polarization of the photon coming out of the double slit "marks" which path it took and so interference pattern disappears. Then the experimenter goes on to erase that which-path information in some way to restore interference. So it seems to me that a photon detector by itself does not work as an eraser and that a horizontally polarized photon entering a photon detector leaves something very different in the detector or the environment than a vertically polarized photon entering the same detector. How is it that a detector that usually cannot tell apart different incoming directions of photons can somehow tell apart different polarization?
 A: The key point here is the notion of distinguishability. If the two paths are made distinguishable, by associating different polarization to each path, then there is no way for the two amplitudes to interfere when they meet again because the "which-path" information is present. It doesn't matter whether or not the observer decide to actually look at the information. 
You're right, if one uses a detector such as a CCD camera, it won't tell you what was the polarization of an incoming photon, so you're not actually retrieving the "which-path" information. But still, since this information is available, you won't observe interference, unless you make the information disappear again, using a diagonal polarizer for instance. 
A: It is very important to keep in mind that the double slit interference is a quantum mechanical phenomenon. The ideal would be to solve the quantum mechanical problem and get the wavefunction , i.e. photon , two slit field pattern, screen , detector and impose the boundary conditions to get the system wave function and see the probability of the photon hitting the screen.
This has not been done as far as I know mathematically. What is being done is various experiments that lead to reasonable conclusions on what is happening when boundary conditions are changed by inserting different detectors and measuring more variables.
In this experiment, for example, Which-way detector unlocks some mystery of the double-slit experiment

First, the scientists used focused ion beam milling to make two nanoslits on a barrier. Then they modified one of the slits by covering it with a filter made of several layers of “low atomic number” material to create a which-way detector for the electrons passing through.
Although the electrons (which were shot one by one) could still pass through the filtered slit, the filter caused more of the electrons to undergo inelastic scattering rather than elastic scattering. As the physicists explained, an electron undergoing inelastic scattering is localized at the covered slit, and acts like a spherical wave after passing through the slit. In contrast, an electron passing through the unfiltered slit is more likely to undergo elastic scattering, and act like a cylindrical wave after passing through that slit. The spherical wave and cylindrical wave do not have any phase correlation, and so even if an electron passed through both slits, the two different waves that come out cannot create an interference pattern on the wall behind them.

shows the importance of changing boundary conditions. If the detector is set up for measuring polarization it is a different detector and a different state function of the system gives the probabilities of detection.
We arrived at the conclusion that the microlevel of nature is quantum mechanical  and quantum mechanics was discovered because of the odd probabilistic behavior of the results of  experiments on elementary particles , which photons and electrons are. One cannot use the deterministic intuition of classical mechanics to describe experiments on elementary particles.
I see that I have not answered the question of

How is it that a detector that usually cannot tell apart different incoming directions of photons can somehow tell apart different polarization?

Polarisation is not a simple state. Polarization of the beam is built up by photons, but the beam polarization is perpendicular to the direction of the wave while the photon polarization is plus or minus on its direction of motion.


Left and right handed circular polarization, and their associate angular momenta

Photon are always polarized along their direction of motion. There can be no vertical or horizontal polarization for a single photon. It will just be tagged as a photon that went through the classical filter of horizontal or vertical polarization.
It is a pictorial example of how the classical  wave, light, is built up by the quantum mechanical photons. Light is a confluence of zillions of photons.## Heading
In your description " horizontal polarizer is used in front of one slit, and a vertical polarizer is used for the second slit" the polarizers are light poarizers, which means a system of filters. To detect a point of arrival of the photon one needs a screen or a corresponding electronic surface. The two requirements cannot be met in one detector , to catch an individual photon and extrapolate its direction and also get its polarization. It would be a different experiment with two detectors and one photon at a time passing the slit, different quantum mechanical setup.
