You have to consider where the infalling matter has come from. A neutron star will normally form at the centre of a planetary system, and the whole system has a non-zero angular momentum not just the star. Since the infalling matter is coming from the planetary system it will be rotating in the same direction as the star.
In principle an interstellar dust cloud, or indeed anything from outside the planetary system, could hit the neutron star with a retrograde motion. However only in exceptional circumstances will this be the dominant source of infalling matter.
A neutron star does have a tendancy to rotate matter in a prograde direction due to frame dragging. However this effect is only strong very near the event horizon of a rotating black hole. It won't have much effect on infalling matter until the matter gets very close.