There are satellites that constantly observe the sun. Among them are the STEREO pair of satellites. These satellites don't orbit Earth, but orbit the Sun–one a little faster than Earth, one a little slower.
At first, I thought the Earth-centered orbit you suggest wouldn't work. I was thinking that if a satellite starts out in a polar orbit, the plane of the orbit would not rotate as the Earth orbited the Sun, leaving the orbit parallel to the sun a quarter year later and behind the Earth at times. However, it is possible. From Wikipedia on polar orbits:
To retain the Sun-synchronous orbit as Earth revolves around the Sun during the year, the orbit of the satellite must precess at the same rate, which is not possible if the satellite were to pass directly over the pole. Because of Earth's equatorial bulge, an orbit inclined at a slight angle is subject to a torque, which causes precession. An angle of about 8° from the pole produces the desired precession in a 100-minute orbit.
Another source cited by the wiki article.
These orbits are generally used for Earth-observing satellites that track Earth conditions (like atmospheric temperatures) at the same times of day.
Also, from a comment by cmaster - reinstate monica:
Those sun-synchronous orbits are also used for satellites that don't care about day and night on earth, but need an unceasing supply of energy. Examples are satellites that use radar to image the earth's surface. Radar is an active technology that needs lots of power to work over 800-km distances, and that power is supplied by the sun. The sun-synchronous orbits puts the satellite into eternal sunlight, allowing it to work 24/7 with a minimal battery and solar cell array size.