If you put a satellite at Geostat altitude travelling the wrong way, will it remain between the Earth and Sun?
Yes, a satellite can "park" between the Earth and the sun. I.e., there does exist a point at which it's possible for a satellite to remain between the Earth and the sun without using a prohibitive amount of fuel to do so. However, that point isn't at the geostationary altitude above the Earth, but rather at a point considerably farther away from the Earth than that, called the $L_1$ Lagrange point.
An object that remains closer to the sun than the Earth is will normally have a shorter orbital period around the sun than the Earth has. But at the $L_1$ Lagrange point, gravity from the Earth counterbalances the sun's gravity, lengthening the orbital period such that it matches the Earth's orbital period.
The $L_1$ Lagrange point is an unstable equilibrium point, meaning that if the satellite is slightly off from the $L_1$ Lagrange point, it will drift away from that point unless the satellite consumes a little bit of fuel to correct its orbit. There exist two other points in the Earth/sun orbital plane called the $L_4$ and $L_5$ Lagrange points, which are stable equilibrium points, i.e., if a satellite is a little bit off from either of those two points, it will remain just a little bit off instead of drifting away, without needing to correct its orbit. $L_4$ and $L_5$ aren't in between the Earth and the sun, however, but rather are at points that form an equilateral triangle with the Earth and the sun.