You have three different questions here:
is 8th Avenue and 14th Street at 3pm is the same place as as 8th and 14th at 4pm?
why can't we move backwards in time
is the answer to (1) connected to the answer to (2)
The answer to (1) is unambiguously NO. If you remember back to Pythagoras' theorem as learnt by generations of schoolchildren, the distance $s$ between two points $(x_1, y_1)$ and $(x_2, y_2)$ is given by:
$$ s^2 = \Delta x^2 + \Delta y^2 $$
where I used $\Delta x$ as shorthand for $x_2 - x_1$ and likewise for $y$. For spacetime we define a distance in a similar way, but the equation is now:
$$ s^2 = -c^2\Delta t^2 + \Delta x^2 + \Delta y^2 + \Delta z^2 $$
All the weird stuff, like time dilation and length contraction, can be derived from this equation for the distance so it's absolutely fundamental to relativity (general as well as special). Two points are only the same if $s = 0$, and it clearly isn't for the same street corner at different times. This sounds a bit like a mathematician making an artificial distinction, but I must emphasise that all of SR depends on this distinction and we know SR works because we test it every day in particle accelerators.
On to question (2) and the answer is unambiguously NO-ONE KNOWS.
And finally question (3) and the answer is that there is no obvious connection between (1) and (2). The equations of special relativity are time symmetric, so they do not dictate that you cannot move backwards in time.
Feel free to Google around for the answer to your question (2). There are lots of interesting articles out there, but in the final analysis the answer is the two word answer I gave above.
Response to comment
You say in your comment:
I was rather more interested in why we believe we can move backward in space. I know mathematically we can but in reality going forward two paces and then back two does not get you back to the same place
Your net displacement is the vector $(\Delta t, \Delta x, \Delta y, \Delta z)$, so the direction you have moved in is the direction of this vector. In your example of two steps forward and two back the vector will be $(\tau, 0, 0, 0)$, where $\tau$ is the time you took to make the four steps. So if you define the word place to refer only to the spatial coordinates (which is after all what most of us mean by place) then you are back in the same place but not at the same spacetime point.
If we assume your steps carried you along the $x$ axis then the displacement for the first two steps was $(\tau/2, 2X, 0, 0)$ and the displacement for the return two steps was $(\tau/2, -2X, 0, 0)$, where $X$ is your step length. Note that the $x$ displacement can be positive or negative. When we say we cannot move backwards in time we are saying that the $t$ displacement can never be negative; it can only be positive.
In fact we can make a stronger statement than that. An observer who is moving relative to you will disagree about your displacement. Suppose you measure your displacement as $(t, x, y, z)$ and the moving observer measures it as $(t', x', y', z')$ then $t \ne t'$, $x \ne x'$ and so on. You and the moving observer will disagree about your displacement (though you will both calculate the same value for $s$ as described above). However, both you and all observers moving (slower than light) with respect to you will agree that the time displacement cannot be less than zero.