This is really philosophy not physics, and if anyone objects to this being discussed here I'll happily delete this answer. Still, it's an interesting question that I bet many of us have discussed in those late night post drinking discussions that physics students are prone to.
Suppose I manage to write an AI program that emulates you (well enough to pass a Turing test) then this program would have the same sense of time passing that you do. However, unlike you, the program is running on a computer driven by a clock. Suppose at some point I stop the clock but preserve the internal state of the computer, then some time later I start the clock again. What would the AI experience? Well, it might notice that things in the outside world suddenly jumped, but it could have no sense of time having stopped because as far as the execution of the program is concerned its time is controlled by the computer clock not time in the outside world. Similarly I could run the computer clock slow or fast and although the AI may notice the outside world speeding up or slowing down, but it would still think time was flowing normally.
The point of the above is that the time for the AI is controlled by a clock register in the PC and is unrelated to external time. So, and this is the step some won't agree with, if I run the program in reverse the AI would still experience time flowing forward as normal.
Now if you think the above makes sense, and if you accept that you're basically a (somewhat squidgy) machine, the same must apply to you. In other words you can't tell whether time is flowing backwards, forwards, jerkily or whatever unless there is some aspect of the outside world you can use as a reference. If time flows at the same rate (whatever direction/rate that is) everywhere then there is no external reference.
The model of an AI on a computer can be pushed further. Suppose at some point I stop the computer clock then transfer the computer state to a different computer and start it again (note that the VMWare VMotion product can do exactly this), what would the AI experience? Again the answer would be nothing without some external reference. Suppose I take this to the extreme and at every tick of the computer clock I transfer the machine state to a new computer. From my previous argument the AI still couldn't notice anything, but now what do we mean by time when none of the computer clocks are actually ticking?
A computer has a finite number of possible states, so suppose I gathered together that many computers and set one to each possible state. Haven't I ended up with the same as I got by moving the AI program at each clock tick? If so does that mean my AI program is running somewhere in my collection of computers even though there is no concept of time? Or maybe every possibly AI is running somewhere in my collection of computers.
Turn this analogy into a phase space, and maybe the universe doesn't need time to flow at all, despite what we AIs might think :-)