Why somebody riding a bicycle keeps balance and doesn't fall left or right, while is prone to falling when the bicycle stops. One force (to front) shouldn't affect in any way balance left/right. I hope that explanation using mathematics and physical concepts should be understandable intuitively. I mean: description is not explanation. If the model of bicycle is too complicated (too many factors to consider), let's talk about a rolling coin. When it is pushed it rolles for a certain amount of time, if it's put - if falls immediately. Contrary to bicycle, no corrections of the rider can be taken into the consideration thank you Zbigniew
It turns out that people have been crazy enough to build bicycles with smaller "spare wheels" alongside their normal wheels that can be "ramped up" to speeds that have the same angular momentum in the opposite direction. When you do this with a unicycle it becomes difficult to balance even at speed, which means that the primary mechanism is gyroscopic. But when you do this with a bicycle, it retains its ability to stay stable.
There has therefore been a lot of research into why bicycles are as stable as they are. It's not merely the feedback loop created by the rider; see here for a video from a research group which just occasionally pushes bicycles around for experimentation. They'll release a bicycle upright without a rider and it will "coast" for a surprising distance before falling.
Basically, the combination of having to "drag around" the rear wheel and the front wheel having a pivot off-center from the wheel-axis itself appears to cause the bicycle to naturally "turn into" any motion that would cause it to fall down, converting those microscopic "falling" actions into a macroscopic bicycle "turning" motion. At higher speeds these "corrections" become self-destabilizing rather than self-stabilizing.
Wikipedia has a lot more information on the general matter.