This simple question requires a complex answer.
The bike comparison is misleading. A bike has one more degree of freedom (or let's say one more degree of control, the handlebar). You don't have it with an ice blade or with inline skates.
I really believe that for ice skating, and rollerblading too, there are two concurring actions that keep balance. One is a self-balancing mechanism where the skater is an "unbrained" passive system. The other one is the voluntary correction - a brain-controlled negative feedback loop - for off balance situations. This second mechanism has already been pointed out, but not the first one.
If you have some ice skating experience you'll agree that a fast spiral on a large curve is self balancing. Once correctly entered, the track develops by itself without any need for correction.
The reasons for such a good performance are manifold: (1) blades are manufactured by experience for having their best behaviour on ice. They have rocker, two edges, and a hollow profile. This builds a small amount of friction when the blade steers left or right. It is necessary to slowly dump any angular momentum on the vertical axis. Otherway it would be very difficult - or even impossible - to control it. (2) skater's height (and his/her center of mass height above the ground, about 100 cm) is several times bigger than the blade length and its effective - few centimetres - footprint on the ice (3) the normal friction of a blade, along its travelling direction, is very small but not completely negligible. In fact the horizontal deceleration is around 1 hundreth of the gravitational acceleration (9.8 m/s2) (4) Said that, the center of mass of a skater - during a passive glide - always stays behind the center point of contact of the blade with the ice. Say 1/100 * 100 cm = 1 cm. In other words the skater's body "lags" just a bit her/his foot trace. (5) This fact develops a torque, on any curved path, that keeps the skate steering just the right amount.