Nowadays, rockets use a Gimbaled Thrust System. The rocket nozzles are gimbaled (An appliance that allows an object such as a ship's compass, to remain horizontal even as its support tips) so they can vector the thrust to direct the rocket. In a gimbaled thrust system, the exhaust nozzle of the rocket can be swivelled from side to side. As the nozzle is moved, the direction of the thrust is changed relative to the center of gravity of the rocket.
Early rockets had Vernier Thrusters which uses small rocket engines on either sides, to control the attitude (vs altitude) of a rocket. Nowadays, they are common in most satellites.
In this Image, The middle rocket shows the normal flight configuration in which the direction of thrust is along the center line of the rocket and through the center of gravity of the rocket. On the left one, the nozzle has been deflected to the left and the thrust line is now inclined to the center line at a gimbal angle $a$. As the thrust no longer passes through the center of gravity, a torque is generated about the center of gravity and the nose of the rocket turns to the left. If the nozzle is gimbaled back along the center line, the rocket will move to the left. On the right one, the nozzle has been deflected to the right and the nose is moved to the right.
Wikipedia says,
In spacecraft propulsion, rocket engines are generally mounted on a pair of gimbals to allow a single engine to vector thrust about both the pitch and yaw axes; or sometimes just one axis is provided per engine. To control roll, twin engines with differential pitch or yaw control signals are used to provide torque about the vehicle's roll axis.
The right & left gimbaling is necessary to direct the rocket to its original path, thereby maintaining its stability... This link gives a good explanation regarding the stability of rockets. This essay is also good, but it's somewhat big...