I know that the balancing toys have their center of mass under the axis on which they are balancing. That's why they stay still. But when we give a little tap on it, it re-balances itself. But how does it happen? Does the center of mass acts like a pendulum?
You are correct that the bird is a carefully disguised pendulum.
Notice the tips of the wings are slightly forward of the beak and slightly lower. The wing tips have heavy weights in them. This allows them to balance out the tail of the bird, and places the center of gravity below the beak. A CG below the pivot point makes it stable to small perturbations.
It is the same concept as with the tumbling self righting toys: what makes it stable is that the center of gravity is located at a point that will move up when the object starts to tip over. So gravity will make the center of gravity fall down again, which makes it move to the upright position again. In this case the wings that are pointing down place the center of gravity a bit below the beak, which from this position will be at the lowest possible point.