Firstly, the centrifugal force is an pseudo force and only appears to exist if you are sitting on the stone and flying along with it. This is because you are in an accelerating (non-inertial) frame of reference. Another common pseudoforce you experience every day is the "force" that throws you into the seat of your car when it accelerates. For further explanations of the centrifugal "force" see this wiki article.
To treat your problem properly the observer (you) needs to be stood still watching the pendulum. The only forces acting on the stone at the instant of your diagram are 1) gravity and 2) the string pulling on the stone. These two forces exist whether or not the stone is rotating and are the only two forces acting on the stone.
Force 1) gravity, always points down towards the earth.
Force 2) the tension points along the string, away from the stone.
When the stone is just hanging at rest (no rotation) it is obvious that these two forces cancel out and the stone does not move. So what is different when the stone is rotating?
If the stone is made to rotate, it now has momentum in some direction. At any instant, the stone is trying to fly off in a straight line (think what happens if the string were to snap) but it can't because the string is holding it back. Instead, the force of the string pulling on the stone changes the stone's direction by accelerating it towards the stick. This happens constantly - the stone is trying to fly off in a straight line while the string pulls it towards the stick.
Now the faster the stone is going, the harder the string must pull to make it travel in a circle. So when the stone is rotating, the tension in the string must apply this force called the centripetal force . The centripetal force acts radially, from edge of the rotation circle to the centre. Here's a picture:
Finally, the tension in the string needs two components. 1) It must counteract the force of gravity on the stone, 2) it must apply the centripetal force to stop the stone from flying away. 1) is the vertical component, 2) is the horizontal component so the combined force 1)+2) will be at an angle (and hence the string will be at an angle relative to the stick).