I suspect Luboš's answer may be a bit complex for you so I'll attempt a simpler explanation (if I'm wrong just ignore this answer).
When you say "I think its because the velocity to outward" you're getting close. To show what's going on I've zoomed in on the diagram you posted in your question.
Let's start at the moment where the velocity of the object is $u$. The thing you need to remember is that velocity is a vector so it has both magnitude and direction. Acceleration may be a change in the magnitude or a change in direction or both. In the case of circular motion it's the direction of the velocity that is changing not it's magnitude.
To see this look at the vector $v$, which is the velocity of the particle after some short time. So the velocity has changed from $u$ to $v$. To get the size of the change we need to find what vector has to be added to $u$ to give $v$. At the top of the diagram I've put the two vectors $u$ and $v$ with their starting point together, so you can see that the vector needed to change $u$ to $v$ is the small vector $a$, and you can see it points downwards. If I move $a$ to the starting point of $u$ on the circle you can see that $a$ points towards the centre of the circle.
The vector $a$ is of course the acceleration (well it's the acceleration times the short time interval between $u$ and $v$) and that's why the acceleration points towards the centre of the circle.
If you didn't have any acceleration the object would move in a straight line so it would just move away at a tangent to the circle. The effect of the acceleration $a$ is to bend the trajectory of the object so it stays on the circle.
Finally, you ask "what keeps the direction of acceleration always point inward". You've mixed up cause and effect. It isn't that there's some magical property of circular motion that causes the acceleration to be central. It's that if in some system the acceleration is always central the object will move in a circle (or strictly speaking in a conic section, but let's not complicate matters).
For example suppose I tie a stone to a string and start whirling it round my head. The force causing the acceleration is provided by the string, and obviously the string always runs from the stone to the centre because my hand is at the centre. It's because the force caused by the string is always central that the stone moves in a circle.