I know that velocity of a particle moving along a curve at a given point in time is tangential to a curve. I can also tell why intuitively. Also I am familiar with the experiment of spinning a ball tied to a rope and then leaving the rope.

But I was wondering if there is a more 'analytical/rigorous' proof for that in case of a general curve. Or is the statement supposed to be an axiom/self evident?

Sort of kinda. The curve exists in physical space. The velocity can be represented by a vector, and that vector will be three-dimensional, and so we can draw it in three-dimensional space, but technically speaking, the velocity vector space and the position vector space are different vector spaces (and to get even more technical, the latter is more of an affine space than a vector space). When we represent velocity by a vector in physical space, that vector has a magnitude that is proportional to the speed (but it can't be equal to the speed, since the speed has dimensions of distance/time, and the physical space has dimensions of distance). The direction of the velocity vector is equal to direction of the instantaneous difference in position. That is, if you take the displacement between the position at time $$t$$ and the position at time $$t+h$$, and take the limit as $$h$$ goes to zero, that will be pointing in the direction of the velocity vector.