How velocity affects different orbits?

Assume a sun with mass $$M$$ and a planet with mass $$m$$. Assume at $$t_0$$ the planet is $$r_0$$ (distance) away from sun and has an initial velocity of $$v_0$$. Also, let’s assume the angle between the $$r_0$$ and $$v_0$$ is $$\theta_0$$ but I suppose if $$\theta_0$$ is $$90$$ degrees we can cover all the cases.

Are there any formulas and relations between $$M, m, r_0, v_0$$ that show wether the planet will make a circular, elliptical, parabolic or hyperbolic orbit ? (Or spiral orbit slowly moving towards the sun making smaller circles ) ?

Yes there is. Total energy is, in polar coordinates in the plane where the movement occurs:

$$E=\frac{1}{2}\,mv^2-G\frac{Mm}{r}=\frac{1}{2}\,m(\dot{r}^2+(r\dot{\theta})^2)-G\frac{Mm}{r}$$

A short rewrite using $$C=\lVert\vec{L}\rVert/m$$, the constant from Kepler's second law, yields:

$$E=\frac{1}{2}\,m\dot{r}^2+\underbrace{\frac{mC^2}{2r^2}-G\frac{Mm}{r}}_{=V_\text{eff}(r)}$$

The second and third terms form what's called the effective potential energy $$V_\text{eff}$$. Notice you necessarily have $$V_\text{eff}(r)\leqslant E$$, which can easily be interpreted on a graph:

In this example, $$E<0$$. Since $$V_\text{eff}$$ must remain below $$E$$, the only part of the plot that is physically accessible is $$r\in[r_1,r_2]$$. In other words, $$r$$ cannot go to infinity, it's a bound state. This is the elliptic case.

On the other hand, if $$E>0$$, $$r$$ cannot go to zero, but it can go to infinity. The movement isn't bounded, it's the hyperbolic case.

The special case $$E=0$$ is the parabolic one ($$r$$ can go to infinity, but then kinetic energy is zero).

The special case where $$E$$ is exactly at the bottom of the potential well is the circular movement (only one allowed value for $$r$$, here $$r_0$$).

In other words, the sign of $$E$$ will tell you exactly in which case you are. $$E$$ can be rewritten as:

$$E=\frac{1}{2}\,mv_0^2-G\frac{Mm}{r_0} \quad \begin{cases} >0 & \text{hyperbolic}\\ =0 & \text{parabolic}\\ <0 & \text{elliptic} \end{cases}$$

If a mass (like a planet) is orbiting a much larger mass (like the sun) under the action of gravity, it has kinetic energy (1/2)m$$v^2$$ and potential energy -GMm/r (negative because the zero is chosen to be at infinite r). If the total energy is negative, the orbit will be elliptical; zero, parabolic; and positive, hyperbolic. The path will be a spiral only if there is a mechanism which is causing a loss of energy. (The earth has been orbiting the sun for a very long time.)
For the planetary motion the total energy $$E=\frac{1}{2}mv^2-\frac{GMm}{r} \tag{1}$$ and the angular momentum $$L=mrv\sin\theta \tag{2}$$ will remain constant for all times $$t$$. Hence you can calculate them at time $$t_0$$ (with $$r_0$$, $$v_0$$, $$\theta_0$$).
From Newtonian mechanic the planetary orbit is known to be a conic section (never a spiral). Using the values for $$E$$ and $$L$$ from (1) and (2) you can calculate the eccentricity $$\epsilon$$ of the orbit. $$\epsilon=\sqrt{1+\frac{2EL^2}{G^2M^2m^3}}$$
From $$\epsilon$$ you can read off the shape of the orbit:
• $$\epsilon = 0$$ : circular orbit
• $$0 < \epsilon < 1$$ : elliptic orbit
• $$\epsilon = 1$$ : parabolic orbit
• $$\epsilon > 1$$ : hyperbolic orbit