There are several ways you could do this. Below I will outline a couple of possible approaches, but it may be that a simpler / more elegant solution exists. Here are my 2cents nonetheless.
Firstly, it would be a lot easier if you first consider the motion of a zero-radius ball and worry about making it bigger and appearing as if it's 'rolling' later on. Here's how I think you could approach the motion of a zero-radius ball, in the bowl.
You would need to program time-evolution equations of motion for each of the three dimensions individually. You could do this in your choice of coordinate system. For the geometry of the apperatus you are trying to model, I think cylindrical coordinates $(r,\theta,z)$ would be easier than cartesian coordinates $(x,y,z)$. In which case, you would need just a circular motion equation (relating $r$ and $\theta$ with time,) with the addition of a resistive term to simulate the energy loss and therefore reduction of the 'orbital' radius. The fall in $z$ could then be determined from the (varying) $r$-value with respect to the geometry of your bowl. I.e. A smaller radial position corresponds to a position nearer the centre of the bowl and therefore a lower height.
In cartesian coordinates you would probably need a damped harmonic oscillator equation for each of the $x$- and $y$-coordinates. The change in $z$ could then be determined in the some way as above.
Alternatively, as opposed to having an educated guess at the equations of motion as described above, you could derive them 'from scratch' by considering the potential and kinetic energy exchange (again for each coordinate individually.) If you did this in a vector formalism (using the coordinate system of your choice) you would just have to solve the equations for the velocity $\vec{v}(t)$, then integrate to find the position $\vec{x}(t)$. Again though, you would also need an additional term to account for the loss of energy due to friction. (Otherwise the ball's speed will not decay and therefore neither will its 'orbit'.)
Once you've got your dynamics set up for the motion of a zero-radius ball (a point I guess,) you can then program a modification to adjust how the ball is displayed so that it looks like an object of the desired size. One option would be (for a ball of radius r) to display the point as moving around a constant distance r higher than the surface of the bowl, then program a few lines to depict the ball as an object extended by the distance r around the point. How complex this gets depends on how realistic you want it to appear.
Edit: In your question you refer to "rotation angle" and "position" - if I understand you correctly, you mean the initial position and velocity of the ball. These values would be put in by the programmer or program user as initial conditions, after which the dynamics equations would evolve the system.