Mass particle trajectory on a sphere So, I am trying to simulate mass particle motion on the outer surface of sphere using cartesian coordinates. Let's conclude just a gravity and frictionless movement.
Sphere $x^2 + y^2 + z^2 = 1$, praticle of mass $m$ starts somewhere near the top. Gravitational force would be just $\textbf{g} = -g\textbf{e}_z$.
Total normal force $\textbf{N}$ would depend on our position, and, using inclation angle $\theta$ from spherical coordinates, can be written as 
$$\textbf{N}\cos{\theta} = mg \cos{\theta} \frac{\textbf N}{|N|}.$$
The components of that vector in cartesian coordinates:
$$N_x = mg \cos{\theta} R \sin{\theta}\cos{\varphi},$$
$$N_y = mg \cos{\theta} R \sin{\theta}\sin{\varphi},$$
$$N_z = mg \cos{\theta} R \cos{\theta},$$
where 
$$\theta  = \arctan{\frac{\sqrt{x^2+y^2}}{z}}, \ \ \varphi = \arctan{\frac{y}{x}}, \ \ R = const.$$
So, the euqations of motion will look like:
$$\ddot{x} = gR \cos{\theta}\sin{\theta}\cos{\varphi},$$
$$\ddot{y} = gR \cos{\theta}\sin{\theta}\sin{\varphi},$$
$$\ddot{z} = gR \cos{\theta}\sin{\theta}\cos{\varphi} - g.$$
Solving these, starting near the top on a sphere radius 1 I am getting some wrong trajectory (green on a picture, grey - rounded surface). So maybe something wrong with equations?

 A: Yes, your equations aren't quite right.  The main issue is that you're assuming a certain form for the normal force that isn't correct.  What follows should illuminate why this is so in some detail.
When using forces and Newton's Laws to solve this problem, it is overwhelmingly helpful to work in spherical coordinates, not just for locating the position of the mass, but also for writing vector components.  In particular, it's advantageous to express all vectors in spherical coordinate unit vectors.
There are two forces acting on the mass: normal force which points in the radial direction, and the gravitational force which points in the negative $z$ direction, and this gives the following net force on the particle
\begin{align}
  \mathbf F = N\hat{\mathbf r} -mg\,\hat{\mathbf z}.
\end{align}
We would like to follow our advice above, and write this in terms of spherical coordinate unit vectors $\hat{\mathbf r}, \hat{\boldsymbol \theta}, \hat{\boldsymbol\phi}$.  If you look in the back of Griffiths' Electrodynamics, or better yet work it out for yourself, you will find that $\hat{\mathbf z} = \cos\theta\hat{\mathbf r} - \sin\theta\hat{\boldsymbol\theta}$, so we can write the net force entirely in terms of spherical coordinates and unit vectors as follows:
\begin{align}
  \mathbf F = (N-mg\cos\theta)\hat{\mathbf r} + mg\sin\theta\hat{\boldsymbol\theta}.
\end{align}
Notice that we do not yet know what $N$, the normal force, is.  The normal force is complicated in this problem because it will turn out to depend on the velocity of the mass, a feature that you can immediately tell is missing from your original work.
Now, in order to write down Newton's Laws, we need the acceleration in spherical coordinates which is an awful mess in general.  But notice that since the mass is constrained to the surface of the sphere, and since there are no forces in the tangential direction, provided the mass starts at the top of the sphere, we will have
\begin{align}
  \dot r = 0, \qquad \dot \phi = 0.
\end{align}
The first of these equations is a constraint we impose by virtue of the particle remaining on the sphere at all times ($r(t) = R$), the second can be argued purely mathematically from Newton's Laws, but it should be clear from the physical argument above, so we omit that step.  The result is a drastic simplification in the expression for acceleration in spherical coordinates:
\begin{align}
    \mathbf a = -R\dot\theta^2\hat{\mathbf r} +R\ddot\theta \hat{\boldsymbol\theta}.
\end{align}
These terms should actually look quite familiar.  The first is just the centripetal acceleration, and the second is the tangential acceleration in the $\theta$ direction.  Compare these to the standard expressions $a_r = -R\omega^2$ and $a_\theta = R\alpha$.  If we now use Newton's Second Law, then we find
\begin{align}
    N - mg\cos\theta = -mR\dot\theta^2, \qquad g\sin\theta = R\ddot\theta.
\end{align}
This is a system of two ODEs in two unknown functions $N=N(t)$ and $\theta = \theta(t)$, but the second is an ODE entirely for $\theta$.  Once you have solved this to find the motion of the mass, you can plug the solution back into the first equation to determine the normal force as a function of time if you wish.
Note that this is consistent will the Lagrangian approach of user yohBS.  Simply plug $\dot\phi = 0$ into his $\theta$ Euler-Lagrange equation an note that it agrees with the equation above.
A: A simpler method would be to use spherical coordinates. To clear the notation, I define: 
$$\begin{align}
z&=R \cos  \theta  (t)\\
y&=R \sin  \theta  (t) \sin  \phi(t)\\
x&=R \sin  \theta  (t) \cos  \phi(t)
\end{align}$$
The kinetic energy is then 
$$T=\frac{m}{2}\left(\dot x^2+\dot y^2+\dot z^2\right)
=\frac{mR^2}{2}\left(\dot \theta^2+\dot\phi^2\sin^2\theta\right)$$
while the potential energy is
$$U=mgz=mgR\cos\theta$$
The equations of motion can be calculated from the energy can be calculated using the Lagrangian formalism. The Lagrangian is
$$L=T-U=\frac{mR^2}{2}\left(\dot \theta^2+\dot\phi^2\sin^2\theta\right)-mgR\cos\theta$$
The equation for $\theta$:
$$\frac{d}{dt}\left(\frac{\partial L}{\partial\dot\theta}\right)=
mR^2\ddot\theta=mgR\sin\theta+mR^2\dot\phi^2\cos\theta\sin\theta=\frac{dL}{d\theta}$$
$$\ddot \theta=\sin  \theta \left(\frac{g}{R}+ \dot\phi^2 \cos \theta  \right)$$
Similarly, the equation for $\phi$:
$$\ddot\phi=-2\dot\theta\dot\phi\cot\theta$$

Addendum:
I agree with the comment that this does not directly answer your question. The mistake you made is writing the wrong expression for the normal force. The expression you wrote is valid only in the static case, that is, only when the acceleration vanishes (in all directions). In order to obtain the correct expression for $N$, you should draw a force diagram and take into account the centripetal acceleration in two directions. This is a mess, and a simpler way to do it is to use the method I've described above.
