A mass in a rotating tube A mass rotates on a horizontal surface inside a frictionless hollow tube with a angular velocity omega. The only force acting on it is a force $N$ with which the tube pushes the mass. 
It is expected that the mass would move away from the center of rotation due to centrifugal force, which is a fictitious force in the frame of reference attached to the tube. But in the frame of reference in which the tube rotates, there are no forces in radial direction. So what actually happens, and why?
 A: Updated answer
The diagram below illustrates what happens to a mass at a number of intervals of time if the mass is subjected only to the normal force due to the tube.  

The mass travels in a straight line in the direction of the force to a new position.
At that new position the line of action of the normal force has changed and the mass again moves in a straight line but not along the same direction as before.
However with every step the mass is moving further from the centre of rotation.  

Using polar coordinates with a radial unit vector $\hat e_{\rm r}$ and a tangential unit vecor $\hat e_{\rm \theta}$.
The tube can only produce a force $\vec F$ on the mass, $m$, at right angles to its surface so $\vec F = F \; \hat e_{\rm \theta}$.
It can be shown that:  


*

*Position $\quad \vec r = r \; \hat e_{\rm r}$   

*Velocity $\quad \vec v = \dot r \; \hat e_{\rm r} + r \dot
   \theta \; \hat e_{\rm \theta}$  

*Acceleration $\quad \vec a= (\ddot r - r \dot \theta^2)\; \hat
   e_{\rm r} +(r \ddot \theta + 2 \dot r \dot \theta) \; \hat e_{\rm
   \theta}$


So applying Newton's second law $\vec F = m \vec a$ gives 
$F \; \hat e_{\rm \theta} = m(\ddot r - r \dot \theta^2)\; \hat e_{\rm r} +m(r \ddot \theta + 2 \dot r \dot \theta) \; \hat e_{\rm \theta}\quad \Rightarrow \quad \ddot r = r \dot \theta^2 $ and $F = r \ddot \theta + 2 \dot r \dot \theta$
$ \ddot r = r \dot \theta^2 $ is equivalent to the formula one would have obtained sitting in the rotating frame of reference when the marble would have been subjected to a centrifugal force.
Solving this differential equation is made easier because the angular speed $\dot \theta$ is constant.
Applying the initial conditions, $r=R$ and $\dot r =0$ when $t=0$, gives $r = \dfrac{R}{2}(e^{\dot{\theta}t}+e^{-\dot{\theta}t})$.
Because $\dot{\theta}t = \theta$ this can be rewritten as $r = R \cosh\theta$ where $\cosh\theta = \dfrac {e^{\theta}+e^{-\theta}}{2}$
Here is the path taken by the mass with $R=1$ and you can see that for the first part of the motion it is "almost" a straight line.  

To find the magnitude of the normal force, $F$.  
$F = r \ddot \theta + 2 \dot r \dot \theta$ with $\ddot \theta =0 \quad \Rightarrow \quad F=2 \dot r \dot \theta$
$\dot r = R \sinh \theta \; \dot \theta \quad \Rightarrow \quad F = 2\; R \;\dot \theta^2 \sinh \theta$
A: In the absence of external forces a mass would move in a straight line. To maintain a circular motion the mass needs to be "constrained" and this is done by the contact normal force $N$ acting by the tube to the mass. This force is real.
The magnitude of this force is found by the difference between the straight line path and the actual path. 
Lemma

If an object moves along a path, and at some instant the tangent vector is $\hat{e}$ and the normal vector is $\hat{n}$ then the velocity and acceleration vectors are decomposed as
$$ \begin{aligned} \vec{v} & = v \hat{e} \\
\vec{a} &= \dot{v} \hat{e} + \frac{v^2}{r} \hat{n} \end{aligned} $$

Where $v$ is the speed, $\dot{v}$ the acceleration rate and $r$ the radius of curvature of the path
To find the force of constraint all you need is $$\vec{N} = m \vec{a}$$
A: The mass will move under the influence of the normal force exerted on it by the tube walls. In order to find the motion of the mass that force would have to be specified in some way. One option would be to specify constant torque of a given magnitude. In your case, however, you seem to be specifying a kinematic condition (constant angular velocity $\omega$ with respect to the center of rotation of the tube, assuming that the tube is mounted radially; note that this is a detail that would need to be specified). The kinematic condition together with Newton's second law will allow you to find the equations of motion of your mass, also see ja72's answer.
