Is the motion of a particle in the surface of a torus always periodic? I am trying to see if there are ballistic trajectories in the surface of the torus that are not periodic and to what extent. Maybe it is not only quasiperiodic but chaotic. I guess there are quasiperiodic trajectories but I tried some angle-action variables approach and could not solve it.
I wrote the Lagrangian
$$L=\frac{(a+b\cos\vartheta)^2\dot\varphi^2}{2}+\frac{b^2\dot\vartheta^2}{2}$$
and from it the Hamiltonian
$$H=\frac{p_\varphi^2}{2(a+b\cos\vartheta)^2}+\frac{p_\vartheta^2}{2b^2}$$
where $a$ is the larger radius, $b$ is the smallest radius, $\varphi$ is the azimuthal angle (motion in the plane of the torus) and $\vartheta$ is the angle with respect to $z$ axis (perpendicular to the plane of the torus). Both $\varphi$ and $\vartheta$ can take values from $0$ to $2\pi$.
The conjugate momentum $p_\varphi$ is conserved and then I wanted to calculate the action-angle variable for $\vartheta$ but I cannot solve the integral:
$$I_\vartheta=\frac{b}{2\pi}\oint \sqrt{2E-\frac{p_\varphi^2}{(a+b\cos\vartheta)^2}} \mathrm{d}\vartheta,$$
where $E>\frac{p_\varphi^2}{2(a+b\cos\vartheta)^2}$ is the energy. I do not know how to integrate this nor I am sure to know what are the turning angles (if any). Hopefully, I could use commensurability condition between the frequencies to see what is the periodic orbit condition.
Maybe there is an easier way to solve it. I guess that if the motion in $\vartheta$ completes full rotations and it is so that $\dot\vartheta=\mathrm{constant}$, some geometrical analysis can be made. I will try toroidal coordinates to see if it helps.
Additional insight: my guess is that there are quasiperiodic trajectories. When discussing KAM theorem, one visualizes the phase-space dynamics with a picture of spiral-like trajectories in a $d$-dimensional torus. If the trajectories are non-commensurate, the trajectories will cover the whole torus. Which implies for the case above, that we could draw some non-periodic spiral-like orbits on the surface of the 2-torus. However, it is unclear if the Hamiltonian above is equivalent to that picture of spiral like trajectories.
 A: The OP has different questions on the title and body:

Is the motion of a particle in the surface of a torus chaotic?

Generically, yes, mechanical systems are chaotic — though not always, and the purely ballistic dynamics here in particular is not chaotic [1].

there are ballistic trajectories in the surface of the torus that are not periodic [?]

Yes, there are nonperiodic trajectories, but that doesn't mean they are chaotic.
This is easiest to visualize if you map your torus to a periodic square and realize that, for the ballistic movement, the trajectories are then straight lines: the $x$ and $y$ components of the speed must be constant, defining a linear flow on the torus: when $v_x$ and $v_y$ are commensurate, then the trajectory will close on itself after a finite time and repeat itself, i.e., it's periodic; if they are incommensurate, i.e., if $v_x/v_y$ is irrational, then the trajectory covers densely the torus, never closing on itself, i.e., it's quasiperiodic — which is nonperiodic, but also regular (i.e., not chaotic).
Note that this problem has an interesting connection with billiards.
And it's also related to the tori one often finds in the phase spaces of Hamiltonian systems, which play a central role in fundamental dynamical systems' theorems such as Kolmogorov–Arnold–Moser's.
For the mechanical system of a particle constrained to move on a physical toroidal surface (instead of on formal one), I suppose the constraint force (Coriolis like) would make the system similar to a pendulum, which displays oscillations about its equilibrium, but no chaos when unperturbed. That much is suggested by exercise 4 in http://www.phys.virginia.edu/Education/Qualifier/ClassicalMechanics.pdf and by a calculation by Kirk T MacDonald on a system that additionally considers gravity.
Thus this (more) physical system could have quasiperiodic orbits (with incommensurate oscillation and rotation frequencies), but probably not chaos — finding enough constants of motion would be a way of proving this.
[1] Geodesics on the torus seem not to be chaotic — see, e.g., Mildenhall's paper and Jantzen's.
A: My guess is no.
Chaos requires two trajectories that start out close together separate exponentially.
If you don't have a potential you have trajectories like spirals that wind around the smaller diameter of the torus, or orbits around the outer equator of the large diameter. If the orbit is above the equator, centrifugal force will move it toward the equator. It will oscillate above and below the equator as it orbits.
If you have a potential, things change. Think of a torus laying on a table. Given two particles sliding along the top, one might slide off the inside and the other outside. You might think that would lead to chaos, but I am not sure.
A simpler version of the problem would be a pendulum that can swing in a complete circle. If you balance two pendulums, they might fall over in opposite directions. But a simple pendulum is not chaotic. The simplest chaotic system known is a double pendulum.
Of course you could invent a potential that would separate particles exponentially and generate chaos.

One last note - You can pick a velocity for a particle, and then choose a direction so that it winds around the tube $10$ times as it orbits the equator $1$ time. This is just right to make a closed orbit.
You could choose a different angle where it winds around $11$ times for a different closed orbit.
You could choose any number between those two. For example it might wind around $10.5$ times. This means after one orbit it comes back on the opposite side of the tube, and it forms a closed orbit after two orbits.
You get a closed orbit for any rational number. But an irrational number never closes.
A: There is an interesting answer over in Math StackExchange. The OP even has a picture of an example of non-periodic motion--see the upper-right illustration below.

