What you have done here is a Galilean transform, that is a non-relativistic transformation. Take your final result (which is quite correct):
$$ t' = \frac{\sqrt{\beta^2 + \alpha^2}}{\sqrt{\eta^2 + \mu^2}} \tag{1} $$
We know that the vertical velocity is $\eta$, so the vertical distance moved in our time $t$ is given by:
$$ \beta = \eta t $$
We also know that if the horizontal velocity is $\mu$ then the horizontal distance moved in our time $t$ is:
$$ \alpha = \mu t $$
If we substitute this into your original equation (1) we get:
$$ t' = \frac{\sqrt{\eta^2t^2 + \mu^2t^2}}{\sqrt{\eta^2 + \mu^2}} $$
and if we take the $t$ outside the square root we get:
$$ t' = t\frac{\sqrt{\eta^2 + \mu^2}}{\sqrt{\eta^2 + \mu^2}} $$
while gives us:
$$ t' = t $$
So both times are the same. And this is the correct result for Galilean relativity because in Galilean relativity time is invariant.
To move from Galilean relativity to special relativity we need an extra assumption, and the assumption we need is called the invariance of the line element. To explain this I'm going to simplify your example a bit to avoid the annoying square roots. We'll assume that all motion is in one direction that we'll call $x$ so we just have two dimensions $x$ and $t$. Now, if you have two points in spacetime separated by a distance $\Delta x$ and a time $\Delta t$ then the line element $\Delta s$ is defined as:
$$ \Delta s^2 = -c^2\Delta t^2 + \Delta x^2 \tag{2} $$
The key assumption we need is that all observers, no matter how they are moving, will calculate the same value for the line element $\Delta s$.
To illustrate how this works I'll do a calculation like yours but I'll replace the moving ball with a clock that ticks every $t$ seconds. Let's say I'm on the train holding the clock while you're watching from the ground. We'll take the train to be moving at a speed $v$. As I pass you I start the clock, so we both measure the first tick to be at the point $(t = 0, x = 0)$.
Consider first my point of view. For me the clock isn't moving so its position doesn't change and $\Delta x = 0$. The tick arrives after a time $t$ so $\Delta t = t$. That means when I feed these into equation (2) I calculate the line element to be:
$$ \Delta s_\text{me}^2 = -c^2t^2 \tag{3} $$
Now consider your point of view. Suppose you measure the clock to tick at some time $t'$ that we need to calculate. So you measure $\Delta t = t'$. For you the train is moving at a speed $v$, so in the time $t'$ it moves a distance $vt'$. So you measure $\Delta x = vt'$. That means when you feed these into equation (2) you calculate the line element to be:
$$ \Delta s_\text{you}^2 = -c^2t'^2 + v^2t'^2 \tag{4} $$
The invariance of the line element means we must both get the same value for $\Delta s$ so:
$$ \Delta s_\text{me} = \Delta s_\text{you} $$
And substituting for $\Delta s^2$ using equations (3) and (4) gives us:
$$ -c^2t^2 = -c^2t'^2 + v^2t'^2 $$
and if we rearrange this we get:
$$ t = t'\sqrt{1 - \frac{v^2}{c^2}} $$
or
$$ t' = \frac{t}{\sqrt{1 - \frac{v^2}{c^2}}} $$
And you should recognise this because it's the well known equation for time dilation. The equation tells us that your time $t'$ will always be greater than my time $t$, that is you observe my time to be running slow.
