I seem to have misunderstood your question. Your reasoning is correct. If you take time needed to cover a distance whilst going against the aether vs the time needed to cover a distance whilst going with the aether, then the time will be the same.
Given $Velocity=\frac{Displacement}{Time}$ where $d$ is the distance between mirror and light source
Case 1: Light source pointed with movement of aether
Time needed to complete first leg of journey:
$t_1=d\div(V_{light}-V_{aether})$
Time needed to complete second leg of journey, after reflection:
$t_2=d\div(V_{aether}+V_{aether})$
Case 2: Light source pointed against movement of aether
Time needed to complete first leg of journey:
$t_3=d\div(V_{light}+V_{aether})$
Time needed to complete second leg of journey (after reflection):
$t_4=d\div(V_{light}-V_{aether})$
Clearly the total time taken in Case 1 is the same as Case 2; $(t_1+t_2)=(t_3+t_4)$ but that's not what the Michelson-Morley experiment is about.
The key here is realising that the experiment's setup made light travel in directions perpendicular to each other through different times. Not one with the aether and one against aether, but perpendicular to each other and then measuring the interference pattern when they recombine. If one of the light beams had "assistance", the timing would be off. They did this by suspending the entire setup in liquid mercury and then rotating it. In other words, one of the light rays would have a different arrival time because it would be influenced by the aether movement, if there was any.
Here is a gif from Wikipedia that illustrates the difference perpendicular light rays would have if there was aether movement.
If there was aether movement in the horizontal direction (like in the gif), then the time taken for the horizontal light beam to reach the beam-splitter/mirror would be
$T_{horizontal}=t_{right}+t_{left}$
where $t_{right}=d\div(V_{light}-V_{aether})$
and $t_{left}=d\div(V_{light}+V_{aether})$
giving you $T_{horizontal}=\frac{d(2V_{light})}{V_{light}^2-V_{aether}^2}$
Think of the aether shifting the "endpoint" of the beam, that's why you see the blue dot bounce after the red dot bounces. Whereas the time taken for the vertical light beam to reach the beam-splitter/mirror would be
$T_{vertical}=t_{up}+t_{down}$
where $t_{up}=d\div V_{light}$
and $t_{down}=d\div V_{light}$
giving you $T_{vertical}=\frac{2d}{V_{light}}$
This is clearly different from the horizontal case, making the light beams out of phase and causing some interference. In this case, the horizontal motion of the aether doesn't affect vertical velocity.