Your calculations are correct. The light does indeed take longer to reach the other end of the box when they are moving in the same direction, and vice-versa.
How could this help, however, to convince the guy in the moving box that he was the one moving, and not me? When he compares the time that the light takes reaching From the endpoint of the box, and the time it takes bouncing back, it is the same, for him.
Byview of the way, this would also happen in classical mechanics with perfectly elastic ballsobserver at rest.