Very good question! The point is that when the elevator begins to move (either upwards or downwards), it's accelerating, while the drone -- having no force acting on it directly -- is still moving at a constant velocity.
Imagine you had a drone in a moving car or airplane, and it was "hovering" next to you. This means that its velocity with respect to you (and the car or airplane) is zero. If the car is moving at a constant velocity, the drone will continue to stay where it is, since no external force is acting on it.
On the other hand, in the elevator, the drone is at rest with respect to the "building". But when the elevator "starts", it begins to accelerate. The fact that the elevator has changed its velocity does not reach the drone as it isn't in contact with the elevator like we are. As a result, the drone continues to be at rest with respect to the building while the top of the elevator accelerates towards the drone, whacking it. Something similar occurs when you start from the ground floor and try to move up. Again the drone is at rest with respect to the ground floor, but when the elevator starts to accelerate upwards, the bottom of the elevator rams at the bottom of the drone.
I don't have a drone to test this, but I assume that something similar would happen in a moving vehicle if the vehicle suddenly accelerated (or turned). It's very similar to the feeling you experience when a car suddenly accelerates: you're slammed back because part of your body hasn't got the information yet that the velocity of the vehicle has changed.
If I had a drone (and a long enough elevator), I would try to show that once the elevator begins moving at a constant velocity (which happens after the initial jerk) I could control the motion of the drone without any trouble.