Clearly, physicists thought gravity caused masses to accelerate. However, Einstein's thought experiment gave way to the reasoning that perhaps there is no difference between a constant gravitational field and a constant acceleration. The significant importance is that this allowed him to speculate that perhaps gravity affects massless objects, like photons.
Think about it. Assuming a constant gravity field is no different than a constant acceleration, then if a beam of light bends downward in an accelerating elevator, it must bend downward in an elevator in a gravity field. That's the breakthrough! We can easily understand a beam bending downward due to acceleration, but if gravity is causing a massless object to change direction just as if it had mass, then Newton's law of gravitation isn't complete. Under Newton, light with no mass experiences no force from gravity and thus doesn't bend.
With this new concept running around, it allowed physicists to speculate that gravity is more than just a force, but perhaps a result or a property even of the geometry of space. Perhaps massive objects cause space itself to curve, which allows even massless things like light to fall towards them.
Naturally one can see where that might lead... To a revolutionary way of considering the geometry of spacetime. That is why this one little notion was so massively important to our understanding of the universe