Most earthquakes with magnitude 5.5 and higher can damage or destroy buildings. However, according to my knowledge and experience, I have never seen someone dying from an earthquake itself. Rather, they die from an associated tsunami, damaged buildings, etc.
This seems counter-intuitive, since you need much more force to destroy a building or damage it than to break the human femur or cause similar damage to other species.
There are several differences between humans and buildings:
Anchoring Suppose I were to push you backward. Just on reflex, one of your feet will move backward to catch yourself. A building, on the other hand, is anchored in place. When the ground moves, the bottom of the building has to move with it. But a human body doesn't have to move with the ground. There's only so much force that a shaking ground can apply to the human body.
Flexibility: The human body is designed to move (anthropomorphizing evolution a bit here). Other than bones, our organs are flexible, and the bones are connected by multiple joints and that can bend without harm to the body (that what a joint is: a part of the body that's supposed to move). A building doesn't have that level of flexibility. Nowadays buildings, especially ones in earthquake-prone regions, are often designed with some flexibility, but that can't match that of a human body.
Scaling: This is probably the largest factor. Things work differently at different scales. If you double every dimension, you multiply the cross-section by four, but you multiply the volume by eight. If a building is a hundred times as large as a human in each direction, then it's going to have a hundred times the volume per unit of the cross-section. And torque is proportional to both mass and distance, so you can get ten thousand times as much torque per area.
The maximum recorded earthquake peak ground acceleration is less than 5g, which is typically survivable, but buildings typically are not designed for such (not very short-term) acceleration.
You are right, that you need more force to break a building than to break a human body.
But an earthquake acts by acceleration (not by force). It suddenly accelerates a large part of the ground (and hence all buildings and humans there) by the same value.
The human body is quite soft and flexible,
and can therefore withstand accelerations of $10\ g$ for a few seconds
(see G-force - Human tolerance).
Also quoted from there:
The human body is flexible and deformable, particularly the softer tissues. A hard slap on the face may briefly impose hundreds of $g$ locally but not produce any real damage;
But hard buildings can only withstand much smaller accelerations.
And they are even more sensitive for horizontal accelerations
than for vertical accelerations.
According to Earthquake effects on buildings (chapter 4):
Poorly constructed buildings begin to suffer damage at about 10 percent $g$ (or $0.1\ g$).
Force is the rate of change of momentum (the product of mass and velocity of the object): $$ F = \frac{dp}{dt}$$ where $p = mv$ is the momentum. An object with twice the mass has twice as much momentum compared to an object moving with its same velocity.
So a building of a million kilograms will have ten thousands times the momentum of a human weighing a hundred kilograms (assuming they are both planted to the ground so that the oscillating floor drives them both with the same velocity). As the floor oscillates, the velocity will be changing, but the mass doesn't.
So the change of momentum - the force - that goes to the building would also be ten thousand times the amount of force that goes to the human.
This is also why an ant can survive a fall from a 20 metres tree all safe, but a human will likely at least break some bones.
In addition to the other answers provided, resonance cannot be ignored (Resonance is where the object is vibrating at its natural frequency and as a result the vibrations are amplified). To put simply, people and buildings have different resonance frequencies and the lack of being anchored to the ground (for people) mitigates the damage done in the way.
The resonance frequencies of buildings play a large role in the destruction caused by earthquakes. Contrary to common misconceptions, the tallest buildings aren’t always the most damaged during an earthquake.
Check out this great demo for a visual representation of how resonance frequencies amplify building destruction. (Begin at the 1:25 mark for best experience).
As an addendum to other answers, pressure and volumetric energy density may be more useful in explaining the total effect on a small body vs large structure.
Since reasonably-sized buildings and people on the surface will be a small fraction of the overall earth mass being moved in a quake, the effective pressure $$P=F/L^2$$ in a localized region will be uniform. In other words, the effective force transmitted from the ground to an object will be proportional to the contact area of the object. A person's feet (or even a person lying prostrate on the ground) has a much smaller area than that of a building's rigid foundation.
Energy density has the same units as pressure, like $$[E/V] = [E/L^3] = [{FL}/{L^3}] = [P]$$ Similar calculations would show that the total energy transmitted to each object will be proportional to its mass, density and displacement. The conclusion is the same... that a small body (with less mass and volume) will receive less overall energy from the quaking ground than an entire rigid building of much greater mass. It does not matter that the human body is standing adjacent to or even leaning up against the larger building. The human body will still only receive its small portion of energy directly from the quaking ground.
flex$\endgroup$