Can we give a sense or put in context the fact that gravity isn't a force yet is one of the fundamental ones? As per the title.
Not that the title fact poses me big troubles, personally, but is at least confusing that the two assertions coexist within the same article, interview, and thread — or can be individually found on sources with reputation.
What I grasp is that both are true and it is matter of point of view (i.e.  gravity isn't a force in GR, and GR and standard model don't need to speak each other unless about "big bang like" conditions). But I would appreciate if someone could clarify and put in the right context this apparent dichotomy (or dichotomy removal, if one wish).
 A: As existing theories are replaced by new theories the existing words are redefined.
Example:
During the time that Caloric theory was assumed to be true the word 'heat' was understood as 'amount of Caloric'. Some time later Caloric theory was replaced by the kinetic theory of heat. Physicists did not stop using the word 'heat'. Instead 'heat' was redefined: heat is kinetic energy of molecules and atoms.
The newtonian definition of 'force' is: that which causes acceleration with respect to the local inertial coordinate system.
So: upon moving from  newtonian dynamics to GR we need to redefine how the word 'force' is to be understood.
One way is to define 'force' as the derivative over distance of potential energy. We all agree that gravitational potential energy is an indispensible concept. Example: as a comet moves from the outer reaches of the solar system to the inner planets gravitational potential energy is converted to kinetic energy.
If it is agreed that there is such a thing as gravitational potential energy then it follows that gravity is a force.

When people say: "Gravity is not a force" then what they are actually saying is:
"If we transition to general relativity, but at the same time we keep using the newtonian definition of force, then gravity is not a force."



[Later edit](currently this answer is at 3 upvotes)
An example of a phenomenon that is not categorized as a force is degeneracy pressure. As we know: the importance of degeneracy pressure cannot be overstated. It defines the Chandrasekhar limit, etc., etc.
Degeneracy pressure plays a fundamental role, but: the concept of potential energy, and conversion of potential energy to kinetic energy, is not applicable in the case of degeneracy pressure. Just that is already sufficient to put degeneracy pressure outside the category 'force'.
It seems: what people are keen to express is that the mediator of gravitational interaction is very different from the respective mediators of the "other three". I assume people feel a need to express that gravitational interaction appears to be in a category of its own. The view that gravity is in a category of its own is good physics; it's just that the assertion "Gravity is not a force" is the wrong vehicle for expressing such a view.


[Later later edit]
About degeneracy pressure.
Let me make a comparison. Geophysicists have performed experiments where a tiny amount of a particular substance is subjected to the kind of pressure that otherwise exists only thousands of kilometers deep in the Earth. (This type of setup is called a 'diamond anvil cell'.) At such high presssure it is observed that substances are in a denser crystal structure than at surface pressure.
That denser crystal structure is unstable at surface pressure because the denser structure is like a highly compressed (coil) spring. It's a higher state of energy than at surface pressure, but at the extreme pressure deep in the Earth it is the lowest available state of energy.
The most extreme state for ordinary matter is the state of energy of the matter that constitutes a white dwarf. Specifically, a white dwarf that is close to the Chandrasekhar limit.
The matter constituting the white dwarf is extremely compressed, putting it in the highest possible state of energy. If some more matter accretes onto that white dwarf, raising the mass above the Chandrasekhar limit, then there is a most profound transition. No longer can the matter climb to an even higher state of energy: that path is exhausted. Then the entire composition of the white dwarf transitions to a neutron star.
To be a neutron star is a higher state of energy than before the transition to being an neutron star.
A: The picture of "four forces" comes from particle physics / quantum field theory (it's also kind of a lie but that's a different story). Within this framework, we can think of gravity (at least, perturbatively, when the gravitational effects are small) as being mediated by a massless spin-2 particle. In this sense, gravity is simply another interaction -- the electromagnetic, strong, and weak forces are mediated by spin-1 particles, and gravity by a spin-2 particle.
General relativity is a purely classical theory and so is not the right language for unifying gravity with other forces, which are fundamentally quantum mechanical in nature. But, GR does not assume the gravitational fields are weak. Within GR, as you said, we typically think of the gravitational "force" as being due to curvature of spacetime.
Fully reconciling these two different pictures requires a quantum theory of gravity. While there are compelling candidates (such as string theory), we don't know the full story here, and may never know it in our lifetimes.
Having said that, this is somewhat of a philosophical issue. At the level of physics, whenever we are in the regime of validity where we can apply both theories (weak gravitational fields and weak quantum effects -- which actually covers an enormous number of situations in practice) -- both frameworks make identical predictions for all observable quantities.
