The short answer for why gravity is unique is that it is the theory of a massless, spin-2 field. To contrast with the other forces, the strong, weak and electromagnetic forces are all theories of spin-1 particles.
Although it's not immediately obvious, this property alone basically fixes all of the essential features of gravity. To begin with, the fact that gravity is mediated by massless particles means that it can give rise to long-range forces. "Long-range" here means that gravitational potential between distant masses goes like $\dfrac{1}{r}$, whereas local interactions most commonly fall of exponentially, something like $\dfrac{e^{-mr}}{r}$, where $m$ is the mass of the force particle (this is known as a Yukawa potential).
Another important feature of massless particles is they must have a gauge symmetry associated with them. Gauge symmetry is important because it leads to conserved quantities. In the case of electromagnetism (a theory of a massless, spin-1 particle), there is also gauge symmetry, and it is known that the conservation of electric charge is a consequence of this symmetry.
For gravity, the gauge symmetry puts even stronger constraints on the way gravity interacts: not only does it lead to a conserved "charge" (the stress energy tensor of matter), it actually requires that the gravitational field couple in the same way to all types of matter. So, as you correctly noted, gravity is very unique in that it is required to couple to all other particles and fields. Not only that, but gravity also doesn't care about the electric charge, color charge, spin, or any other property of the things it is interacting with, it only couples to the stress-energy of the field. For people who are familiar with general relativity, this universal coupling of gravity to the stress energy of matter, independent of internal structure, is known as the equivalence principle. A more technical discussion of the fact that massless, spin-2 implies the equivalence principle (which was first derived by Weinberg) is given in the lecture notes found at the bottom of this page.
Another consequence of this universal coupling of gravity is that there can only by one type of graviton, i.e. only one massless, spin-2 field that interacts with matter. This is much different from the spin-1 particles, for example the strong force has eight different types of gluons. So since gravity is described by massless, spin-2 particles, it is necessarily the unique force containing massless spin-2 particles.
In regards to the geometric viewpoint of gravity, i.e. how gravity can be seen as causing curvature in spacetime, that property also follows directly (although not obviously) from the massless spin-2 nature of the gravitons. One of the standard books treating this idea is Feynman's Lectures on Gravitation (I think at least the first couple of chapters are available on google books). The viewpoint that Feynman takes is that gravity must couple universally to the stress tensor of all matter, including the stress tensor of the gravitons themselves. This sort of self-interaction basically gives rise to the nonlinearities that one finds in general relativity. Also, the gauge symmetry that was mentioned before gets modified by the self-interactions, and turns into diffeomorphism symmetry found in general relativity (also known as general covariance).
All of this analysis comes from assuming that there is a quantum field theoretic description of gravity. It may be concerning that people generally say we don't have a consistent quantum theory of gravity. This is true, however, it can more accurately be stated that we don't have an ultraviolet complete theory of quantum gravity (string theory, loop quantum gravity, asymptotically safe gravity are all proposed candidates for a full theory, among many others). That means that we don't believe that this theory of massless spin-2 particles is valid at very high energies. The cutoff where we think it should break down is around the Planck mass, $M_p \approx 10^{19}$ GeV. These energies would be reached, for example, at the singularity of a black hole, or near the big bang. But in most regions of the universe where such high energies are not present, perturbative quantum general relativity, described in terms of gravitons, is perfectly valid as a low energy effective field theory.
Finally, you noted that the extremely weak coupling of gravity compared to the other forces also sets it apart. This is known as the hierarchy problem, and to the best of my knowledge it is a major open problem in physics.
Regardless, I hope this shows that even hierarchy problem aside, gravity plays a very special role among the forces of nature.
