If you go back to the origins, the difficulty in merging gravity with the other forces mostly stems from general relativity being a purely geometric theory -- again, that's in its original form -- and all the other forces being quantum, by which I mostly mean they are conveyed by well-defined force particles. The photon as the particle that conveys the electromagnetic field is the simplest example, but the idea carries over very well to both the weak and strong forces.
General relativity in contrast works very, very well without even invoking such concepts, or for that matter particles in general. As Einstein formulated it, GR really, truly is all about curved spaces.
By analogy it was subsequently assumed -- I think sometime back in the 1960s or perhaps 1950s? -- that gravity must also have a quantum form, but it's always been an assumption, not an absolute proof, a sort of "it worked here, and here, and here... so surely it also works just as well for the last force, gravity?"
But it's a bit tough to bridge such a huge gap. It's reasonably easy to provide a general description of gravity as a universally attractive force, although even there you quickly get into odd infinity problems not seen with other forces. But if you do that... what happened to all that part about the space being curved? The simplest possible all-attractive quantum force model would simply keep space as a rigid framework and do everything pretty much as with the electromagnetic force, only with a single type of charge (mass).
So, you can do one (curved space), or you can do the other (universal graviton-mediated quantum attraction), but it's not trivial to do both. And no matter how you do it, the other forces don't directly bend space, which keeps gravity pretty unique.
Consequently, the details never seem to work out quite right, and many books have been written about why that might be.