Theoretically, What is the difference between a black hole and a point particle of certain nonzero mass. Of-course the former exists while its not clear whether the later exists or not, but both have infinite density.
We should probably distinguish between a particle being "point-like" and a particle being "structure-less". In classical mechanics we talk of "point-like" particles, objects with no extension. It is the case that in general relativity any "point-like" mass would be inside of its event horizon and so would be a black hole.
In quantum-mechanics even a "structure-less" particle - a particle with no consitituent parts - is wave-like and has extension, though not a fixed size, and it can never be come exactly point-like since that would take an infinite amount of energy. I do not believe it to be the case, therefore, that quantum-mechanically all particles are black holes in any sense.
One big difference is that all electrons, for example, are identical, but all black holes are not. In particular, a black hole can have any mass at all, whereas a particle like an electron has a fixed value for its mass. This property of fundamental particles like electrons is ultimately what allows us to define fixed scales of length and time in the laws of physics. In a universe that didn't have massive fundamental particles, the laws of physics would have a certain kind of symmetry called conformal invariance, which would make it impossible to construct clocks or rulers according to universally standardized rules.
Another difference is that there are fundamental particles such as electrons and neutrinos that are stable (don't spontaneously undergo radioactive decay), whereas it is believed that black holes will ultimately evaporate into fundamental particles.
You say that both have infinite density, but this is probably not actually true. The mass of a particle like an electron is probably attributable to the soup of virtual particles that surrounds it, whereas in general relativity a black hole's mass really is localized at a mathematical point. (Of course this is kind of an unfair comparison, since we know that GR is wrong below the Planck scale. It's possible that GR's singularities aren't really singularities. I'm just trying to give an answer in terms of established physical theories.)
It's tempting to imagine that fundamental particles are black holes, but this is not possible. Classically, a spinning, charged black hole has constraints on its angular momentum and its charge in relation to its mass. Otherwise, there is no event horizon, and we have a naked singularity rather than a black hole. An electron violates both of these limits, but we don't observe that electrons have the properties predicted for these naked singularities. For example, naked singularities have closed timelike curves in the spacetime surrounding them, which would violate causality, but there is no evidence that electrons cause causality violation.
The notion of a point-like particle (electron, for example) is a very bad notion dwelling in Physics despite experiments say contrary. But if an electron tends to explode, the gravitating mass tends to collapse. Both features are manifestations of our blunders in describing physical phenomena.