Some people speculate that the mysterious dark matter in the universe could be tiny black holes. But on the other side, could dark matter particles attract each other by gravity and finally form a black hole? Since dark matter is even more abundant than normal matter, the dark matter black hole should not be rare.
I think the problem with matter that only interacts gravitationally is that it's hard to get it all to stay in one place. Nebula slowly form stars and planets in part because of collisions between particles lead to larger particles, which tend to attract further particles. But particles that just wizz right through each-other can't coalesce without violating conservation of angular momentum. That's not to say that it's impossible, however. Just that the dynamics are different.