It seems that the answer to my question is that there's a difference between the pre-inflation universe "singularity" and that of other types of singularities like a black hole. It wasn't a singularity as describes a black hole, it was a point in time where the scale of the universe was zero. All that exists didn't occupy a single point in space in the way matter collapses into a singularity in the universe we know, and the density of the universe was still homogenous. But the scale of it was reduced so much that the distance between everything was essentially zero, despite being an infinite universe, hence the big bang "happening everywhere at once" instead of at what we consider in our narrow view to be a single point.
As pointed out in the comments, this question has a very good explanation of how that works.
Edited with corrections suggested in the comments, thanks!