There are several problems with this argument.
First, cosmological models having a singularity just means that they blow up as $t\to 0^+$ and aren't defined at $t=0$. That's interpreted as a limitation of the models, not evidence that real-world quantities were infinite at $t=0$. Several other answers have already covered this.
Second, even inasmuch as you can complete the topological manifold and treat the model as having a $t=0$, the completion usually isn't a single point, because no single point can be in the causal past of the whole space at early $t$ (this is the horizon problem). Even if $k>0$, meaning the space at every $t>0$ has a finite volume and it goes to zero at $t\to 0$, the completion is still topologically a 3-sphere, not a point.
Third, even if you could complete the manifold with a single point, that isn't obviously a problem. If you put polar coordinates on Euclidean space, there is no homotopy from the space at $r=0$ to the space at any $r\ne 0$. To be fair, I don't think anything closely analogous to that Euclidean example can happen with Lorentzian signature, for the handwaving reason that at the central point all directions would have to be timelike, which is inconsistent with the assumed signature. But to prove that you have to make a coordinate-independent argument, not an argument that depends on a $t$ coordinate and associated foliation.
I think there are (not widely accepted) models in which the universe starts with a Euclidean signature, so that $t=0$ can be a mere coordinate singularity like the $r$ of the previous paragraph. I don't know how the transition from one signature to another is dealt with.