I suppose there are many aspects to look at this from, anna v mentioned how Calabi-Yau manifolds in string theory (might?) have lots of holes, I'll approach the question from a purely General Relativity perspective as far as global topology.
Solutions in the Einstein Equations themselves do not reveal anything about global topology except in very specific cases (most notably in 2 (spatial dimensions) + 1 (temporal dimension) where the theory becomes completely topological). A metric by itself doesn't necessarily place limits on the topology of a manifold.
Beyond this, there is one theorem of general relativity, called the Topological Censorship Hypothesis that essentially states that any topological deviation from simply connected will quickly collapse, resulting in a simply connected surface. This work assumes an asymptotically flat space-time, which is generally the accepted model (as shown by supernova redshift research and things of that nature).
Another aspect of this question is the universe is usually considered homogenous and isotropic in all directions, topological defects would mean this wouldn't be true. Although that really isn't a convincing answer per say...