There is no a-priori sense in having an infinitely large potential, you can't calculate with infinite numbers (except for some very special mathematical considerations). So, the physical meaning that is given to such a potential must be implicitely contained in the accompanying text that explains the infinite potential well. An this meaning is commonly: "the particle is not allowed to penetrate the outside of the well". Or in mathematical terms: $\psi(x<0)=0$ and $\psi(x>L)=0$.
So you are absolutely right in suspecting that the infinite potential well can be considered equivalent to a bounded universe (with the presumably small dimension of the well, of course), as long as your definition of a bounded universe implies impenetrable boundaries. Note, however, that this is not the only way a (conceptual) universe can be bounded. You could also demand a finite universe with periodic boundary conditions $\psi(0)=\psi(L)$, which would be inequivalent to the infinite potential well.
If you find the boundary conditions of the infinite potential well rather unexpected, think about the similar finite potential well for a moment. This has everywhere "well"-defined potential, so clearly no math crash here. As the closer analysis shows, the walls of the finite well can actually be penetrated. If the energy $E$ of an eigen-state is above the upper potential value $V_{out}$, the solution outside the well is purely oscillatory, as is the case for the inside. But, if the energy is below the upper potential value (so that classically the particle would not be able to be outside), the solution becomes damped oscillatory, with the damping coefficient being proportional to the "missing" energy. So the more energy the particle is missing with respect to the outside potential, the shorter the distance until the wave function has attenuated to $1/e$ or beyond. This is actually the basis of the effect of "quantum tunneling".
Now you can imagine what happens, if you let $V_{out}\to\infty$: the energy of an arbitrary given eigen-state will less and less likely be above $V_{out}$, but more and more likely be below it. So it will be damped more and more strongly. Moreover, the damping goes to infinity because the state is missing more and more energy to the boundary of the well. In the limit, you can consider the wave function as being damped to zero within an infinitely short distance, which is nothing else than what you expect from the infinite potential well. Note, that this is not meant as a rigorous mathematical treatment, but just a mental picture, or as you have desired: a physical justification.
