It would depend on two things - how much air is in the balloon and the tensile strength of latex. To see why, I hope you'll find the following useful.
In a balloon on earth near the surface, the pressure inside depends on how much air you blow into it. But in order for the balloon to be in static equilibirium, that internal pressure has to be matched by the (constant) atmospheric pressure that opposes outward expansion PLUS the elastic surface tension of the surface, which also tends to oppose outward expansion (wanting to minimize the surface area). $$P_i = P_0 + S$$ If you blow more air into it, $P_i$ increases, and since $P_0$ is constant at the same height, the surface tension has to increase for LHS and RHS to match. This will continue until the surface tension exceeds the tensile strength of the balloon, at which point the balloon pops.
Now let's say you have blown only enough air into the balloon to make it taut -- if you let this balloon go so that it floats up, the atmospheric pressure $P_0$ starts dropping with height, and so again, the surface tension has to keep rising so that the RHS matches the constant internal pressure $P_i$.
Two things could happen now.
- Even before $P_0$ has dropped to zero (in the vacuum of outer space), the surface tension has had to become so high that it exceeds the tensile strength and makes the balloon pop. This is what happens to most helium balloons let go - they eventually pop even before reaching vacuum - because whatever material the surface is made of has a tensile strength too low to be able to sustain the minimum amount of internal pressure required to make the balloon taut once the external pressure has dropped enough. Note that this minimum amount of internal pressure could be made smaller by reducing the size of the balloon but you can't do this beyond the limit at which the balloon's mass overcompensates and makes it impossible for it to float in the first place!
- The other possibility is that the tensile strength is high enough that even after $P_0$ is zero, the surface tension required to match the internal pressure doesn't exceed the tensile strength and you'll have a nice happy balloon floating in the vacuum. This is what happens with space stations, for instance :)