What's lowest mass object that can form from collapsing gas? Is there a minimum mass for an object that can form on its own from collapsing interstellar gas without being bound to a larger star's protoplanetary disk first?
There are red dwarfs, brown dwarfs, sub-brown dwards and there may be even lower mass objects we just can't detect because they are to faint.
What I read about rogue planets and rogue comets it seems they can only be ejected from a star system and can't form on their own. Why?
 A: The Jeans instability occurs when there is a cloud with a mass larger than the Jeans length, or $$M_J > \frac{\pi}{6}\frac{c_s^3}{G^{3/2}\rho^{1/2}}\sim (2M_\odot)\left(\frac{c_s}{0.2 \text{km s}^{-1}}\right)^3\left(\frac{n}{1000 \text{cm}^{-3}}\right)^{-1/2}$$
where $c_s$ is the speed of sound in the gas and $n$ is the number density. The above values make sense for normal astrophysics.
However, by making $n$ larger we can clearly make $M_J$ smaller. In an ideal gas the speed of sound only depends on temperature, not the pressure or density since they increase in lockstep ($c_s=\sqrt{\gamma (p/\rho)}=\sqrt{\gamma k_B T/m}$ where $m$ is the molecular mass). In practice this will fail when intermolecular forces become significant. So somewhere between air density ($n\approx 2.53\times 10^{17}$ per cubic centimeter) and water density (about 1000 times more) this formula will really break down, mostly because the cloud is no longer a gas but a liquid. That implies a Jeans mass somewhere close to $4.24\times 10^{-7} M_\odot$. That is 14% of an Earth mass.
But even smaller gas clouds could of course collapse if we make more extreme assumptions, like assuming a very low temperature. One could even have a virialized tiny cloud of gas in an otherwise empty universe, and this would slowly lose energy from gravitational radiation until it coalesced. In this case even two atoms orbiting each other would collapse (but it would take long time; for two hydrogen atoms starting 1 meter apart the coalescence timescale is around $10^{143}$ years).
In normal astrophysical environments tiny objects do not condense because temperatures are too high and densities too low. Even the cool molecular clouds allowing star formation are some tens of kelvin and have at most a thousand atoms per cubic cm; the super-dense clouds envisioned above will not occur unless you are inside a collapsing cloud. There, of course, you do get planet and comet formation as a side effect of the star formation.
