The immiscibility transition is when the bosons and fermions separate and stop occupying the same point in space. Think of oil and water.
To approach that condition you can simply look at the mean field excitation spectrum for the homogeneous(unseperated) system. For a stable system, the energies of your excitations will all be real valued. At the transition, some of the energies start to become imaginary. As you can imagine, if you put these energies in an evolution operation, they would create divergences. These divergence signal the instability.
Roughly you can expect this to happen when the inter-species interaction is greater then the intra-species interaction.