Someone is making a mod for the video game Kerbal Space Program (KSP), which implements propper N-body with point masses into the game (and possibly in the future also other dynamics), instead of the stock patched conics approximation. The "solar system" in KSP is a simplified and scaled down version of our own solar system. Someone did a N-body numerical simulation of this system, in which it can be seen that not all orbits are actually stable, namely two moons around the gas giant Jool, called Vall and Bop.
The gas giant Jool has, similar to Jupiter, three inner Moons which have a 1:2:4 resonance, however the middle one Vall gets kicked out very early on in the simulation.
This raises a question about when 1:2:4 resonance is stable/self-correcting. The main difference I noticed between the KSP and the real solar system is that in KSP the moons are much heavier compared to their parent body. For Io, Europa and Ganymede the mass ratios vary roughly between $2.5\ 10^{-5}$ and $7.8\ 10^{-5}$, while for Laythe, Vall and Tylo (their KSP equivalents) the mass ratios vary roughly between $7.3\ 10^{-4}$ and $1.0\ 10^{-2}$. The orbital elements and physical parameters of the celestial bodies in KSP can be found here. So this makes me wonder if there is a known limit (rough estimate) to this.
I have done N-body simulations with point masses myself with only two of the moons in 1:2 resonance with their default masses (so Laythe and Vall or Vall and Tylo) in which case their orbits seems to remain stable. Here are a few results, in which I calculated the semi-major axis, $a$, and eccentricity, $e$, as follows, $$ a = \frac{\mu r}{2\mu - v^2 r}, $$
$$ e = \sqrt{1 + \frac{v_\theta^2 r}{\mu} \left(\frac{v^2 r}{\mu} - 2\right)}. $$
For a reference in terms of time scale on the results the orbital period of the most inner moon Laythe is equal to roughly $5.3\ 10^4$ seconds. But I also have done simulations with all three moons in which I decreased the masses of Laythe and Tylo by a factor 10 (Vall already has a mass an order of magnitude smaller), in which case the their orbits also seem to remain stable. These results can be viewed here. This does looks more chaotic than any of the two moons configuration.
This is a related question. It doesn't has an answer yet, but one of its comments does link to a document, but this does not mention mass ratios as a criteria for whether three body resonance would be stable or not.
To be clear, I am only considering gravitational interaction between point masses, so for now no tidal forces or nodal precession. I am looking for some limit in terms of mass ratios between the parent and its satellites, similar as for example the Hill sphere which gives an approximate limit for the distance at which stable orbits are possible. Even though the N-body problem is considered chaotic I still would suspect that there would be some way to determine this ratio. Intuitively I would think that with only gravitational interaction such a system still can stable and self-correcting.
For example if the middle satellite would get perturbed at the closest approach of the inner satellite, such that the opposite side of its orbit would get raised, then that would mean that its period would increase and the points of closest approach with the other two satellites would start to drift backwards in the orbit. This initially would amplify the initial perturbation, since when the inner satellite would be at the initial position of closest approach it would be slightly ahead of the middle satellite, pulling it along slightly, increasing the opposite side of the orbit of the middle satellite even more. But once the points of closest approach would have drifted back far enough that the separation of would be to large to induce any significant pull. Once this drift of the points of closest approach would nearly have looped around 360° the separation distance with the inner satellite would decrease again, but his times each closest approach the inner satellite would pull the middle satellite back, counteracting the initial perturbations.