Physically, a negative total energy is a necessary and sufficient condition for avoiding all particles flying off to infinity separately.
However, it is always possible for some particles to be given enough energy to escape a system. In fact, this tends to happen in real life. Planets get ejected from their solar systems over millions of years, and stars "evaporate" from globular clusters. The escaping bodies get energy from the rest of the system. Note that two Newtonian point masses can have an arbitrarily negative total energy as their orbital separation shrinks, so one can in principle extract arbitrary energy from any pair of particles to give to other particles.
In my experience, though, this sort of behavior is almost certainly indicative of a bug. The three most common pitfalls are:
- Using the wrong integration algorithm. If you're using an unstable scheme like forward-Euler, it is expected that everything will go wrong. People who know better than to use forward-Euler will still often recommend 4th-order Runge-Kutta. This is better, but still a bad choice for long-term integrations, since it doesn't conserve total energy in the system. Leapfrog is a good choice to start with.
- Using too large a timestep. How exactly are you calculating the timestep? All too often people choose a number that looks small, without answering the necessary question of "small compared to what?"
- Not softening the potential. Remember from above how point particles can get arbitrarily close, seeing arbitrarily rapidly changing potentials and undergoing arbitrarily large velocities and accelerations? Arbitrarily large values are bad for simulations. At a minimum, timesteps should decrease as needed when particles get close to each other. To avoid timesteps going to zero, "softened" potentials are often employed. For example, change Newton's law of gravity to be $F = G m_1 m_2 / (r^2 + \epsilon^2)$ for some small length scale $\epsilon$ of your choosing.
For diagnosing the problem, try plotting the particles' positions simultaneously with the system's total energy. If you see the energy change every time two particles get close, you are suffering from hard potentials and large time steps. If you simply see a steady trend in the energy, you probably aren't using a symplectic integrator.
As for the initial conditions, again, everything should be fine with just a negative total energy. For small numbers of objects, you can try Steven Mathey's suggestion of particles equally spaced on a circle. Note this is physically unstable, so you shouldn't expect the formation to hold indefinitely.1
Once you have enough particles (at least 20, or better yet at least 100), you can leverage statistical averaging. Choose a velocity dispersion $\sigma$. Randomly place particles to sample the density distribution $\rho = \sigma^2 / 2\pi G r^2$. Give them velocities drawn from a Maxwell-Boltzmann distribution ($\sigma^2 = kT/m$), which will be isotropically distributed and independent of position. You have now constructed a singular isothermal sphere.
1In fact, no simulation should be expected to work arbitrarily far into the future, and you should always be aware of the limits of any simulations. Symplectic integrators preserve energy and momentum for long times, but they quickly scramble the internal state (you wouldn't necessarily use one to predict the location of Mars 10 years from now). Higher-order non-symplectic methods are often better at short-term accuracy (but unreliable for determining things like whether the solar system is stable on billion-year timescales).