This answer is an interpretation of the paper helpfully referenced by John Rennie.
First of all, as you note, the L2 Lagrangian point is an unstable equilibrium, so all orbits about it are unstable, requiring active station-keeping to damp out perturbations.
Orbits around the Lagrangian point are categorized as either:
- Lyapunov (orbits entirely in the plane of the earth-sun motion)
- Halo (orbits that extend above and below the earth-sun rotation plane)
Halo orbits are preferred for spacecraft, because the sun is never in-line with the spacecraft and the earth, minimizing communication interference (I think).
Lyapunov (in-plane) orbits. The small-signal stability analysis of in-plane motion yields 4 eigenvalues (natural frequencies): one real stable, one real unstable, and a pair of (complex conjugate) oscillatory eigenvalues. With just a single oscillatory mode, there are periodic orbits of arbitrarily small amplitude. (Again, station-keeping is required to damp out any appearance of the unstable mode.)
Halo (3 dimensional) orbits. Allowing out-of-plane motion adds another pair of oscillatory eigenvalues to the list (now at 6). Crucially, this new frequency is in general not related rationally to that of the in-plane oscillatory frequency, so periodic halo orbits do not exist at small amplitudes.
Enter the non-linearity: as the amplitude of an orbit increases, its dynamics change, because of the non-linearity. (One analogy is describing function analysis, which models a non-linear element with a near sinusoidal signal of amplitude $A$ by a linear element whose gain depends on $A$. As the amplitude increases, the gain changes.)
Back to the halo orbits. Qualitatively, as the orbit amplitude increases, the natural frequencies shift, and eventually a pair of complex eigenvalues becomes real (one stable, one unstable). (See Figure 3 in the paper. The analogy here is a root locus plot.) Since there is already an unstable eigenvalue, one more isn't that big a deal. The important thing is that now there is only a single complex pair of eigenvalues, corresponding to a single oscillatory mode, and periodic solutions become possible.
For the earth-sun L2 point, the paper reports the minimum amplitude for halo orbits to exist to be about 210,000 km (in the radial x direction), and calculates orbits in the range of 500,000 km.
Note: The paper refers to the Next Generation Space Telescope (NGST); it's since been renamed as the James Webb Space Telescope (JWST).