In the particular problem I encountered, an electric field was zero at the origin and we were meant to prove that a particle at the origin was in an unstable state of equilibrium.
Is it enough to state that for any non-null coordinates, the electric field isn't zero, ergo the equilibrium is unstable? Or is there a more elegant way of proving it?
In the centre of a bowl there is equilibrium.
Put a ping pong ball in it. If this ball is ever shaken slightly away from equilibrium, it will immediately roll back. A "shake-proof" equilibrium is called stable.
Now, turn the bowl around and put the ball on the top. This is an equilibrium. But at the slightest shake, the ball rolls down. This "non-shake-proof" equilibrium is called unstable.
It is about the potential energy. Because, systems always tend towards lowest potential energy. The bottom of the bowl is of lowest potential energy, so the ball wants to move back when it is slightly displaced. The top of the flipped bowl is of highest potential energy, and any neighbour point is of lower energy. So the ball has no tendency to roll back up.
Mathematically, it is thus all about figuring out if the equilibrium is a minimum or a maximum. Only a minimum is stable.
You might for many practical/physical purposes be able to determine this by simply looking at the graph of the potential energy.
But mathematically, this can be solved directly from the potential energy expression $U$. Just look at the sign of the double derivative (derived to position).
If you have a 2D function, then you have more than one double derivative, $U''_{xx}$, $U''_{xy}$, $U''_{yx}$ and $U''_{yy}$. In this case, you must collect them into a so-called Hessian matrix and look at the eigenvalues of that matrix. If both positive, then the point is a minimum; if both negative, then the point is a maximum. (And if a mix, then the point is neither a minimum nor a maximum, but a saddle point).
This may be a bit more than you expected - but it is the rather elegant, mathematical method.
Is it enough to state that for any none-null coordinates, the electric field isn't zero, ergo the equilibrium is unstable?
No, that is not enough.
You are right with: At the point of equilibrium the electric force needs to be null.
But furthermore: The direction of the electric force in the surroundings of the equilibrium position is important.
Or is there a more elegant way of proving it?
It is usually easier to analyze equilibrium with potential energy, instead of with forces.
If there is no charge at the origin (that is producing the electric field), then by Gauss's law (in derivative form), the divergence of the electric field there is 0: $\frac{\partial ^2 V}{\partial x^2} + \frac{\partial ^2 V}{\partial y^2} + \frac{\partial ^2 V}{\partial z^2}= 0$. Unless all three of these quantities are zero, then one of them must be negative, which means that in that direction, the equilibrium is unstable.
Long story short, you're looking for a minimum in the potential energy. In most cases the tests described in the other answers will work well. The second derivative test discussed in previous answers can be inconclusive, though. In one dimension, when $U''=0$ you don't know if you have a minimum, maximum, or neither. To understand how to handle these more difficult situations, it's best to think of the potential energy as a Taylor series around the point of interest: $$U(x) = U(a) + U'(a) (x-a) + \frac{U''(a)}{2}(x-a)^2 + \frac{U'''(a)}{3!}(x-a)^3 + \ldots$$
Your first requirement to be an equilibrium, is $U'(a)=0$. Your requirement for stability is then $U''(a) > 0$, unstable if $U''(a)<0$, inconclusive if $U''(a) = 0$.
If your test is inconclusive, and that is unacceptable, then you move to the next higher derivative, $U'''(a)$. The condition there is if $U'''(a) \neq0$ then your potential looks cubic there, so the equilibrium is meta-stable (saddle point - stable one direction, not the other). If $U'''(a) = 0$ you can look to the next higher derivative.
If $U^{\mathrm{IV}}(a) > 0$, and all previous derivatives vanish, then your equilibrium is stable. $U^{\mathrm{IV}}(a) < 0$ is unstable, and $U^{\mathrm{IV}}(a) = 0$ is inconclusive (see next higher derivative).
Do you see the pattern? The character of an equilibrium point is fixed by the first non-vanishing derivative higher than first. If that derivative is even, then you use the positive/negative distinction for stable/unstable. If that derivative is odd, then you have a meta-stable point.
All of this gets tremendously complicated by linear algebra when you go multi-dimensional - you would need to deal with the equivalent of eigenvalues for tensors with ranks higher than 2 (I confess, I've never worked out the details).
The algorithm outlined above does have its limitations, though. Consider the function $e^{-x^{-2}}$ near the point $x=0$ (if we plug the hole there). Is that a stable equilibrium? If you graph the function, it certainly looks like it. If you start doing derivative tests, though, you'll run into the problem that all of the functions derivatives vanish there! Handling cases like this would require getting in to a more technical definition of how to characterize the local behavior of a function (involving sets, epsilons, and "there exists" type statements).
While checking for any point that its for unstable or stable equilibrium, graphical method can be used. If slope at that point is negative i.e. with increase in one coordinate other decreases and graph goes back to same point again,then its stable and for unstable its vice-versa.
