Can the quartic oscillator Hamiltonian be made quadratic?  I'm interested in turning a quadratic + quartic oscillator Hamiltonian,
$$H = \frac{p^{2}}{2m} +\frac{kx^{2}}{2} + \lambda x^{4},$$
into a purely quadratic problem. One of the simplest things I can think of is replace the quartic term by
$$x^{4} \rightarrow \langle x^{2}\rangle x^{2}.$$ 
Has anyone come across such an approximation? If yes could you give me some reference to the work where this approximation has been used. Is there any other simple approximation which can turn the above Hamiltonian into a quadratic problem?
 A: Classical case
The Hamiltonian you have introduced is the one of a nonlinear oscillator. While for a linear oscillator period does not depend on the amplitude for this one it does. This means that quadratic approximation will always fail for some amplitudes.
If you want to describe oscillations of small amplitude you can just neglect the last term.
If you want to describe oscillations of some substantial amplitude you can use the trick with $\left<x^2\right>$. This approximation will work well for some range of amplitudes but will fail for others.
You have a potential of the following form:
$$
V(x) = m \cdot \Omega^2(x) \cdot x^2;
$$
and want to replace it with this one:
$$
\overline{V}(x) = m \cdot \overline{\Omega}^2 \cdot x^2.
$$
The average frequency for symmetric potential can be found with the following formula:
$$
\frac{2\pi}{\overline{\Omega}} = \sqrt{2m}\int_0^A \frac{dx}{\sqrt{V(A)-V(x)}} \qquad (1)
$$
where $A$ is the amplitude of the oscillations. This is just the integral over half of the period of one of the Hamilton's equations:
$$
\frac{dx}{dt} = \frac{p(x)}{m}
$$
where $p(x)$ is determined from the energy conservation law: $H(p,x)=\text{const}$.
Formula (1) gives the exact value of the frequency for amplitude $A$. If you don't want to deal with integrals you can suppose $\left<x^2\right>=\frac{1}{2}A^2$ (this is true for harmonic $x$). This is a less exact approximation. 
Quantum case
The method discussed above can be used here for the higher states of the oscillator where quasiclassical approximation works well even for a nonlinear oscillator.
As for the lower states, the energy of the ground state is finite and if $\lambda$ is large enough you can not neglect it. You can find Perturbation theory useful in this case.
A: This is an extraordinarily famous approximation, perhaps the most famous one, called the "Hartree Fock" or "Self-consistent field" approximation. It is the earliest and most famous approach to solving complicated self-interacting systems.
But in the case you are using it, it is very poor. The problem you state can be solved much better by Bohr-Sommerfeld quantization, or BKS, to a far better accuracy than a self-consistent field. What you do with the self-consisten field is replace the quartic by a best-fit quadratic potential, which will give completely wrong wavefunctions, and not good energy levels (BKS will certainly be better for everything).
The point of this approximation is that it works for many-body systems, where you expect that many-body interactions do take the form of a self-consistent field to a good approximation. This idea is verified in both atomic and nuclear physics.
