Classical case
The Hamiltonian you have introduced is the one of a nonlinear oscillator. While for a linear oscillator period does not depend on the amplitude for this one it does. This means that quadratic approximation will always fail for some amplitudes.
If you want to describe oscillations of small amplitude you can just neglect the last term.
If you want to describe oscillations of some substantial amplitude you can use the trick with $\left<x^2\right>$. This approximation will work well for some range of amplitudes but will fail for others.
You have a potential of the following form:
$$
V(x) = m \cdot \Omega^2(x) \cdot x^2;
$$
and want to replace it with this one:
$$
\overline{V}(x) = m \cdot \overline{\Omega}^2 \cdot x^2.
$$
The average frequency for symmetric potential can be found with the following formula:
$$
\frac{2\pi}{\overline{\Omega}} = \sqrt{2m}\int_0^A \frac{dx}{\sqrt{V(A)-V(x)}} \qquad (1)
$$
where $A$ is the amplitude of the oscillations. This is just the integral over half of the period of one of the Hamilton's equations:
$$
\frac{dx}{dt} = \frac{p(x)}{m}
$$
where $p(x)$ is determined from the energy conservation law: $H(p,x)=\text{const}$.
Formula (1) gives the exact value of the frequency for amplitude $A$. If you don't want to deal with integrals you can suppose $\left<x^2\right>=\frac{1}{2}A^2$ (this is true for harmonic $x$). This is a less exact approximation.
Quantum case
The method discussed above can be used here for the higher states of the oscillator where quasiclassical approximation works well even for a nonlinear oscillator.
As for the lower states, the energy of the ground state is finite and if $\lambda$ is large enough you can not neglect it. You can find Perturbation theory useful in this case.