Deducing instability growth rates from the Hamiltonian for the non-linear Schrödinger equation Consider the following nonlinear Schrödinger equation (NLSE):
$$A_t+iA_{xx}+i|A|^2A = 0, \tag{1}$$
where $A$ is a complex valued function of $(x,t)$.
A solution to this equation is $$A=a_oe^{-ia_o^2t}.\tag{2}$$ We investigate the stability of these solutions by considering a perturbation to the above solution of the form
$$a_oe^{-it} (1+\alpha_+e^{i(kx-\Omega t)}+\alpha_-e^{-i(kx-\Omega t)})\tag{3},$$ for constants $(\alpha_+,\alpha_-, k,\Omega).$ 
Putting this back into the NLSE and collecting terms of order $\alpha_{\pm}$, we find a system of equations
$$ \left( \begin{array}{cc}
a_o^2-k^2-\Omega & a_o^2 \\
a_o^2 &  a_o^2-k^2+\Omega \end{array} \right) \left( \begin{array}{c}
\alpha_+ \\
\alpha_- \end{array} \right) = 0.\tag{4}$$
This has non-trivial solution when the determinant of the matrix is zero, which is the condition that  $$\Omega^2 = k^2(k^2-2a_o^2) \tag{5},$$ which takes on imaginary values (ie leads to unstable growth) when $$k < \sqrt{2} a_o.\tag{6}$$
(Note: The way the problem has been non-dimensionalized, all of the parameters (eg $k,a_o$) are unitless.) This has maximal growth rate $$\mathrm{Im}(\Omega_*) = a_o^2\tag{7}$$ for $k_* = a_o$.
Now, this is equivalent to the so called Benjamin-Feir (or modulation instability) inherent to a variety of physical systems (eg water waves, lasers). Next, recall that the NLSE can be derived from a Hamiltonian density ${\cal H}$, where
$${\cal H} =  |A_x|^2 - \frac{1}{2} |A|^4,\tag{8}$$
and Hamilton's equations take the form
$$i\frac{\partial A}{\partial t} = \frac{\delta {\cal H}}{\delta A^{\ast}}\tag{9}$$ 
and the complex conjugate of this. 
My question is, is there anything in the structure of this Hamiltonian that can give us the results of the spectral stability analysis in a different manner? Naively, I want to know if I can deduce the instability criteria and the growth rate directly from the structure of the Hamiltonian.
I am vaguely aware of certain stability criterion used in the study of the stability of Hamiltonian systems (eg Krein signatures etc), but am not familiar with how it works in practice (and in particular in this relatively simple example) and would greatly appreciate any tips, or references to relevant resources.
 A: I) The Hamiltonian density for the NLSE reads
$$ {\cal H}_{NLSE}~=~|A_x|^2 - \frac{1}{2} |A|^4, \tag{A}$$
where we are interested in the case with an unstable sign in front of the quartic term. The Hamiltonian Lagrangian density is
$$ {\cal L}_H~=~i A^{\ast} \dot{A}  -{\cal H}_{NLSE},\tag{B}$$
cf. e.g. this Phys.SE post.
II) Let us replace 
$$A \quad\longrightarrow\quad (a_0\!+\!A)e^{i(k_0x-\omega_0t)}, \qquad \omega_0~:=~k_0^2-a_0^2,\qquad a_0~>~0,\tag{C}$$
since we are interested in the Benjamin-Feir modulational instability around a carrier wave $a_0e^{i(k_0x-\omega_0t)}$, and the new variable $A$ is a perturbation $|A| \ll a_0 $.
The Hamiltonian Lagrangian density (B) becomes
$$ {\cal L}_H~=~(A^{\ast}\!+\!a_0) \left\{\omega_0(A\!+\!a_0)\!+\!i  \dot{A}\right\} -\left|A_x+ik_0(A\!+\!a_0)\right|^2 +\frac{1}{2} |A\!+\!a_0|^4
~\sim~i A^{\ast} \dot{A}  -{\cal H}.\tag{D}$$
After discarding total derivative terms, the modified Hamiltonian density reads
$${\cal H}~=~\underbrace{(k_0^2-\omega_0)}_{=a_0^2}|A\!+\!a_0|^2 
+ ik_0\left\{A^{\ast}_x A - A_x A^{\ast}\right\} 
+ |A_x|^2 - \frac{1}{2} |A\!+\!a_0|^4  $$
$$ ~\stackrel{k_0=0}{=}~\frac{1}{2} |a_0|^4 + |A_x|^2 - \frac{a_0^2}{2} (A\!+\!A^{\ast})^2 + {\cal O}(|A|^3). \tag{E}$$
In the last equality, we specialized to OP's case $k_0=0$. Note that $\omega_0=-a_0^2 <0$ is then negative.
III) Let us mention for completeness that the quadratic term $H_2$ in the modified Hamiltonian 
$$ H~=~\int \! \mathrm{d}x ~{\cal H}~=~H_0 +H_1 +H_2 + {\cal O}(|A|^3) \tag{F}$$ 
reads
$$H_2~=~$$ 
$$ - \frac{1}{2}\iint \! \mathrm{d}x ~\mathrm{d}y ~
\overline{\begin{pmatrix} A(x) & A(x)^{\ast}\end{pmatrix}}
\begin{pmatrix}  (\partial_x^2 + a_0^2) \delta(x\!-\!y) & a_0^2 \delta(x\!-\!y) \cr a_0^2 \delta(x\!-\!y)   & (\partial_x^2 + a_0^2) \delta(x\!-\!y)  \end{pmatrix} 
\begin{pmatrix} A(y) \cr A(y)^{\ast}\end{pmatrix}
A(x).\tag{G}$$ 
IV) The Hamilton's equation reads
$$ i \dot{A}~=~\frac{\delta H}{\delta A^{\ast}}
~\stackrel{(E)}{=}~-A_{xx} -\left\{|A\!+\!a_0|^2-a_0^2 \right\} (A\!+\!a_0)
~=~-A_{xx} - a_0^2(A\!+\!A^{\ast}) + {\cal O}(|A|^2)
. \tag{H} $$ 
Note that because of the $A \leftrightarrow A^{\ast}$ mixing we need to include at least 2 modes:
$$ A ~=~ \sum_{\pm} \epsilon_{\pm} e^{\pm i(kx-\omega t)}. \tag{I}$$
The linearized Hamilton's equation (H) reads
$$ \begin{pmatrix}  \omega -k^2 +a_0^2 & a_0^2 \cr a_0^2 &   -\omega -k^2 +a_0^2  \end{pmatrix}
 \begin{pmatrix} \epsilon_+  \cr  \epsilon_-\end{pmatrix}
~\stackrel{(H)+(I)}{=}~0, \tag{J}$$
leading to OP's equation
$$\omega^2 ~=~  k^2(k^2- 2a_0^2),\tag{K} $$
and OP's criterion 
$$|k|~<~ \sqrt{2}a_0\tag{L} $$ 
for instability.
V) If we insert 
$$ A ~=~   \epsilon \cos(kx)   \tag{M}$$
in the quadratic Hamiltonian
$$ H_2~\stackrel{(E)+(M)}{=}~\epsilon^2 \int \! \mathrm{d}x 
~\left\{ k^2 \sin^2(kx) - 2 a_0^2 \cos^2(kx)\right\}, \tag{N}$$
one may argue that this is unstable if the inequality (L) is satisfied.
A: The sign of your interaction term is wrong. The energy is not bounded from below. You can not minimise the energy to find the most stable state. The energy is minimised by $|A| \to \infty$.
If you change the sign of the non-linear term in your equation you get a similar linearised problem although with the replacement $a_o^2 \to -a_o^2$. Then your frequencies are real and your smallest possible energy is zero.
As I know it this is a model for bosons with repulsive interactions. There, when the density grows, the bosons interact more strongly and repel each other. This, in turn, decreases the density again. In your case you have attractive interactions. If you increase the density, you bring the particles closer to each other and their interaction is facilitated. The density grows even more. You have an instability.
