Consider your potential as a quadratic form, $$ V= (\varphi,\kappa ) \begin{pmatrix}\omega^{2} & -A \\ -A & \omega^{2}\end{pmatrix} \begin{pmatrix}\varphi\\ \kappa\end{pmatrix} . $$
For $|A|< \omega^2 $ , it is concave, so stable, with the minimum at the origin of the fields. Oscillations occur with the two eigenfrequencies you found in the directions (1,1) and (1,-1).
For $|A| = \omega^2 $, it amounts to an infinite trough, with the flat direction along (1,1). (Such trough potentials afflict SUGRA).
For $|A| > \omega^2 $, it is an unstable saddle; since the energy is unbounded below, the system will transition down the saddle slope, again along the (1,1) direction, "forever", just like unstable 2-state molecules. So your state is readily $\varphi+\kappa$.
No oscillations, and soon your hamiltonian is irrelevant/inapplicable.