Spontaneous symmetry breaking by two scalar multiplets Consider a theory with two multiplets of real scalar fields $\phi_i$ and $\epsilon_i$, where $i$ runs
from $1$ to $N$. The Lagrangian is given by:  $$\mathcal L = \frac{1}{2} (\partial_{\mu} \phi_i) (\partial^{\mu} \phi_i) + \frac{1}{2} (\partial_{\mu} \epsilon_i) (\partial^{\mu} \epsilon_i) − \frac{m^2}{2}[\phi_i \phi_i+ \epsilon_i \epsilon_i] − \frac{g}{8}[(\phi_i \phi_i)(\phi_j \phi_j ) + (\epsilon_i \epsilon_i)(\epsilon_j \epsilon_j)] − \frac{λ}{2}(\phi_i \epsilon_i)(\phi_j \epsilon_j ),$$ where $m^2 < 0, g > 0 $ and  $\lambda > −g/2.$  Summation over repeated indices is implied. 
Is the following accurate? The lagrangian can be written in vector notation and we can see it is then invariant under a simultaneous transformation of $\phi$ and $\epsilon$ such that $\epsilon \rightarrow R_{ij}\epsilon_j$ and $\phi_i \rightarrow R_{ij} \phi_j$ if $R_{ik} R_{ij} = \delta_{kj}$ The symmetry group is then $O(N) \otimes O(N)$ with generators $T_a^{O(N) \otimes O(N)} = T_a^{O(N)} \otimes \text{Id}_{N \times N} + \text{Id}_{N \times N} \otimes T_a^{O(N)}$ so there are $\text{dim}O(N)$ number of generators. 
The vacua of the theory can be found as the minimum of the potential $$V(\phi, \epsilon) = \frac{m^2}{2} ( \vec \phi^T \vec \phi + \vec \epsilon^T \vec \epsilon) + \frac{g}{8} ((\vec \phi^T \vec \phi)^2 + (\vec \epsilon^T \epsilon)^2) + \frac{\lambda}{2} (\vec \phi^T \vec \epsilon)^2$$ I am a bit confused here - to find the vacua I could demand $$\frac{\partial V}{\partial \phi^T \phi} = \frac{\partial V}{\partial \epsilon^T \epsilon} \overset{!}{=} 0$$ but what happens to the term proportional to $\lambda$? 
 A: No, your "following" is not accurate. You wrote a SB Lagrangean invariant under O(N)×O(N)   (⊂ O(2N)), except for the λ term, which is only invariant under its diagonal subgroup O(N), instead. 
The N  φs and the N εs fit into  2 N - vector, $(\vec{\phi},\vec{\epsilon})$,  so the symmetry starts as O(2N) but the g term is only invariant under its O(N)×O(N) subgroup.The indices of the  φs and the εs need not know about each other, except for the  λ term contracting them together: think of synchronized swimming. The λ term is thus invariant under O(N), not O(N)×O(N).
To tease a stay against confusion, pick N =3, so six real scalar fields. Display the O(6) -invariant part, which the g term restricts to O(3)×O(3), and finally the λ term to  O(3). Now study the further SSB built in---this is a popular problem I sometimes assign. 
The key is always in the  2N×2N Goldstone mass matrix $\langle \delta_i \delta_j V \rangle $ and, in particular, its kernel consisting of the null eigenvectors.
Now simplify the algebra by defining 
$$
\frac{-2m^2}{g} \equiv v^2 , 
$$ 
so that the positive overall scale of the potential, g/8, may be safely dropped. 
Further shifting the potential by the innocuous constant terms to convert it to a sum of squares, obtain
$$
V=(\vec{\phi}^2 - v^2)^2+(\vec{\epsilon}^2 -v^2)^2+ \frac{4\lambda}{g} (\vec{\phi}\cdot\vec{\epsilon})^2. 
$$
It is evident that, for λ =0, this is the two standard Goldstone hyper-sombrero potentials superposed, so their minima are at $\langle \vec{\phi}^2 \rangle= \langle \vec{\epsilon}^2 \rangle= v^2 $. 
Naturally, $\langle \vec{\phi} \rangle$ may pick any orientation in the bottom of its sombrero hypersurface, and  $\langle \vec{\epsilon} \rangle$ an arbirary, in general different one, in its own; so the group is SSBroken down to  O(N-1)×O(N-1). Your  2N×2N Goldstone mass matrix will have 2(N-1) null vectors, so goldstons. For N = 3, you get 4 goldstons.

For λ ≠0, however, the symmetry is only O(N), as stated. 
For  λ >0, it is manifest from the potential sum of positive squares form that 
$$
\langle \vec{\phi}^2 \rangle=v^2;\qquad \langle \vec{\epsilon}^2 \rangle=v^2; \qquad\langle \vec{\phi} \cdot \vec{\epsilon}\rangle =0~.
$$
That is to say, the vacuum orientations of $\vec{\phi}$ and $\vec{\epsilon}$ must be orthogonal. W.l.o.g, take  $\langle \phi_1 \rangle=v =\langle \epsilon_2 \rangle$.
The surviving invariance is then only O(N-2), and the goldstons 2N-3, so 3 for N =3 -- can you see it in your Goldstone matrix? (Hint: Confirm only $\phi_1, \epsilon_2$ and $\phi_2+\epsilon_1$ are massive, for all N.)

The plot thickens for 0> λ > -g/2. Now the λ term is compelled to grow, not shrink, and, magnitudes being equal (dictated by the other terms), it presses to align $\langle \vec{\phi} \rangle$ with $\langle \vec{\epsilon} \rangle$, so then, 
$\langle \vec{\phi} \rangle=\langle \vec{\epsilon} \rangle$.
Specifically, consider the first variation that apparently stymied you in the first place, (recall that stationarity is required for every component of the fields, not just the magnitudes of their group vectors!),
$$\langle \frac{\delta V}{\delta\vec{\phi}}  \rangle =
\langle \frac{\delta V}{\delta\vec{\epsilon}}  \rangle =0, $$ 
and thus 
$$
0=-v^2\vec{\phi}+(\vec{\phi}\cdot\vec{\phi})\vec{\phi} +\frac{2\lambda}{g} (\vec{\phi}\cdot\vec{\epsilon})\vec{\epsilon} \\
0=-v^2\vec{\epsilon}+(\vec{\epsilon}\cdot\vec{\epsilon})\vec{\epsilon}+ \frac{2\lambda}{g}  (\vec{\epsilon}\cdot \vec{\phi})\vec{\phi}.
$$
It is manifest that $ \vec{\epsilon}\propto \vec{\phi}$, so define $\vec{\epsilon}\equiv a  \vec{\phi}$ for real nonvanishing a.
The extremizing conditions then reduce to just
$$
v^2=\langle \vec{\phi}\cdot \vec{\phi} \rangle \left(1+ \frac{2\lambda}{g} a^2\right); \qquad 
v^2=\langle \vec{\phi}\cdot \vec{\phi} \rangle \left(a^2+ \frac{2\lambda}{g}\right),
$$
so, then, $a^2=1$, recalling the condition $\frac{2\lambda}{g}+1>0$.
Take a = 1, perfect alignment of $\langle \vec{\phi} \rangle$ with $\langle \vec{\epsilon} \rangle$, and $\langle \vec{\phi}^2 \rangle=\langle \vec{\epsilon}^2 \rangle=v^2/(1+2\lambda/g)$. You then have an unbroken residual subgroup O(N-1),
so only N-1 goldstons now, 2 for N =3. Observe the v.e.vs increase without bound with decreasing negative λ.
Given this alignment, you might go back to the potential and monitor how λ <- g/2, beyond the pale, would overwhelm the terms in the Sombrero potentials and flip them, destabilizing them, thus preventing SSB, among other calamities.
