I'm going to add a long comment in the form of an answer for future learners spelling out a bit @RogerV. excellent answer.
We start with the energy perturbation series up to second-order $$E_n(\lambda) = E_n^{(0)} + 0 + \lambda^2 E_n^{(2)} + \mathcal{O}(\lambda^3),$$
where the first order correction it's zero.
Next, we work on the second-order corrections due to the $a_i^\dagger a_j^\dagger$ terms
$$
E_n^{(2)} = \sum_{n\neq m} \frac{|\langle n_0, n_1, \cdots | a^\dagger_i a_j^\dagger | m_0, m_1, \cdots \rangle|^2}{E_{m_0, m_1, \cdots}-E_{ n_0, n_1, \cdots}}
= \sum_{n\neq m} \frac{(m_j+1)(m_i+1)|\langle n_0, n_1, \cdots | m_0, m_1, \cdots, m_i + 1, \cdots, m_j + 1, \cdots \rangle|^2}{E_{m_0, m_1, \cdots}-E_{ n_0, n_1, \cdots}}
$$
we proceed with the algebra
$$
E_n^{(2)} = \sum_{n\neq m} \frac{(m_j+1)(m_i+1)\delta_{n_0,m_0}\cdots\delta_{n_i, m_i + 1} \delta_{n_j, m_j + 1}\cdots}{(E_{m_0, m_1, \cdots}-E_{ n_0, n_1, \cdots})} = \frac{n_jn_i}{E_{n_0, n_1, \cdots, n_i,\cdots, n_j, \cdots}-E_{n_0, n_1, \cdots, n_i-1,\cdots, n_j-1, \cdots}}
$$
and we spot that
$$
E_n^{(2)} = \frac{n_jn_i}{n_0\omega_0+\cdots-n_0\omega_0-\cdots-(n_i-1)\omega_i-\cdots} = \frac{n_jn_i}{\omega_i + \omega_j}.
$$
We conclude that, up to the second-order, the corrections due to the $a_i^\dagger a_j^\dagger$ terms are.
$$
E_n(\lambda) = \sum_j \omega_j n_j + \sum_{ij}\frac{\lambda^2}{\omega_i + \omega_j}n_jn_i + \mathcal{O}(\lambda^3)
$$
On the other hand, it is now easy to see that the second-order corrections due to the $a_i^\dagger a_j$ terms are
$$
E_n^{(2)} = \sum_{n\neq m} \frac{|\langle n_0, n_1, \cdots | a^\dagger_i a_j| m_0, m_1, \cdots \rangle|^2}{E_{m_0, m_1, \cdots}-E_{ n_0, n_1, \cdots}}
= \sum_{n\neq m} \frac{(m_i+1)(m_j)|\langle n_0, n_1, \cdots | m_0, m_1, \cdots, m_i + 1, \cdots, m_j - 1, \cdots \rangle|^2}{E_{m_0, m_1, \cdots}-E_{ n_0, n_1, \cdots}}
$$
we proceed with the algebra
$$
E_n^{(2)} = \sum_{n\neq m} \frac{(m_i+1)(m_j)\delta_{n_0,m_0}\cdots\delta_{n_i, m_i + 1} \delta_{n_j, m_j - 1}\cdots}{(E_{m_0, m_1, \cdots}-E_{ n_0, n_1, \cdots})} = \frac{n_i(n_j+1)}{E_{n_0, n_1, \cdots, n_i,\cdots, n_j, \cdots}-E_{n_0, n_1, \cdots, n_i+1,\cdots, n_j-1, \cdots}}
$$
and we spot that
$$
E_n^{(2)} = \frac{n_j(n_i+1)}{n_0\omega_0+\cdots-n_0\omega_0-\cdots-(n_j+1)\omega_j-\cdots} = \frac{n_j(n_i+1)}{\omega_i-\omega_j}.
$$
We conclude that, up to the second-order, the corrections due to the $a_i^\dagger a_j$ terms are
$$
E_n(\lambda) = \sum_j \omega_j n_j + \sum_{ij}\frac{\lambda^2}{\omega_i - \omega_j}n_j(n_i+1) + \mathcal{O}(\lambda^3).
$$
As Roger V. pointed out, when $\omega \gg \lambda$ the corrections due to the $a_i^\dagger a_j$ and $a_i a_j^\dagger$ terms are large compared to the corrections due to the $a_i^\dagger a_j^\dagger$ and $a_i a_j$ terms and can be ignored.