The answer-in-brief is that this is not a good way of thinking of the Casimir force. A better way is in terms of semiclassical paths and if the scattering amplitudes differ by a sign (the Casimir force is dominated by photons who intersect both plates). This is equivalent to mode summation (and in fact, the Lifshitz equation just uses reflection matrices as a way to calculating all modes in the system).
Before I go into detail on why reflection can given an intuition for why Casimir energy can be repulsive, let me just say three equivalent methods of calculating the Casimir energy which each superficially sound like different physics:
- Mode summation, $E_c = \frac12 \sum_n \omega_n$ by just considering the actual zero point energy of every mode in the system.
- Reflection of photons and tracing all paths a photon can take. This is characterized by the Lifshitz equation (for parallel plates at distance $L$):
$$ E_c \sim \int d^2 q d \omega \, \mathrm{tr}\, \mathrm{ln}[1 - R_{A} R_{B}e^{-q_z L}],$$
where $R_{A}$ and $R_{B}$ is the reflection matrix for plate $A$ and $B$ respectively, and $q$ and $\omega$ are the momentum and frequency of the photons being exchanged between plates.
- Summation of vacuum diagrams from quantum field theory (see for instance the supplement to arXiv:1312.6754), where you keep only diagrams that intersect both disjoint objects (all others will not contribute to a force):

In detail:
First, consider the actual energy between the plates. Believe it or not, it's not terribly changed. While formally divergent, one can think of it as the first moment of the density of states
$$E_c = \frac12 \sum_n \omega_n = \frac12\int E \rho(E) dE, $$
where $\rho(E) = \sum_n \delta(E - E_n)$. Balian and Bloch (paywall warning) considered the density of states for a free field and finite domain and found that very roughly:
$$ \rho(E) \approx Volume*\sqrt{E} \pm Surface + \frac1{\sqrt{E}}*Curvature + \cdots$$
at large energy. The first term is just what you get in free space:
$$ \frac{\rho(E)}{V} = \int \frac{d^3k}{(2\pi)^3} \delta(E - k^2) = const. * \sqrt{E} $$
Therefore, the energy between the plates is not significantly modified between the plates (and regarding the Casimir force, the surface term and curvature term for rigid surfaces will not change as the plates move).
These volume, surface, and curvature terms are often discarded as not contributing to a Casimir force (any kind of regularization usually is just discarding these terms which are formally divergent).
Thus, the important difference comes by going to higher asymptotics in the density of states. At this point, semiclassical paths are what matters for the calculation and they can either increase or decrease the density of states (and usually the length of paths enters as an inverse proportionality). For a careful discussion of the semiclassical paths and their role in repulsion, see my paper: arXiv:0703248.
But really, it comes down to the amplitude of a photon that bounces off of both plates. If that final amplitude is positive (i.e. most cases, especially when the materials are similar) you get an attractive force and if it is negative you get a repulsive force. This is roughly still the idea for 3D with the electromagnetic field.
For instance, another way of finding the Casimir energy between two plates A & B is roughly (neglecting many higher order terms):

Notice that if one of the plates give differing amplitudes for the photons (squiggly lines), the Casimir energy will have a different sign from when they are the same! And if you now return to the Lifshitz equation you can see that if $R_A R_B < 0$ you can get repulsion (most materials have $R_A R_B>0$). This is the same principle.
One detail that is important to remember: The sign of the Casimir energy just happens to correspond to the sign of the Casimir force in a lot of situations, but you need to be careful that when you consider repulsion you are considering $F_c = -\frac{d E_c}{d L}$ where $L$ is the length between plates. This derivative can have a different sign than the Casimir energy itself!