You might well have asked the same question of any calculation in quantum mechanics: why do different regulators for formally divergent quantities give the same result? The answer is that assuming those regulators don't break important symmetries that affect our measurements (like for instance Lorentz invariance) then regulators only mess with physics in the UV, and leave IR physics the same. If we're interested in low energy physics, then different regulators will all leave the quantities we're interested in untouched, and modify only irrelevant high energy details. This isn't a miracle! See also the discussion in: Which renormalisation techniques are available for 3+1 QED?
In Casimir's original paper, he presented a derivation that shows any "sensible" choice of regulator will give the same result. This is essentially what Terry Tao runs through in his blog post, but his post is quite detailed and maybe a more digestible presentation could be useful! It goes like this:
In a one-dimensional box, the frequencies are quantised to those of plane waves, $\omega_n=\tfrac{\pi}{\ell}n$ where $\ell$ is the size of the box. Let us then define
\begin{align}
E(\ell)=\sum_n\frac{1}{2}\omega_n
\end{align}
the sum of the zero point energies of these modes. Let's now modify this sum with a (mostly) arbitrary regulator,
\begin{align}
E(\ell)=\frac{\pi}{2}\sum_n\frac{n}{\ell}\ f\ \left(\frac{n}{\ell\Lambda}\right)
\end{align}
where we just multiplied each term in the sum by $f(\frac{n}{\ell\Lambda})$. We assume only that this function $f(x)$ dies off faster than $x^{-1}$, and that $f(0)=1$. The first requirement is simply that the regulator only shuts off modes at high energies - the UV frequencies go right through the box. The second requirement is just that the regulator shouldn't affect the spectrum in the IR.
Let's now envisage we have a pair of places separated at a distance $a$, inside a larger box of size $L$, shown below$^1$:
If we leave $L$ fixed, and vary $a$, the force on the walls of our plates at distance $a$ is $-dE/da$. The energy of the $L-a$ side of the box is
\begin{align}
E(L-a)=\frac{\pi}{2}\sum_n\frac{n}{(L-a)}\ f\ \left(\frac{n}{(L-a)\Lambda}\right)
\end{align}
and if we take the continuum limit $L\rightarrow\infty$, we get
\begin{align}
E(L-a)\rightarrow\frac{\pi}{2}(L-a)\Lambda^2\int xf(x) dx
\end{align}
So now adding the discrete sum for between the plates to the continuum sum for the outside of the plates, we get
\begin{align}
E_{total}=E(a)+E(L-a)=\frac{\pi}{2}L\Lambda^2\int x\ f(x) dx+\frac{\pi}{2a}\left(\sum_n\ n\ f\left(\frac{n}{a\Lambda}\right)-\int \ n\ f\left(\frac{n}{a\Lambda}\right) dn\right)
\end{align}
The difference between the sum of a function and its integral is given by the Euler-Maclaurin formula:
\begin{align}
\sum_{n=1}^Nf(n)-\int_0^Nf(n)dn=\frac{f(0)+f(N)}{2}+\frac{f'(0)-f(0)}{12}+...B_j\frac{f^{(j-1)}(0)+f^{(j-1)}(N)}{j!}
\end{align}
where $B_j$ are the Bernoulli numbers.
Okay, now we only care about the pieces of this expression that depend on $a$, since we're going to differentiate with respect to $a$. Isolating these terms,
\begin{align}
E_{total}=-\frac{\pi f(0)}{24a}-\frac{B_4}{4!}\frac{3\pi}{2a^3\Lambda^2}f^{''}(0)+...
\end{align}
where the $...$ terms are suppressed by increasing powers of $\Lambda$. If $\Lambda$ is taken to be large, we throw away everything except the first term; recalling we required $f(0)=1$ we then obtain the Casimir force
\begin{align}
F=-\frac{dE}{da}=-\frac{\pi}{24a^2}
\end{align}
Now, it's often mysterious to people why for e.g. esoteric manipulations like zeta regularisation give the right answer. The above argument demystifies why good regulators are good regulators: you can expand the difference between the actual sum summed up to a cut off and the zeta sum (or whatever sum - heat kernel sum perhaps) using Euler-Maclaurin and show that there's a universal piece which isn't touched by the choice of cut-off. Bad things can only happen if the regulators break symmetries.
You're worried that different geometries or dimensions this doesn't work - but it does! So long as the regulator doesn't break the symmetry of the geometry, you're fine. It's not clear what your $2d$ scalar example you have in mind is, but if it's a Casimir type problem, then the infinite part certainly will not depend on the plate separation, so we're fine. An earlier and interesting answer points out that sometimes if the plates have a strange boundary, the regulator needs to do more than just "subtract the zero point energy" - the bottom line though is you need to mess with the UV physics only in a way that respects the symmetries of the boundary.
$^1$Image (and derivation) stolen shamelessly from Matt Schwartz's QFT textbook.
