Calculation of expectation of number of particle using Bogolubov transformation we have two bases connected by Bogoljubov transformations. In the first basis, creation and annihilation operators are ($a,a^\dagger$) and in the second basis ($a',a'^\dagger$). They are connected by
\begin{equation}
a=\alpha a'-\beta^*a'^\dagger
\end{equation}
\begin{equation}
a^\dagger= -\beta a'+\alpha a'^\dagger
\end{equation}
We can prove the first vacuum is connected with the second by following
\begin{equation}
|0'\rangle=Ae^{\frac{\beta^*}{\alpha}a^\dagger a^\dagger} |0\rangle
\end{equation}
Where $A$ is normalization.
So, particle detected with respect to the second frame
\begin{equation}
\langle 0'|a^\dagger a|0'\rangle=\langle0|A^2e^{({\frac{\beta^*}{\alpha})^\dagger} a a} a^\dagger a e^{\frac{\beta^*}{\alpha}a^\dagger a^\dagger} |0\rangle
\end{equation}

How to calculate this?

 A: You did not say it explicitly, but I'm going to assume you're dealing with a boson, ie you have the CCR:
$$
[a,a^\dagger]=1
$$
I'm also going to assume that the lack of conjugation for $\alpha$ isn't a mistake, and that you're assuming $\alpha$ to be real.
By definition, $|0'\rangle$ is the vacuum of the second basis, so it is better to calculate in the second basis. You get:
$$
\langle 0' |a^\dagger a |0'\rangle = \langle 0' |(-\beta a'+\alpha a'^\dagger)(\alpha a'-\beta^* a'^\dagger) |0'\rangle \\
= |\beta|^2
$$
You could also prove it directly, but it is longer. It amounts to looking at the proof of the connection between $|0\rangle,|0'\rangle$. More generally, $a,a'$ are related by a conjugation by a unitary transformation that you can calculate. It's actually the squeeze operator:
$$
S = e^{\frac{1}{2}(z^*a^2-za^\dagger{}^2)} \\
a' = S^\dagger aS
$$
with $z = re^{i\phi}$ and $\alpha = \cosh r$, $-\beta^* = e^{i\phi}\sinh r$. This is how you obtain your formula for the the new ground state:
$$
|0'\rangle = S^\dagger|0\rangle
$$
which is the same formula you gave above. The benefit of complicating your formula is that now you know how $S$ commutes with $a$ so you are back to the first proof.
Hope this helps.
