In Holstein-Primakoff and Dyson-Maleev representation, spin operators are represented by bosonic operators. Roughly speaking, a state with $S^z=S-m$ corresponds to a state containing $m$ bosons. In the Dyson-Maleev representation, $(S^+)^\dagger\neq S^-$, so we have a non-unitary transformation.
Are eigenvalues and matrix elements invariant under such a transformation?
In matrix form, what exactly are the two transformations that will give rise to the two representations, say for spin $S=3/2$?
I) We are given an angular momentum operator $\vec{S}$ in an (unitary, finite-dimensional, irreducible) spin $s$-representation
$$ \vec{S}^2~=~s(s+1){\bf 1}, \qquad s\in \frac{1}{2}\mathbb{N}_0, \tag{1} $$
$$\begin{align} [S_i,S_j]~=~&i\sum_{k=1}^3\epsilon_{ijk} S_k, \qquad i,j,k\in\{x,y,z\}, \cr S_i^{\dagger}~=~& S_i,\end{align}\tag{2} $$
or in terms of raising and lowering ladders operators
$$ S_{\pm}~:=~S_x\pm i S_y, \qquad S_{\pm}^{\dagger}~=~ S_{\mp}, \tag{3} $$ $$ [S_z, S_{\pm}]~=~\pm S_{\pm}, \qquad [S_+,S_-]~=~2S_z. \tag{4} $$
Here we have put the reduced Planck constant $\hbar=1$.
II) The Heisenberg algebra in terms of annihilation operator $a_-\equiv a$ and creation operator $a_+\equiv a^{\dagger}$ reads
$$[a_-, a_+]~\equiv~[a,a^{\dagger}]~=~{\bf 1}, \tag{5}$$
$$ a_{\pm}^{\dagger}~=~ a_{\mp}.\tag{6}$$
The number operator is
$$n~:=~a_+a_-~\equiv~ a^{\dagger}a.\tag{7}$$
One has
$$ [n,a_{\pm}]~=~\pm a_{\pm}, \qquad f(n)a_{\pm}~=~a_{\pm}f(n\pm 1), \tag{8}$$
where $f$ is an arbitrary function.
III) The Holstein-Primakoff unitary realization of the spin $s$-irrep is given as
$$ S_+~=~ a_+h(n)~=~ h(n-1)a_+,\tag{9} $$ $$ S_-~=~ h(n)a_-~=~ a_-h(n-1),\tag{10} $$ $$ S_z~=~n-s, \tag{11}$$
where
$$ h(n)~:=~\sqrt{2s-n}~=~\sqrt{2s} \sqrt{1-\frac{n}{2s}}.\tag{12}$$
It is straightforward to check that eqs. (9-11) yield the Lie algebra (4).
IV) The Dyson-Maleev non-unitary realization of the spin $s$-irrep is of the form
$$ J_+~=~ S_+g(n)~=~g(n-1)S_+~=~\sqrt{2s}a_+g(n)^2, \tag{13}$$ $$ J_-~=~ g(n)^{-1}S_-~=~S_-g(n-1)^{-1}~=~\sqrt{2s}a_-,\tag{14} $$ $$ J_z~=~S_z, \tag{15}$$
where
$$ g(n)~:=~ \sqrt{1-\frac{n}{2s}}.\tag{16}$$
It is straightforward to check that eqs. (13-15) yield the Lie algebra (4) even without using the explicit form (16).
V) Let us define new creation and annihilation operators
$$ A_+~:=~ a_+g(n)~=~g(n-1)a_+, \tag{17}$$ $$ A_-~=~ g(n)^{-1}a_-~=~a_-g(n-1)^{-1}, \tag{18}$$
with the same number operator
$$ N~:=~A_+A_-~=~a_+a_-~=~n,\tag{19}$$
and the same Heisenberg algebra (5).
VI) Note that the new creation and annihilation operators $A_{\pm}$ are not each others $\dagger$-conjugate a la eq. (6). But one can introduce another Hermitian involution $\ddagger$ as
$$ a_-^{\ddagger}~:=~ a_+ g(n)^2~=~g(n-1)^2 a_+ \tag{20}$$ $$ a_+^{\ddagger}~:=~ g(n)^{-2}a_-~=~a_- g(n-1)^{-2},\tag{21} $$ $$ n^{\ddagger}~=~ n, \qquad (FG)^{\ddagger}~=~G^{\ddagger}F^{\ddagger}, \qquad F^{\ddagger\ddagger}~=~F, \tag{22} $$
where $F$ and $G$ are two arbitrary operators in the universal enveloping algebra. With the new Hermitian involution $\ddagger$, the creation and annihilation operators $A_{\pm}$ are each others $\ddagger$-conjugate
$$ A_{\pm}^{\ddagger}~=~ A_{\mp}.\tag{23}$$
VII) Conclusion. The Dyson-Maleev realization (13-15) built with the annihilation and creation operators $a_{\pm}$ can be viewed as a Holstein-Primakoff realization built with the annihilation and creation operators $A_{\pm}$. Moreover, the Dyson-Maleev realization (13-15) is unitary wrt. the $\ddagger$-conjugation but not wrt. the original $\dagger$-conjugation, cf. eq. (23). We believe that these observations essentially answer OP's original questions(v1).
