Suppose I have two operators $J^2$ and$J_z$ where they represent the length of angular momentum and its $z$ component respectively.
Sure, it's legal to write new operators like $J^2-J_z^2$ or $J_x+J_z$ but it's not legal to write $J^2-J_z$ because they don't have the same physical dimension.
However, I want to know what prevent them from becoming operators in ket space mathematically.
If $A,\,B$ are operators you can sum, then $\langle\phi|A+B|\psi\rangle$ exists and is equal to $\langle\phi|A|\psi\rangle+\langle\phi|B|\psi\rangle$; so those terms have the same dimension, and hence so do $A,\,B$.
Suppose $|\phi\rangle$ lives in a Hilbert space $H$ over the field $F$, so $a,\,b\in F$ satisfy $a+b+ab\in F$, making $a,\,b$ dimensionless. While a Hilbert space $H'$ of operators from $H$ to $H$ is closed under summation with coefficients $\in F$, these coefficients are dimensionless, so $J_z$ is not required to live in the same Hilbert space as $J^2$.
There’s nothing inherently illegal about something like $\hat A^2+\hat A$. Indeed, functions of an operator $A$ are usually defined by their series expansion, the most common examples being the time evolution $$ U(t)=e^{-it\hat H/\hbar}=1-i\frac{t\hat H}{\hbar} +\frac{1}{2} \left(\frac{i\hat H}{\hbar}\right)^2+\ldots \tag{1} $$ and the rotation about an axis - say $\hat z$: $$ R_z(\alpha)=e^{-i\alpha \hat L_z/\hbar}=1-i\frac{\alpha \hat L_z}{\hbar} +\frac{1}{2}\left(\frac{i\hat L_z}{\hbar}\right)^2 + \ldots \tag{2} $$ Of course, every term in the sums (1) or (2) is dimensionally consistent: in this specific case $t \hat H/\hbar$ and $\alpha\hat L_z/\hbar$ are both dimensionless. They need to be consistent for the same reason that you cannot add something measured in meters with something measured in meters squared. Indeed imagine you work with an eigenbasis of $\hat A$: then there is no physics sense in adding powers of dimensionful eigenvalue, as it would precisely amount to adding quantities having different units.
Maybe you are worried that there are expressions like $j(j+1)$ in the angular momentum algebra, but really the $J$'s have the same dimensions as $\hbar$ and eigenvalues $\hbar j$ where $j$ is an integer or a half integer.
Operators can be added if they have the same dimension or if they are dimensionless. So $J^2/h^2 + J_z/h$ makes sense but $J^2 + J_z$ does not.
