Ladder operators are usually constructed to form a Lie algebra (we want them to have specific conmutation relations). The mathematical basis is weight theory.
The important thing of Lie algebras is that they are a vector space and their elements, which are called generators obbey this conmutation rule: $$[X_i,X_j]=f_{ijk}X_k$$
Where we have used the summation convention. $f_{ijk}$ are just constants, so we call them structure constants.
In our case generators will be just matrices.
In general, we will have n number of generators, which will form an algebra. There will be m simultaneously diaganolizable generators (i.e. they conmute with each other). These generators are called Cartan generators and they form the Cartan subalgebra. We will denote them by $H^i$ and the non Cartan generators by $E^i$.
Each eigenvector asociated to the Cartan generators is called a weight vector, $|t_i\rangle$. Their components $t_i$ are called weights. Weight vectors will correspond to physical states.
A Cartan generator will act on a weight vector as:
$$H^i|t_j\rangle =t^i_j|t_j\rangle$$
At this point I should explain roots, but we shall just skip them.
Now, here is when ladder operators come into play. When a non Cartan generator acts on a state (weight vector) the new eigenvalue will be shifted by $\pm e_j^i+t_k^i$ . When the value is raised we denote the generator by $E^j$ and when its lowered $E^{-j}$. We take that they are the hermitian conjugates of each other.
Then, it possible to prove that $[H^i, E^j]=e^i_jE^j$ and $[E^j,E^{-j}]=e^k_jH^k$. These conmutation relations are very important and they will be used in the angular momentum and harmonic oscillator case.
So we are done, we just need to identify our Cartan and non Cartan generators. Then, the non Cartan generators will move us around the possible states.
Angular momentum
We have that $J^1,J^2,J^3$ are the generators of SU(2). We choose one of this generators to be diagonal one, typically it's $J_3$ (this is the Cartan generator). Then, each state $|j,m\rangle$ is labeled by the eigenvalues of $J_3$, which we'll identify as the angular momentum $m$ and the maximum angular momentum is $j$.
Since $J^1,J^2$ don't satisfy $[J^3,J^i]=\alpha J^i$ nor $[J^i,J^{-i}]=\alpha J^3$, we have to take linear combinations of them. We could show, solving a linear system, that this combination is:
$$N^\pm=\frac{1}{\sqrt{2}}(J_1\mp J_2)$$
These operators will change the value of the angular momentum. We can check that they satisfy the conmutation rules.
$$[J^3,J^\pm]=\pm J^\pm$$
$$[J^+,J^-]=J^3$$
Harmonic oscillator
(I'm a bit confused with SU(1,1) algebras and that stuff, so someone else should explain it)
In this case the Cartan generators are two, the identity $\mathbb{I}$ and the Hamiltonian $H$ (I think that the Hamiltonian could be interchanged by the number operator $N=a^\dagger a$). We also know from QM that $[x,p]=i$ ($h=1$). As in the previous case, we take linear combinations to form the ladder operators. We obtain:
$$[H,\hat{a}]=-\hat{a}$$
$$[H,\hat{a}^\dagger]=\hat{a}^\dagger$$
$$[\hat{a},\hat{a}^\dagger]=\mathbb{I}$$
$$[\hat{a},\hat{a}]=0$$
$$[\hat{a}^\dagger,\hat{a}^\dagger]=0$$
The harmonic oscillator can be extended in QFT to study bosons and fermions.
If you want more information about the math of ladder operators in angular momentum you should have a look at Georgi's book. For the harmonic oscillator there is not so much information, I like this notes: https://web.archive.org/web/20140804024331/http://www.math.columbia.edu/%7Ewoit/QM/old-fermions-clifford.pdf .