How do we know we have the minimal ladder operators?

When we have ladder operators, like that for angular momentum and the energy levels of the oscillator, how do we know that they produce the minimal increment?

ie, for the harmonic oscillator we have: $$H|n\rangle=\hbar\omega(n+1/2)|n\rangle \implies Ha_+|n\rangle=\hbar\omega(n+3/2)a_+|n\rangle$$

How do we know that the increment caused by $a_+$ is the smallest possible increment, that there is not a $b_+$ such that $$Hb_+|n\rangle=\hbar\omega(n+1/2+\epsilon)b_+|n\rangle$$

At its core, the argument you're presenting only uses the fact that your prospective ladder operator $\hat \ell=\hat a$ has a specific commutation relation with the hamiltonian, $$[\hat H,\hat \ell] = -c\hat \ell \tag{*}$$ for some constant $c$; this is enough to ensure that if $|E\rangle$ is an eigenstate of $\hat H$ with eigenvalue $E$, then $\hat \ell|E\rangle$ must obey $$\hat H\hat \ell |E\rangle =\hat \ell(\hat H-c)|E\rangle =(E-c)\hat \ell |E\rangle, \tag{\star}$$ as in your question. The key issue here, though, is that the relationship $\bf (*)$ is not unique to $\boldsymbol{\hat\ell=\hat a}$, and indeed if $(1)$ is true for $\hat \ell$ then it is also true for ${\hat{\ell}}^2$ by changing $c$ for $2c$, and by induction for $\hat \ell^k$ for any power $k$ with constant $nk$.
Thus, the question is perfectly valid: how do we know that we're not tricking ourselves, and there is in fact some operator $\hat b$ such that $\hat b^2=\hat a$, and which causes increments half as big as those caused by $a$? Indeed, for all that the algebraic method knows as of $(\star)$, there could be some other half of the basis $|\frac12\rangle,|\frac32\rangle,\ldots$, interleaved with the usual Fock states, and $\hat a$ is taking the stairs two at a time.
The answer to that is the behaviour at the edges, and more specifically at the lower bound of the spectrum of $H$. We know that the spectrum is bounded from below, which means that the descending stair in $(\star)$ on $|E\rangle, \hat \ell |E\rangle, \hat \ell^2|E\rangle, \ldots$ needs to terminate at some point, and that can only happen if at some point that descending ladder gets a zero vector, i.e. if for some state $|\psi\rangle$ we have $$\hat \ell |\psi\rangle=0.$$ Now, here's the crucial bit: if $\hat \ell$ is taking on more stairs than it needs to, then the solution to this equation will have a kernel that's bigger than it needs to, and it will involve more than one eigenspace. This is the case with the square of the usual ladder operator, which has $\hat a^2|0\rangle = 0$ and $\hat a^2|1\rangle = 0$.
In contrast, the correct ladder operator $\hat a$ has a well-defined single-(effective-)dimensional kernel, in the sense that if we set $|\psi\rangle$ to $$\hat a |\psi\rangle=0$$ then this corresponds to a single eigenvalue of the hamiltonian, a fact that relies on the structure $\hat H=c\, \hat a^\dagger\hat a + \delta$ of the hamiltonian. Thus, if $\hat a |\psi\rangle=0$, then we also know that $\hat a^\dagger\hat a |\psi\rangle=0$, so that therefore $\hat H |\psi\rangle=\delta|\psi\rangle$, i.e. a single eigenvalue.
This then places the completely strict constraint on the energies $E$ that can ladder-descend down to the ground state to energies of the form $E=\delta + n c$, which completes the proof that no smaller steps are possible.
Take your hypothetical state of energy $n+\frac{1}{2}+\epsilon$ with $\epsilon\in (0,\,1)$. The number operator $\widehat{N}=\frac{\widehat{H}}{\hbar\omega}-\frac{1}{2}$ then has eigenvalue $n+\epsilon$. To the eigenstate we may apply $a_{-}$ repeatedly as long as we don't get an eigenvalue of $0$, since that's the only way for the resulting ket to vanish and not qualify as an eigenvector. And since we never hit that exact eigenvalue, $\widehat{N}$ has $\epsilon-1<0$ as an eigenvalue. But this contradicts $\widehat{N}=a_{-}^\dagger a_{+}$.