The sup in the equation is a supremum over all states out of the Hilbert space $\mathcal{H}$. In other words, you pick the state $|\psi\rangle$ out of $\mathcal{H}$ for which the number $\frac{\mid\mid A|\psi\rangle\mid\mid}{\mid\mid |\psi\rangle\mid\mid}$ becomes the largest. It can be shown that the operator norm of $A$ corresponds to the largest eigenvalueseigenvalue of $\sqrt{A^{\dagger}A}$. As an operator acting on $\mathcal{H}$, $A$ maps one state out of it to another, which may be not normalized since you have no guarantee on $A$ being unitary. So you can think of the action of $A$ as a measure for “how much” non-normalized $A|\psi\rangle$ is, given that $|\psi\rangle$ is, i.e. $\langle\psi | \psi \rangle=1$.