We define the norm of the operator as $\left\lVert A \right\rVert = \sup \frac{\left\lVert A\psi \right\rVert}{ \left\lVert A \right\rVert} = \sup \left\lVert A\psi \right\rVert$ for $A ∈ L(H)$. It is said that $||A||$ measures the magnitude of the action of $A$.
What is meant by the action of $A$ and what is sup in this equation?
Also, how can we check if $A$ is bounded or unbounded with this statement?
The sup in the equation is a supremum over all states out of the Hilbert space $\mathcal{H}$. In other words, you pick the state $|\psi\rangle$ out of $\mathcal{H}$ for which the number $\frac{\mid\mid A|\psi\rangle\mid\mid}{\mid\mid |\psi\rangle\mid\mid}$ becomes the largest. It can be shown that the operator norm of $A$ corresponds to the largest eigenvalue of $\sqrt{A^{\dagger}A}$. As an operator acting on $\mathcal{H}$, $A$ maps one state out of it to another, which may be not normalized since you have no guarantee on $A$ being unitary. So you can think of the action of $A$ as a measure for “how much” non-normalized $A|\psi\rangle$ is, given that $|\psi\rangle$ is, i.e. $\langle\psi | \psi \rangle=1$.
An operator is unbounded if its supremum norm is infinite. This follows trivially by negating the definition of a bounded operator in a preBanach space.
"A linear operator $A :D(A) \rightarrow X$ is called bounded, iff
$$\forall \psi\in D(A), \exists k\in \mathbb R, \text{so that} ||A\psi|| \leq k ||\psi||. $$
The maximum value of $k$ over the domain of A is called the supremum norm of A.
