The point is that the notation is ambiguous. Interpreted as $\langle \beta \lvert A \rvert \alpha \rangle = \langle \beta , A \alpha \rangle$ (where on the right I have reverted to the unambiguous math notation)$-$i.e., $A$ is understood to act to the right, then everything is fine. But the symmetric "look" of the notation suggests that $A$ can act to the left, but this is only true if $A$ is self-adjoint, because by definition, then, $\langle \beta , A \alpha \rangle = \langle A\beta , \alpha \rangle$, whereas interpreting
$$
\langle \beta \lvert A \rvert \alpha \rangle =
\langle \beta \rvert \left(A \rvert \alpha \rangle \right)\,,
$$
then
$$
\langle \beta \lvert A \rvert \alpha \rangle^* =
\left(
\langle \beta \rvert \left(A \rvert \alpha \rangle \right)
\right)^*
=
\left(A \rvert \alpha \rangle \right)^{\dagger}
\lvert \beta \rangle
=\langle \alpha \rvert A^{\dagger}\rvert \beta\rangle\,,
$$
where now we can again interpret this as matrix multiplication, i.e., $A^{\dagger}$ acts to the right. One can see how this makes a difference. For more details, see this answer and the answers in the duplicate question.