One of the postulates of quantum mechanics is that every physical observable corresponds to a Hermitian operator $H$, that the possible outcomes of the measurements are eigenvalues of the operator, and that after a measurement the state collapses into the eigenfunction corresponding to the observed eigenvalue.

Some authors state that this is the reason quantum mechanics is "built on" linear algebra. However from a linear algebra standpoint this postulate seems strange. In linear algebra operators are linear transformations that take vectors to vectors; hence I would expect an operator also be a linear transformation, physically realised by a black box that takes in a particle in a state $\psi$ and emits the particle in the transformed state $H \psi$.

In fact, if I look at the creation operator (from say an electron in a harmonic oscillator potential), this clearly can't correspond to an observable because it's not hermitian. I can imagine the black box to be implemented by a machine that fires a photon into the electron. I can also imagine operators that for example rotate the angular momentum of an electron without measuring it. Can I create a physical apparatus that "implements" the position operator (in position space, it takes a state $\psi(x)$ and returns $x \psi(x)$)? Why does QM use operators in these two different senses and how do I distinguish them?