Are operators in quantum mechanics linear transformations? Observables in quantum mechanics correspond to self-adjoint linear operators. If $\psi$ is an eigenvector of $\hat A$, then $\hat A\psi=\alpha\psi$ where $\alpha$ is the eigenvalue of $\psi$. Otherwise, if the initial state $\psi$ is not one of the eigenvectors of $\hat A$, then the terminal state will be one of $\hat A$'s orthonormal eigenbasis with probability determined by the projection. 


*

*Then what is the physical meaning of $\hat A\psi$ in this situation? 

*Shouldn't observables be more appropriately described by maps from the Hilbert space to the set of probability distributions on that space?
 A: You are maybe making confusion between the action of an observable (operator), and the measurement process. In particular:


*

*$A\psi$ is simply a vector of the Hilbert space. In my opinion it has not much sense of talking about "initial" and "terminal" state because you are not looking at a dynamical situation. If you want to know the average value of an observable in your state, you look at $\langle \psi, A\psi\rangle$. Obviously if $\psi$ is a normalized eigenvector corresponding to $\alpha$, you will obtain, as expected, $\alpha$ as the average value. If else, you will obtain some value that can be interpreted (if $A$ admits an eigenbasis) as the average
$$\sum_{\alpha\in\mathbb{N}}p_n \alpha_n\; , \; p_n=\lvert\langle\psi,\phi_n\rangle\rvert^2$$
and $p_n$ represents the probability of "finding the state $\psi$ in the eigensubspace corresponding to $\alpha_n$" (I'm assuming all multiplicities to be 1 for simplicity). The spectral theorem may provide you with a better insight on the behavior of self-adjoint operators in Hilbert spaces.

*Observables are defined in this way because the quantization, that is actually a mystery (quoting E.Nelson), works very well and it postulates that the correspondant of classical variables (position and momentum) are linear operators on a Hilbert space, and thus their functions (the classical observables) would become linear operators too. Changing the meaning of observables would require a radical change in the meaning of quantization, and thus QM itself.
[Advanced content, just for whoever is interested: Actually, maps of states (not observables) to probability distributions emerge in the context of quantum systems, but in a rather different setting: that of mean field or classical approximation. Given a quantum system with many particles, it is possible to describe effectively the system in the one-particle space (also called phase space, because the exact same procedure applies to the semiclassical limit of quantum theories) in the limit when the number of degrees of freedom is infinite (or $\hbar\to 0$). Well, in that situation a state of the full quantum system is mapped, in the limit, to a probability distribution on the one-particle space (phase space for classical limit).]
