# Intuitive meaning of Hilbert Space formalism

I am totally confused about the Hilbert Space formalism of Quantum Mechanics. Can somebody please elaborate on the following points:

1. The observables are given by self-adjoint operators on the Hilbert Space.

2. Gelfand-Naimark Theorem implies a duality between states and observables

3. What's the significance of spectral decomposition theorem in this context?

4. What do the Hilbert Space itself corresponds to and why are states given as functionals on the Hilbert space.

I need a real picture of this. I posted in Math.SE but got no answer. So I am posting it here.

-

UPDATE: the insights and conceptual meanings advocated below in this answer are detailed and technically developed, for example, in these wonderful articles:

For intuitions and insights on the meaning of the formalism of quantum mechanics, I eagerly recommend you read carefully the following wonderful reference books (especially Feynman on intuition and examples, Isham on the meaning of mathematical foundations, and Strocchi or Blank et al. on the $C^*$-algebras approach):

As Feynman said (something similar): "If you think you understand quantum mechanics then you do not understand it at all". The whole issue of understanding its Hilbert space formalism, aside from the interpretation of the physical theory itself, can be dealt with more easily (in fact, that is what most physicists do: understand the mathematical formalism with "naïve" empirical intuitions of its meaning so that the theory is predictive and useful, but the issue of its real ontology and epistemology is not at all settled yet). The best approach to grasp the quantum formalism may be to give a parallel interpretation of its classical mechanics analogue, so I will try to elaborate this a bit regarding your different points you mention. Note that quantum mechanics and its postulates can be formulated in different equivalent forms: Dirac's bra-ket formulation of Schrödinger's picture (wave mechanics), Heisenberg's operator formulation (matrix mechanics) and the density operator formulation of states (in Heisenberg's picture). Since your different points are very intertwined, I will try to explain a little bit of everything all at once.

In classical mechanics one measures empirical quantities like the position and speed (so linear momentum) of bodies and idealized particles, thus defining a configuration space and phase space of all physical possible states. Any other observable property must be a function of the system intrinsic parameters (usually constants like rest mass, charge...) and those dynamical variables, so the algebra of classical observables (like Energy) is the commutative ring of typical functions on phase space. Given a measurement one constrains the system to a localized region of phase space within the instrumental precision available, thus obtaining the inicial conditions of the system of interests (where the rest of the universe is usually either ignored or put into an effective/statistical external action over the relevant degrees of freedom). After much experimental observation, physicists obtained the "dynamic laws of classical mechanics" by which given that initial observed state, the system shall evolve with respect to an external clock variable, so at a later time its new observed state can be predicted within the restrictions of precision and chaos theory.

Quantum mechanics is the experimental realization that at the microscopic level, the degrees of freedom of any system behave differently. Classical observables, like position and linear momentum arise as a large scale statistical result of their quantum mechanical counterparts. Concretely, phase space is not commutative (Heisenberg's uncertainty bound) so classical position and momentum define a minimum "quantum chunk" of phase space. Thus, as functions of these noncommutative basic observables, the rest of quantum observables (with classical counterpart) must come from generically noncommuting operators, and these act naturally on Hilbert spaces.

Points 1 & 3 have really the same justification. One assumes that any real experiment, i.e. any empirical observation made by any instrument and read by any sentient being, measures real numbers: position is located by distances to reference systems, timing is kept track with periodic movements also so speed and then momentum all reduce to movement measurements in the end (masses are measured for example with the distance of stretching of a spring by which an object is hung). So observables must give real values upon measurement, which because of precision errors are actually approximated by rationals, or even computable reals, in practice. Now, one can think of an observable as the set of possible values of it, i.e. the set of different possible states of a measurable property. If our quantum mechanical observables must be in general noncommutative operators and be totally specified by a spectrum of real values, then the natural choice is to consider self-adjoint operators since operators have a real eigenvalue spectrum in a suitable basis if and only if they are self-adjoint. So quantum mechanics is "just" the transition to operators which are not simultaneously diagonal in the same basis. This is the significance of the spectral decomposition theorem for the whole formalism: a quantum observable is just the set of its possible classical empirical values codified as an operator by a spectral decomposition with these values as eigenvalues. Thus, all the quantum observables must be self-adjoint operators, with the quantumness manifesting itself by the general noncommutativity of them. Self-adjoint operators which are diagonalizable in the same eigenbasis are said to be "compatible", and they are so if and only if they commute with each other (that is why commutators $[A,B]$ play such an important role in the formalism). The fact of existing noncommuting sets of operators, like position and linear momentum, was called "complementarity" by Bohr.

Point 2 & 4 is just the mathematical realization of all the previous discussion. Since classical observables form a commutative $C^*$-algebra, and quantum observables a noncommutative $C^*$-algebra, by the Gel'fand-Naimark theorem any of the latter is isometrically isomorphic to an algebra of (bounded) linear operators in some Hilbert space. This is a practical realization of the abstract concept of observables as being sets of values and having general algebraic relations between them, i.e. a kind of calculational representation. Once the Hilbert space is introduced, the Gel'fand-Naimark-Segal construction shows that pure states correspond then to rays in the Hilbert space (i.e. to vectors/point of the projective Hilbert space). This corresponds to the quantum case of the classical situation were the abelian version of Gel'fand-Naimark states that every commutative $C^*$-algebra (with unity) is isometrically isomorphic to the algebra of continuous functions for some compact Hausdorff space, thus recovering phase space. This establishes the kinematics of the theory, after which dynamics can be introduced and studied by either evolving states with operators fixed (Schrödinger's picture) or evolving operators with states fixed (Heisenberg's picture), and these are dual to each other. In the statistical physics point of view, when a system is in a given state, all one really measures about an observable is its expectation value with respect to the probability distribution of that state; thus, a state can be viewed as a positive linear functional on the $C^*$-algebra of observables (not to be confused with the functionals given by Dirac's "bras" discussed below), establishing the duality between states and operators: a "state" is a probability distribution on the algebra, determining the probability of possible values of any observable in the next measurement, now by Gleason's theorem any such distribution implies the existence of a density operator which represents a pure (or more generally mixed) state of the system. Thus, you can either see a pure state as an eigenvector of a complete set of compatible observables, or either a positive linear functional on their algebra. This is very deep and important because removes any ontological weight to the state vectors beyond the mere fact of being "the collection of probabilities of possible actualization of values" of the system.

Point 4 can now be understood naturally. When measuring observables, the instruments are localizing "where in the spectrum of eigenvalues" the system has each property. If measuring a particular observable has no effect on the value of another, then they are compatible, and by exhausting the measurements of a complete set of compatible observables for a system, one specifies the state of the system at that moment: since our system is characterized by the properties we observe, particular defined properties for each of its features characterizes the system completely. Since by measuring compatible observables we are selecting eigenvalues in the spectrum of the operators, we are actually projecting from the complete Hilbert space down to a particular vector labeled by the eigenvalues, as a complete set of commuting self-adjoint operators define a common eigenbasis. Thus the "string of data of observed values" is our measured observed state, so one thinks of the vector (ket $|\psi\rangle$) as the pure state of the system. Since predictions of the theory do not depend on the norm of the vector, the state of the system is actually a ray, i.e. a vector in the projective Hilbert space. (In fact since this must be done also for continuous spectrum of unbounded self-adjoint operators, the right formalism is that of rigged Hilbert spaces). Therefore, each (ray) vector basis of the Hilbert space corresponds to a choice of which set of compatible observables one is measuring, with each vector of each basis being a possible state to get in observations, i.e. a possible array of (eigen)values of our chosen properties to measure and characterize the system. In between observations, the isolated system evolves unitarily, so the "hidden" state of the system gets into a general superposition of eigenvectors in any chosen eigenbasis. Besides the duality on the operator-state level, there is the other dual notion of "bra" $\langle\psi|$ which are the linear functionals on the vectors of Hilbert space. Do not confuse Hilbert vector states, their vectorial duals as final states, and the state seen as a positive linear functional on the algebra of observables: the pure measurable states $|\psi\rangle$ are given by eigenstates in a common eigen-basis of commuting self-adjoint operators, and general states as a superposition of those; now since most of the prediction of the formalism are given by the Hilbert scalar products of vector states, $(|\psi\rangle,\, |\chi\rangle)$, the Riesz representation theorem guarantees that there is a functional $\phi_\psi:\mathcal H\rightarrow\mathcal C$ such that $\langle\psi|\chi\rangle :=\phi_\psi(|\chi\rangle)=(|\psi\rangle,\, |\chi\rangle)$, so Dirac rewrote the whole formalism in terms of bras $\langle\chi|$ and kets $|\chi\rangle$ to denote things like the probability amplitud to observe the sate $|\psi\rangle$ after having observed state $|\chi\rangle$, so: $\mathcal P(\chi\rightarrow\psi)=|\langle\psi|\chi\rangle |^2$. Since probabilities are scalar products squared, if $\chi$ or $\psi$ are linear superpositions of other eigenstates, the transition probability is not the sum of individual possibilities but there are also cross terms which are responsible for the interference quantum effects and the wave-particle duality. So you can think of bras, the functionals, as "final states" in a calculation.

The fundamental experimental fact is that there are properties which cannot be measured simultaneously (not even in perfect ideal conditions). If one measures the position of an atom, one gets a region in $\mathbb R^3$ more localized/small as more precise is the measurement device, but then the measurement of its linear momentum spreads in size over the possible values. Of course each measured value is as precise and definite as possible, but if you repeat the position measurement after the momentum one, the old value is not conserved and position randomly takes a new value within its spectrum, with a probabilistic distribution of dispersion/variance bigger as smaller is the uncertainty in momentum. What is happening is not a mysterious magic, but the fact that position and momentum operators do not commute, so they do not have a common eigenbasis, thus the system cannot be at the same time in a defined position and defined momentum. It is an old philosophical misconception that the act of observing by perturbing the system alters the value and makes simultaneous measurements impossible, complementarity is one of the core features of the quantum world regardless of who or what makes a measurement. Since "a state" is a set of defined properties of our system, there is no meaning to talking about simultaneously defined incompatible observables, in the same sense as there is no meaning in talking about the color of a music note (putting aside synesthesia). The confusion appears for trying to conceptualize the quantum world within classical realism (structural empiricism is much better in this regard). After the observation was made, if the system is allowed to evolve in isolation again, the theory predicts the probability of observing another possible eigenstate, maybe from another basis, at a later time. The formalism does this by evolving the initial observed eigenstate into a general linear superposition by Schrödinger's equation, so at a later time the complex components of the evolved vector over every eigenstate of any chosen basis have changed, with the square of the modulus of each component being the probability for observing the eigenvalues of that eigenvector. This is Schrödinger's picture, which is misleading philosophically (as Dirac himself claimed!), from an empiricist stance the more meaningful picture is Heisenberg's: the state is only the observed state, the ket characterized by the string of observed (eigen)values of a chosen set of compatible observables; when the system is isolated, its observables/operators evolve unitarily, the state does not change until observed again, but the new observation gives randomly new values as the operators have changed. Thus the whole formalism can be cast into operator algebras, with observables and states characterized by specific types of operators (as the states themselves can be seen as the projections of the spectral decomposition in a common eigenbasis, so if you just measure some of the compatible observables your state is a projector not into an eigenvector but into a common eigen-subspace of your chosen set of compatible observables).

SUMMARY: Systems are completely described by the observable properties they have, which may not be simultaneously defined. This is because having defined properties of certain kinds makes impossible to have defined properties of another kinds, so observing some aspects of a system destroys/"undefines" the previous properties which are incompatible with the new ones. This forces the algebra of observables to be noncommutative: the order of which observables are measured in succession does not necessarily commute. Since any physical measurement is numerical, in general real-valued, the noncommutative algebra fits nicely among self-adjoint operators, as they are the only unitarily equivalent to a real spectrum. Thus, we determine experimentally which sets of observables commute, so we can talk of complete sets of compatible operators, which define the state of the system completely by a string of eigenvalues (the actual values of the parameters/properties measured). Such a set defines a basis of a Hilbert space upon which the operators act, so different sets of compatible operators define different basis, so that if our system is given by a common eigenvector of one basis, it will be generally a linear superposition of the eigenvectors of another basis; since the eigenvalues are the observable properties of the system, the superposition in the other basis has no uniquely defined eigenvalues and thus those properties of the system in that state are not well-defined at that time. Since common eigenstates can be given by projection operators which project onto the successive eigensubspaces, a (pure) state of the system can be given either by a ray, by a suitable projection operator on the Hilbert space, or by a positive linear functional on the noncommutative $C^*$-algebra of quantum observables. One can work only with operator algebras and characterize which are observables and which are states and interrelate them by linear evaluations to get the predictions of the theory (mostly expectation values).

-

This is a rough version of the summary of the complete answer I posted on math.stackexchange, where more details are discussed in a long digression, in particular mathematical motivations for your points 1., 3. and 4. (Any reader interested in more explanations and a longer updated list of references should check out that other answer in Math.SE). I eagerly recommend you read carefully the following wonderful references:

Systems are completely described by the observable properties they have (or the degrees of freedom available for measurement or chosen to be described), properties which may not have simultaneously-defined values. This is because observing some aspects of a system may destroy/"undefine" some of the previous properties, precisely those which were incompatible with the new ones. For example, composing Stern-Gerlach apparatus in succession in different axis, allows you to measure the spin components $S_z$ and $S_x$, but after the second filter, a new measurement of the first will be random again, i.e. the observable property "having defined spin in direction __" is not simultaneously defined for independent directions: measuring one makes the system lose its definite value in the other. Another example is the position and linear momentum, which were the commuting degrees of freedom of classical phase space. Quantum mechanical experiments, very related to the originally called "wave-particle duality", determined that position and momentum wave functions (measuring the probability amplitude distributions of possible measurements) were related by a Fourier transformation of each other, thus implying that their commutator was $[\hat x,\hat p_x]=i\hbar$, so they became noncommuting variables better studied under operator algebras.

Since classically all observables are functions on phase space, this forces the algebra of general observables to be noncommutative: the order of which observables are measured in succession may not commute. Since any physical measurement is numerical, in general real-valued, the noncommutative algebra fits nicely within self-adjoint operators on Hilbert spaces, because these are the only operators unitarily equivalent to multiplicative diagonal operators of real spectrum. Since we can regard an observable property as just the set of its possible values, working with operator algebras allows for different such properties/observables to be encoded in the mutual algebra they define. Thus, we determine experimentally which sets of observables commute, so we can talk of complete sets of compatible operators, which define the state of the system completely by a data string of eigenvalues (the actual values of the parameters/properties measured). Such possible strings of eigenvalues for each operator in a compatible set define a "state of the system at a certain moment", but also define basis vectors of a Hilbert space upon which the operators act; hence different such sets define different basis, so that if our system is given by a common eigenvector of one basis, it will be generally a linear superposition of the eigenvectors of another basis, producing interfering cross terms in the transition probabilities from one state to another. This approach is found experimentally to be the right one since, besides complementarity, the other major quantum feature is linear superposition of states. Since the eigenvalues are the observable properties of the system, the superposition in other basis has no uniquely defined eigenvalues and thus those other properties of the system in that state are not well-defined at that time. Since common eigenstates can be given by projection operators which project onto the successive eigensubspaces, a pure state of the system can be given by either a ray of the projective Hilbert space or by a suitable projection operator: every operator of a complete commuting set is expanded by the spectral theorem as a sum of projection operators weighed by its respective eigenvalues, where the projectors project onto each common eigenvector, and that is the way in which operators encode both the list of possible measurable values of a property and their mutual algebra. Then, one can formalize quantum mechanics only with operator algebras (where such things as C*-algebras and Gel'fand-Naimark enter) and characterize which operators are observables and which are pure states, or mixed states if one includes uncertainties in the preparation initial states. In this sense one recovers Heisenberg's picture/matrix mechanics, where the state was just the list of observed values for a chosen set of compatible measuring devices, and the observables evolved with time between measurements so one obtained probabilistically a new set of values for maybe a different set of compatible measurements. Complementarity and uncertainty was a direct mathematical consequence of the noncommutativity, and interference/wave-particle duality a direct byproduct of the linear character of unitary evolution in between observations.

SUMMARY: Therefore, observables are given by algebras of self-adjoint operators because these are those having real spectra corresponding to the possible real empirical values. Maximal commuting subsets of operators are simultaneously diagonalizable, i.e. they represent compatible observables that can be measured at the same time, so the state of the system at every observation is characterized by such a list of eigenvalues. This defines a Hilbert space by linear superposition upon which the operators act: an observable has a definite value if and only if the state is in one of its eigenvectors. Given an eigenstate in a chosen basis, unitary evolution in Schrödinger's picture moves the vector in the Hilbert space so its projected component to every other possible basis vector changes, where the squared modulus of those complex components of the present vector state over any other vector give the probability of observing the second set of eigenvalues in our next measurements given the first set in our initial measurement. Since these components are given in terms of scalar products $(|\chi\rangle,\, |\psi\rangle)$, by Riesz theorem one can use linear functionals $\langle\chi|$ to act on vectors and give the desired transition amplitudes; since these functionals form also a (dual) linear space, one can think of them as possible final states in our computations. Physical states however are just empirical list of values for every observed degree of freedom of the system, very careful metaphysical considerations must be taken into account before giving meaning to intermediate, not observed superposition states: they statistically correlate observed states at different times but the system cannot be said to have a superposition of its properties. As Dirac claimed: Heisenberg's picture is the [empirical] right one (no one ever observes superpositions!).

As Asher Peres said: "Experiments occur in a laboratory, not in a Hilbert space" and "unperformed experiments have no results". The ontological and epistemological interpretational problems of the formalism enter when one attempts to think of it beyond an empiricist stance, attributing reality to parts of the formalism (e.g. superposition of cats) which cannot be observed experimentally (yet!?) beyond the structural/correlational level. (If you are interested in these issues I would recommend reading articles by Carlo Rovelli on relational quantum mechanics, and anything on the consisten/coherent histories formalism, besides the book by Isham listed at the top, in order to contrast standard orthodox references with warier understandings).

-
Thanks for taking the time and writing up such clear and understandable answer. By the way the Feynman lecture volumes have been made available for free by Caltech, see here. So you can the hyperlink used for that one. On another note, it would be definitely interesting to have your view on this question. Cheers –  user929304 Dec 15 '14 at 17:17
@user929304 thank you, I have added the link as you suggested. Regarding your other question, I am going to post a short answer now. Cheers. –  Javier Álvarez Dec 15 '14 at 17:55