About the definition of expectation value in quantum mechanics In quantum mechanics, the expectation value of a observable $A$ is defined as
$$\int\Psi^*\hat A\Psi$$
But in probability theory the expectation is a property of a random variable, with respect to a probability distribution:$$E(X):=\int X\;d\mu$$
I can't see how probability theory can be adapted to quantum mechanics. Observables are associated with linear operators, not measureable functions, so how can we talk about the expectation of a linear operator? And quantum mechanics textbooks use expectations and variances without mentioning underlying probability spaces. Does quantum mechanics use something other than ordinary probability theory?
 A: Since you want a bit of mathematical rigor:
A quantum state is a self-adjoint positive trace class operator on a Hilbert space with trace 1. This is called density matrix $\rho$. In its simplest form, given $\psi\in \mathscr{H}$, $\rho$ is the orthogonal projector on the subspace spanned by $\psi$. Let $E_\rho(\cdot):D_\rho\subset\mathcal{A}(\mathscr{H})\to \mathbb{R}$ be the map defined as:
$$E_\rho(A)=\mathrm{Tr}(A\rho)\; ,$$
where $\mathcal{A}(\mathscr{H})$ is the space of self-adjoint operators, $\mathrm{Tr}$ is the trace on $\mathscr{H}$ and
$$D_\rho=\{A\in \mathcal{A}(\mathscr{H})\; ,\; \mathrm{Tr}\lvert A\rho\rvert<+\infty\}\; .$$
The map $E_\rho(\cdot)$ has all the properties of an expectation in probability theory. I don't know if it is possible to characterize the measure $\mu$ associated to it (maybe by means of the projection valued measures associated to $\rho$ by the spectral theorem, but it is not straightforward at least for me).
A: $$
<\hat{A}>  = \int \psi^*(x)\hat{A}\psi(x) dx
$$
now $\hat{A}\psi(x)=a(x)\psi(x)$ so,
$$<\hat{A}>=\int a(x)|\psi(x)|^2dx$$
Let $|\psi(x)|^2dx =  d \mu$ now $\int|\psi(x)|^2dx=\int d\mu = 1$
$$
<\hat{A}>  = \int a(\mu)d\mu
$$ 
A: There are two answers to this. One answer simply points out that the probability of the jth outcome specified by the Born rule $p_j = tr(\rho\hat{P}_j)$, where $\hat{P}_j$ is the projector  onto the jth outcome, satisfy the axioms of probability:
http://mathworld.wolfram.com/ProbabilityAxioms.html.
Another answer is that the Born rule can be explained using decision theory:
http://arxiv.org/abs/0906.2718.
A: 
Observables are associated with linear operators, not measureable functions, so how can we talk about the expectation of a linear operator?

The "expectation of a linear operator" is a term from quantum theory. It is defined by the integral
$$
\int \psi^*(x)\hat{A}\psi(x) dx
$$
or similar. The meaning of this number is not necessarily the same as expectation in ordinary probability theory.
This number has been used in the meaning of actual value, average actual value, expected average actual value, expected average value of results of measurements or others. Although there is agreement on usefulness of the basic formulae, concept of probability and especially probability in quantum theory carries with it a lot of puzzles on which there is no universal agreement.

And quantum mechanics textbooks use expectations and variances without mentioning underlying probability spaces. Does quantum mechanics use something other than ordinary probability theory?

People use many rules, some quantum, some from ordinary probability theory, to get probabilities. These are not always compatible. Then people interpret these probabilities differently. Unfortunately there is a lot of confusion in this.
A: Let us use bra-ket notation. Suppose that the operator $\hat A$ has discrete and bounded eigenvalues $\mathcal A =\{A_1, A_2, \ldots\}$ with eigenkets $|A_1\rangle, A_2\rangle,\ldots$. The eigenkets form a complete orthogonal set since $\hat A$ is symmetric. Then any ket $|\psi\rangle$ can be expanded, $$|\psi\rangle = \sum \psi_n |A_n\rangle.$$
Since the $|A_n\rangle$ are orthogonal, $$\langle \psi |\psi \rangle =\sum \psi_n^* \psi_n = \sum |\psi_n|^2 = 1.$$
Clearly the map $$\mu_\psi : 2^\mathcal{A} \to \mathbb{R}^+$$
defined on singletons by $$\mu_\psi : \{A_n\} \mapsto |\psi_n|^2$$
and extended by additativity defines a measure on the $\sigma$-algebra of subsets of $\mathcal A$. Since $\mu_\psi(\mathcal A) = 1$, it's a probability distribution. By expanding $\langle \psi |A |\psi\rangle$ you can see that this quantity is indeed the expectation value of the distribution, defined in the usual sense.
For a continuous or unbounded, spectrum, sums go over to integrals. But, one has to be more careful, since such an operator is not defined on all of the Hilbert space, and may not have eigenfunctions in the Hilbert space (for example the momentum operator and position operators are like this; their eigenfunctions are not $L^2$). But with some effort the formal steps above can be repeated.
One can use more abstract sledgehammers like in another answer to make the correspondence obvious by definition, but I think this explanation is easier for a beginning student.
