A confusion about base states of a quantum system

I have been told that the eigenkets of a operator of a space form a basis for the state of the quantum system. The eigenbasis obtained from the position operator $\textbf{x}$ is the (continuously) infinite set $|x\rangle$, whereas for a general angular momentum operator the basis $|\ell,m\rangle$ is countably infinite, $|m|\leq \ell$, $\ell\in\mathbb{N}_0$. This is a contradiction as in this case the dimension of the state space does not have a well-defined cardinality.

At this point one could say that these are actually different informations and the total state is some kind of a direct product of these states lying in different vector spaces. But then again I have been told that $\langle x|j,m\rangle=Y_l^m$ which means that these actually reside in the same space (because we have their inner product).

So what is the state space of a quantum system with both $x$ and angular momentum and what is the correct basis?

-
The state space (/correct basis) is the tensor product of state spaces (/of the kets of both basis), so the resulting dimension is the product of dimensions. Inner products only make sense in the common base, but usually nobody thinks about that when facing the inner product of your example, because it is equivalent to the wave function in the positions representation. –  Eduardo Guerras Valera Jan 20 '13 at 9:30
I suggest that you fully learn the basic postulates and algebraic manipulations (it is just a few hours, very easy, the difficulties come later) before extracting conclusions. With incomplete information about the basics, Quantum Mechanics can be a hell, no matter that you are an expert in algebra. QM has its own notation and conventions, very easy after having studied blindly a couple of rules at the beginning. A good book for that is Sakurai's Modern Quantum Mechanics, the first chapters. –  Eduardo Guerras Valera Jan 20 '13 at 14:28
For example, the inner product that puzzles you here is not incorrect, but rather it is understood that you are dealing with a common basis whose dimension is the product of dimensions, and that that basis exists, because the observables involved commute. –  Eduardo Guerras Valera Jan 20 '13 at 14:35

The state space is the space of square integrable functions $L^2(\mathbb{R}^3)$. (Edit: in the following I added reference to the radial wavefunction) One way to get a basis that spans this space is to combine (i.e. take the tensor product of) the angular momentum basis $|l,m\rangle$ and some basis for the radial wavefunction (such as the eigenfunctions of the Hydrogen atom). Such a basis determines the cardinality of the state space to be countably infinite.

However, sometimes it is convenient to consider a larger space: the space of all continuous functions $C(\mathbb{R}^3)$, which is spanned by the position basis $|x\rangle$, and therefore has a larger cardinality (Edit: this space is analogous to the rigged Hilbert space referred to this answer). This space contains functions that have an infinite norm, and therefore cannot represent quantum states, but they can still be useful in our calculations.
Take for example the wavefunction $e^{i k x}$. It is part of $C(\mathbb{R}^3)$, but not part of $L^2(\mathbb{R}^3)$ since it cannot be normalized. Nevertheless, it is very useful to use this wavefunction in calculations, for example in scattering theory. Once we know how this wavefunction evolves over time, we can use the linearity of Quantum mechanics to know how any superposition of such wavefunctions evolve over time. Using the Fourier transform we can express any function in $L^2(\mathbb{R}^3)$ as a linear combination of functions of the form $e^{i k x}$, and therefore we have solved the time evolution of any wavefunction in $L^2(\mathbb{R}^3)$.

In conclusion, using a function space that has a larger cardinality is useful, as long as we remember that in the end of the calculation we need to return to the state space we started with.

-

OP wrote(v3):

[...]This is a contradiction as in this case the dimension of the state space does not have a well-defined cardinality.

$$Y_{\ell m}(\theta, \varphi)~=~\langle \theta, \varphi |\ell, m \rangle$$

live on the two-sphere $S^2$ parametrized by the coordinates $(\theta, \varphi)$; and how we may view $\mathbb{R}^3\backslash\{\vec{0}\}\cong ]0,\infty[\times S^2$ as a half-line times a two-sphere using spherical coordinates $(r,\theta, \varphi)$.

So in this answer we will just mention that the alleged contradiction (in counting the dimension of the state space) has a simpler analogue for the one-sphere $S^1\cong\mathbb{R}/2\pi\mathbb{Z}$, i.e. the circle. Recall that functions $\psi(\theta)=\psi(\theta+2\pi)$ on the circle $S^1$ are periodic functions of some angular position variable $\theta\in[0,2\pi[$. A countable basis $(|m\rangle )_{m\in\mathbb{Z}}$ is via Fourier series

$$\psi(\theta)~=~\sum_{m\in\mathbb{Z}} c_m e^{im\theta}, \quad e^{im\theta} ~=~\langle \theta| m \rangle,\quad c_m~=~\langle m |\psi\rangle,\quad \psi(\theta)~=~\langle \theta |\psi\rangle.$$

The Hilbert space is a $L^2$-space $$H~:=~L^2(S^1)~\cong~\ell^2(\mathbb{Z})~:=~\left\{(c_n)_{n\in\mathbb{Z} } ~\mid~ \sum_{m\in\mathbb{Z}} |c_n|^2 < \infty\right\},$$ which is isomorphic to the set of square integrable series. See also e.g. this Phys.SE post.

A continuous basis is $(|\theta\rangle )_{\theta\in[0,2\pi[}$ does not belong to the Hilbert space $H$ but rather to the so-called rigged Hilbert space. See also e.g. this Phys.SE post. So there is no contradiction. We are just talking about two different state spaces. Essentially the same explanation (for the alleged contradiction in counting the dimension of the state space) also applies to e.g. the two-sphere $S^2$ and the three-space $\mathbb{R}^3$.

-

The kets $|l,m\rangle$, which correspond to the spherical harmonics $Y_l^m$ are just the eigenstates of definite orbital angular momentum (with respect to some origin). The angular momentum operator, in Cartesian coords is $$\hat{L}=\hat{R}\times\hat{P} =-i\epsilon_{ijk}x_j\frac{\partial}{\partial x^k}$$ Switching to spherical polars, you can then define the total $l^2$ and z component $l_z$ operators. The eigenfunctions of these 2 operators are the functions $Y_l^m$. These functions just depend on $\theta$ and $\phi$ so you can think of them as living on the 2-sphere $S^2$.

Now a wavefunction of a particle in $\mathbb{R^3}$ which is an eigenfunction of $l^2$ and $l_z$ can be factorized as $$f(r)Y_l^m(\theta, \phi)$$ Not every wavefunction in $L^2(\mathbb{R^3})$ can be written like this, but for any wavefunction you can find an expansion in terms of this form which converges to the wavefunction. So essentially $$L^2(\mathbb{R^3}) = L^2({\mathbb{R_+}})\otimes L^2(S^2)$$ where $L^2({\mathbb{R_+}})$ is the space of square integrable functions of the radial coordinate.

In the expression $\langle x|l, m\rangle$ you're just formally taking the inner product of a position eigenstate with one of the angular momentum eigenstates, i.e. writing the position representation of an angular momentum eigenstate, $f(r)Y_l^m(\theta,\phi)$

So to get back to the question, the state space of a particle with both $x$ information and orbital angular momentum is just $L^2(\mathbb{R^3})$, the state space of square integrable functions on $\mathbb{R^3}$, which can be decomposed as above to bring out the orbital angular momentum part.

If, in your last sentence, you were instead referring to spin angular momentum, then you would be tensoring the position eigenstates with the intrinsic spin eigenstates to produce spinor wavefunctions.

-