What kind of things can go wrong if we try to do quantum mechanics on a nonseparable Hilbert space? I have heard that usual mathematical manipulations that we take for granted will no longer hold. What I am looking for is either a general high-level argument explaining why things would go wrong or some specific examples where it is so.
I'm normally loath to just quote Wikipedia, but it does have some relevant things to say:
The article on separable spaces tells us that
(cf densely defined operator) and that
Furthermore, the article on Hilbert spaces contains the following:
In case of field theory, it states:
The main thing that goes bad in nonseparable Hibert space is the loss of Stone-von Neumann theorem. Loosely speaking, the Stove-von Neumann theorem assures us that Schroedinger representation of the canonical commutations rules is irreducibile, and it is unique modulo unitary equivalence. Hence Schroendinger and Heisenberg pictures are physically equivalent, and all pictures obtained by Schroedinger's one by means of the adjoint action of the group of the unitary operators on the Hilbert space is physically equivalent to that of Schroedinger.
Remark. Separability of the Hilbert space is part of the conlusion of the Stone-von Neumann theorem, not of hypotesis. In fact, there is no serious physical request for the separability of the Hilbert space. All that is required in hypotesis is that Weyl commutation rules can be irreducibility represented on some Hilbert space, which a posteriori turns out to be separable. This is a physical request, being the direct formalization of the Heisenberg-Born-Jordan commutation rules extrapolated from expertiments.
Remark. One may note that also in QFT Stone-von Neumann falls. This is true, but for a different reason: one of the hypotesis is that we are dealing with finite degree of freedom, while specifying the status of a field at a given time requires to give the value of an infinite number of parameters. This is effectively a phyisically different situation, mathematically encoded in the different quality of the statements.
The best known example of a theory endowed with a nonseparable Hilbert space is Loop Quantum Gravity. In order to see how nonseparable character plays his role, it is convenient to consider its "first-order approximation" (in a sense), the so-called polymer quantum mechanics. In this context, it is particulary simple to see that usual physical quantities, e.g. the spectrum of the harmonic oscillator, gain a correction of order $\hbar^2$ or smaller. Hence, such a model provides a method to verify its correctness (or not), but corrections are so small that this will be hard to do in practise. Obviously, so far LQG and related models are not physical theories, so this point is purely accademic. I am not aware of any example of fully meaning physical theory having a nonseparable Hilbert space.
Remark. On separable Hilbert spaces, orthonormal bases can be explicitely constructed by means of Grahm-Schmidt algorithm. Thanks to the Zorn's Lemma, each vector space has an algebraic (Hamel) basis, so we can affirm that even a nonseparable Hilbert space has an algebraic basis but we can't construct it explicitely. (in the context of Banach spaces, this is the case for $\ell^\infty(\mathbb N)$.)
Personal considerations. Separability means the existence of a dense countable subset. At the moment, this doesn't sound to me like a true physical necessity. It is more like the consequence of more fundamental facts. This is different from the hypotesis of being Hausdorff, which is the mathematical transcription of the fact that we cannot erase completely experimental errors, but we can make them smaller and smaller, at least in principle.