What kind of things can go wrong if we try to do quantum mechanics on a nonseparable Hilbert space? I have heard that usual mathematical manipulations that we take for granted will no longer hold. What I am looking for is either a general high-level argument explaining why things would go wrong or some specific examples where it is so.
I'm normally loath to just quote Wikipedia, but it does have some relevant things to say:
The article on separable spaces tells us that
every continuous function on a separable space whose image is a subset of a Hausdorff space is determined by its values on the countable dense subset.
(cf densely defined operator) and that
many theorems that can be proved for nonseparable spaces have constructive proofs only for separable spaces. Such constructive proofs can be turned into algorithms for use in numerical analysis [...]. A famous example of a theorem of this sort is the Hahn–Banach theorem.
Furthermore, the article on Hilbert spaces contains the following:
A Hilbert space is separable if and only if it admits a countable orthonormal basis.
In case of field theory, it states:
Even in quantum field theory, most of the Hilbert spaces are in fact separable, as stipulated by the Wightman axioms. However, it is sometimes argued that non-separable Hilbert spaces are also important in quantum field theory, roughly because the systems in the theory possess an infinite number of degrees of freedom and any infinite Hilbert tensor product (of spaces of dimension greater than one) is non-separable. For instance, a bosonic field can be naturally thought of as an element of a tensor product whose factors represent harmonic oscillators at each point of space. From this perspective, the natural state space of a boson might seem to be a non-separable space. However, it is only a small separable subspace of the full tensor product that can contain physically meaningful fields (on which the observables can be defined).
The main thing that goes bad in nonseparable Hibert space is the loss of Stone-von Neumann theorem. Loosely speaking, the Stove-von Neumann theorem assures us that Schroedinger representation of the canonical commutations rules is irreducibile, and it is unique modulo unitary equivalence. Hence Schroendinger and Heisenberg pictures are physically equivalent, and all pictures obtained by Schroedinger's one by means of the adjoint action of the group of the unitary operators on the Hilbert space is physically equivalent to that of Schroedinger.
Remark. Separability of the Hilbert space is part of the conlusion of the Stone-von Neumann theorem, not of hypotesis. In fact, there is no serious physical request for the separability of the Hilbert space. All that is required in hypotesis is that Weyl commutation rules can be irreducibility represented on some Hilbert space, which a posteriori turns out to be separable. This is a physical request, being the direct formalization of the Heisenberg-Born-Jordan commutation rules extrapolated from expertiments.
Remark. One may note that also in QFT Stone-von Neumann falls. This is true, but for a different reason: one of the hypotesis is that we are dealing with finite degree of freedom, while specifying the status of a field at a given time requires to give the value of an infinite number of parameters. This is effectively a phyisically different situation, mathematically encoded in the different quality of the statements.
The best known example of a theory endowed with a nonseparable Hilbert space is Loop Quantum Gravity. In order to see how nonseparable character plays his role, it is convenient to consider its "first-order approximation" (in a sense), the so-called polymer quantum mechanics. In this context, it is particulary simple to see that usual physical quantities, e.g. the spectrum of the harmonic oscillator, gain a correction of order $\hbar^2$ or smaller. Hence, such a model provides a method to verify its correctness (or not), but corrections are so small that this will be hard to do in practise. Obviously, so far LQG and related models are not physical theories, so this point is purely accademic. I am not aware of any example of fully meaning physical theory having a nonseparable Hilbert space.
Remark. On separable Hilbert spaces, orthonormal bases can be explicitely constructed by means of Grahm-Schmidt algorithm. Thanks to the Zorn's Lemma, each vector space has an algebraic (Hamel) basis, so we can affirm that even a nonseparable Hilbert space has an algebraic basis but we can't construct it explicitely. (in the context of Banach spaces, this is the case for $\ell^\infty(\mathbb N)$.)
Personal considerations. Separability means the existence of a dense countable subset. At the moment, this doesn't sound to me like a true physical necessity. It is more like the consequence of more fundamental facts. This is different from the hypotesis of being Hausdorff, which is the mathematical transcription of the fact that we cannot erase completely experimental errors, but we can make them smaller and smaller, at least in principle.