Does mathematical sloppiness in quantum mechanics ever produce incorrect predictions? Does mathematical sloppiness in standard quantum mechanics ever produce predictions that don't pan out? I'm not talking about things like the WKB approximation, but instead subtle functional analytic issues, such as assuming every Hamiltonian is self-adjoint, has an eigenbasis of bound states, domain issues, etc. I don't know of any such experiment, but it is conceivable that a bad enough physical Hamiltonian exists so that the standard methods fail. 
I emphasize that I am looking for actual experiments people have done, not thought experiments and not contrived counterexamples.
 A: I do not know if this will completely answer your question, but there is quite a debate about a proper definition of a quantum phase operator.  In


*

*Pegg, D. T., and S. M. Barnett. "Unitary phase operator in quantum mechanics." EPL (Europhysics Letters) 6.6 (1988): 483,

*Barnett, Stephen M., and David T. Pegg. "Phase in quantum optics." Journal of Physics A: Mathematical and General 19.18 (1986): 3849.

*Barnett, S. M., and D. T. Pegg. "On the Hermitian optical phase operator." Journal of Modern Optics 36.1 (1989): 7-19,

*Pegg, D. T., and SM Barnett. "Phase properties of the quantized single-mode electromagnetic field." Physical Review A 39.4 (1989): 1665


David Pegg and Steve Barnett precisely proposed such a definition.  Their suggestion remains the topic of debate.  You can read one critique in 


*

*Bergou, János, and Berthold-Georg Englert. "Operators of the phase. Fundamentals." Annals of Physics 209.2 (1991): 479-505.


One relevant part reads:

The insistence that the $N\to\infty$ limit is taken after all is said and done is of no help in our opinion. For, does the injunction, to pick $N$ sufficiently large, depending on the state of the physical system, not signify that the operators themselves are state dependent? For those who, as we do, answer yes, does this not make havoc of the linearity of the “operators” ?

Basically Bergou and Englert argue that Pegg and Barnett have done sloppy math somewhere, taking a result true in some limit and using it in the finite $N$ regime. Unfortunately, the jury is still out on who's right as there is no experimental decision available.
A: One slopiness that many teachers are guilty of is teaching this misleading thing:
The position operator $\hat{x}$ has eigenvectors $|x_0\rangle$ that obey
$$
\hat{x} |x_0\rangle = x_0|x_0\rangle
$$
and are represented by distributions on domain of $x$: $\delta(x-x_0)$ for different $x_0$. (WRONG)
The incorrect predictions come when student uses this "representing function" as a simple initial condition to find out how a localized psi function spreads out in time, or to calculate expected average of position.
Let me demonstrate the latter case: calculating expected average of position in such state $|x_0\rangle$ using the standard algorithm, we get
$$
\langle x \rangle = \langle x_0|\hat{x}|x_0\rangle = x_0 \langle x_0|x_0\rangle
$$
It is tempting to  put $\langle x_0|x_0\rangle = 1$ now, but this is not correct, because we already said that $|x_0\rangle$ is represented by delta distribution. The expression is just not defined, as the integral
$$
\int \delta(x-x_0)\delta(x-x_0)dx
$$
is not defined (or, sometimes said to be infinite). So, here the slopiness of assuming position operator has eigenvectors leads us to incorrect prediction that there is no expected average of position. Such result would be correct for, say, the Cauchy distribution, but it is incorrect for a localized one we implicitly assume to describe here. For any well-localized psi function around $x_0$, the correct answer is close to $x_0$. 
The correct way  to handle this is to teach that position operator has no eigenfunctions, but we can assign it improper eigenvectors $|x_0\rangle$ that are however no realizable psi functions. So that fact the very position operator used to define such kets has no expected average for such kets is no problem, because physical kets can never be equal to such kets.
