In many introductory courses to quantum mechanics, we see $\delta$-functions all over the place. For example when expressing an arbitrary wave function $\psi(x)$ in the basis of eigenfunctions of the position operator $\hat x$ as $$ \psi(x) = \int\mathrm d\xi\, \delta(x-\xi)\, \psi(\xi). $$ In bra-ket notation this corresponds to $$ \left|\psi\right\rangle = \int\mathrm d\xi\,\left|\,\xi\,\right\rangle\!\left\langle\,\xi\,\middle|\,\psi\,\right\rangle, $$ where $\left|\,\xi\,\right\rangle$ is the state corresponding to the wavefunction $x\mapsto\delta(x-\xi)$. Now the $\delta$-function is really not a function, but a distribution, that's defined by how it acts on test-functions, i.e. $\delta[\varphi] = \varphi(0)$.
Do you know of an introductory text on quantum mechanics that stresses this point and uses the language of distributions properly, avoding any functions with seamingly infinite peaks?