Another way to look at this problem is to consider it in position space, and then transform the solution to it's momentum space representation. While this may seem like an unnecessary amount of work, it may illuminate to you the delta function solution in a different way. So, in position space we have
$$-\frac{\hbar^2}{2m}\frac{d^2\psi}{dx^2}=E\psi\:\:\:\xrightarrow{\kappa^2\:\equiv\:2mE/\hbar^2}\:\:\: \frac{d^2\psi}{dx^2}=-\kappa^2\psi\:\:\:\rightarrow\:\:\:\psi(x)=Ae^{i\kappa x}+Be^{-i\kappa x}$$
Before casting this into its momentum space representation, recall the integral representation of the Dirac delta function (which can be arrived at by considering orthogonality of position or momentum eigenstates):
$$\delta(\alpha-\beta)=\frac{1}{2\pi\hbar}\int e^{ix(\alpha-\beta)/\hbar}dx.$$
Using the above, let's Fourier transform our solution to get its momentum representation:
$$\psi(p) = \frac{1}{\sqrt{2\pi\hbar}}\int\psi(x)e^{-ipx/\hbar}dx = \frac{A}{\sqrt{2\pi\hbar}}\int e^{ix(\kappa-p/\hbar)}dx + \frac{B}{\sqrt{2\pi\hbar}}\int e^{ix(-\kappa-p/\hbar)}dx\\ = \sqrt{2\pi\hbar}\Big[A\delta(\kappa-p/\hbar)+B\delta(-\kappa-p/\hbar)\Big].$$
Now stick in $\kappa = \sqrt{2mE}/\hbar$, and use the fact that $\delta(-x) = \delta(x)$ and $\:\delta(\alpha x) = \delta(x)/|\alpha|$ to rewrite this as
$$\psi(p) = \tilde{A}\delta(p-\sqrt{2mE}) + \tilde{B}\delta(p+\sqrt{2mE}),$$
where I've collected constants and called them $\tilde{A}$ and $\tilde{B}$ for simplicity of the final solution. Obviously this is more work than noticing the solution in momentum space corresponds to delta function behavior, but perhaps you'll find this route illuminating; or, if nothing else, a nice consistency check.