The relation
$$\langle a|b\rangle\propto\delta(a-b)$$
is nothing unusual, it is simply an orthogonality condition. If the proportionality was an equality, and in addition we had completeness, the set of states would form an orthonormal basis. The reason why the delta function shows up is that you assume your operator to have a continuous spectrum of eigenvalues.
Since we are working with a vector space, it only seems natural that there should be some way to define an orthogonality condition. The delta distribution, as a linear functional on the Hilbert space, provides the appropriate structure for that. If you worry about infinities, there are two things to keep in mind: formally, the delta function is not really infinite, as it is strictly only defined under an integral. This is due to its nature as a distribution. The other thing is that it is not an observable quantity anyways: what is physically relevant are eigenvalues of operators and probabilities.