Inverse of a Hermitian Operator

The momentum operator $p = i\frac{\mathrm d}{\mathrm dx}$ (with $\hbar = 1$) is hermitian. Hence its imaginary exponential i.e. $U = e^{ip}$ must be unitary. $U$ being a unitary operator, must have a unique and well defined Inverse $U^{-1}$ such that $U^{-1} = U^{\dagger}$ and hence both the right and the left inverse exist and are unique with $$U_{l}^{-1}=U_{r}^{-1} = U^{-1}.$$ But then Taylor expanding $U$ we have $$U = e^{ip}= e^{-\frac{\mathrm d}{\mathrm dx}}=\left(I-\frac{\mathrm d}{\mathrm dx}+\frac{1}{2!}\left(\frac{\mathrm d}{\mathrm dx}\right)^2 +\cdots\right).$$ But we know that $\frac{\mathrm d}{\mathrm dx}$ has no left inverse but has right inverses, with the right inverses being not unique. In the same reasoning $U$ must also have no unique right inverse and no left inverse. What is the fallacy in the argument here?

• The fallacy seems to be that the self-adjoint operator associated with the inverse unitary operator $U^{-1}$ is $-p$, not $p^{-1}$. – Qmechanic Jun 24 '17 at 15:08