You've made a small mistake when looking at the units of $\psi$, which may be where your confusion comes from. Done correctly, theThe units do add up in the example you provide:
You've correctly established that the delta distribution has the dimension of the inverse dimension of its argument. This can be seen in a variety of ways, e.g. by looking at $\int f(x)\delta(x-x_0) dx = f(x_0)$, where the LHS has to have whatever dimensions $f$ has.
Looking at your example, $\psi_p(x)=A\exp(\frac{ipx}{\hbar})$, we want $\int \lvert \psi \rvert^2 dx$ to represent a probability, which is dimensionless. From this it follows that $A$ has to have dimensions $\textsf{L}^{-\frac12}$, not $\textsf{L}^{-1}$ as you stated.
With this, we look at the dimensions of $\lvert A \rvert^2 2\pi\hbar \;\delta(p_1-p_2)$: $$ \left[\lvert A \rvert^2 2\pi\hbar\; \delta(p_1 - p_2) \right] = \textsf{L}^{-1} \left[ \hbar \delta(p_1-p_2) \right] = \textsf{L}^{-1} \frac{\left[ \hbar \right]}{[(p_1-p_2)]} = \textsf{L}^{-1} [x] = \textsf{L}^{-1} \textsf{L}^{1} = 1, $$where I have used that $\frac{p}{\hbar}$ has to have inverse dimensions of $x$ to make the argument of the exponential dimensionless.
NB: For $\psi_p(x)=A\exp(\frac{ipx}{\hbar})$, we want $\int \lvert \psi \rvert^2 dx$ to represent a probability, which is dimensionless. From this it follows that $A$ has to have dimensions $\textsf{L}^{-\frac12}$.