First of all, the state |ψ> does not have any dimension. You are correct to point out that these constants only appears when one ore more of the observables of the complete commuting set take on a continuous range of possible values. Normalization constants and weight functions are used to properly construct the identity operator for a given orthogonal eigenbasis in terms of an indiscrete sum.
Let me take 2 cases to illustrate. Let p be an observable and form a complete commuting set all on its own.
p takes on only integral values like 1,-5 etc. So Σ|p>< p| = I, where I is the identity operator sum is taken over all integers. In this case you can see that there will obviously have no dimension. So we don't need constants.
p takes on all real numbers between a and b. The sum of all values of a continuous function in [a,b] has infinitely dense number of terms and is thus proportional to ∫ab fdx, regardless of whether the summation value diverges or not, it is proportional to the integral.
Therefore ∫ab |p> dp*W < p| = I. In some cases with more than one observable, the terms of the the indiscrete sum need to be weighted to get consistency across all co-ordinate systems chosen which is why we have the weight function W(p).
As <ψ|ψ> = 1 due to condition of normalization, putting I in between should give the same result, we can determine the constant to be present in W from this condition. dp*W(p) is thus effectively the 'volume' element dV divided by V. For convenience we multiply the square root of 1/V into , in which case we don't need to explicitly write it out in the integral.
You might also ask what if the limits of p tend to infinity and the integral diverges? In that case V diverges, so we cannot evaluate it.
In a nutshell, there is no physical dimension for |ψ> at all.