Experimental Data for Mass Distribution of a Galaxy My goal here is not to discuss dark matter in general. I know there are many other observational clues that hint us towards Dark matter. My goal is simply to understand this argument here a little better.
One of the arguments for Dark matter are the observed rotational velocities of stars in the outer part of galaxies. 
If we assume that most of the galaxies mass lies inside, we can neglect the outside mass and get $$ \frac{mv²}{r}=G\frac{mM(<r)}{r^2} , $$ 
where $M(<r)$ denotes the mass inside the radius $r$ (This is done for example in Perkins book). This yields
$$ v \propto \frac{1}{\sqrt{r}}, $$
which isn't observed in experiments (see for example here).
Nevertheless most Galaxies are flat discs with a spherical hub in the middle
and therefore maybe we can assume in the outside region for the mass density $\rho \propto  \frac{1}{r}$, which seems reasonable because the mass density of a galaxy should get thinner in the outer regions.
Then we have for the mass inside radius $r$
$$ M(<r) =  \int_0^r  \rho(r') dA = \int_0^r  \frac{C}{r'} r'  dr' d\phi = 2 C \pi r $$
and therefore using Newtonian mechanics
$$ \frac{mv²}{r}=G\frac{mM(<r)}{r^2}= G\frac{m2 C \pi r}{r^2} $$
$$ \rightarrow v \propto const $$
This is exactly what is observed in experiments.
Obviously somewhere this argument must be flawed and my best bet would be $\rho \neq \frac{C}{r}$. The standard approach seems to be to assume something of the form $\rho \propto e^{-C R} $. What experimental data shows that $\rho \neq \frac{C}{r}$ and therefore that we need Dark matter to explain other phenomena.
 A: Well there are problems in your question and analysis. First off, there have been a few SE questions recently about this "Keplerian" treatment of dark matter. The shell theorem, that the gravitational field is the equivalent of that due to the mass inside radius $r$, and that exterior masses can be ignored is only true for spherically symmetric mass distributions or cases where most of the mass is centrally concentrated inside radius $r$. Any book that fails to point this out is making a serious omission, probably in the interests of simplifying the argument. Real work in this area does not make that assumption (e.g. Sofue 2011).
Even adopting the shell theorem, it does not imply $ v \propto r^{-1/2}$, it implies $v \propto (M(R)/r)^{1/2}$.
The correct argument for dark matter is that if we assume that the visible matter and gas traces the mass and has a certain mass-to-light ratio, then we find that (i) the rotation speeds of stars and gas are too high and that (ii) one would expect the velocities to drop as $r^{-1/2}$ at large radii, whereas they appear to actually be flat or even increasing.
The latter point works because the visible matter implies that there is hardly any mass at large radii, and so the Keplerian approximation is valid there.
Now, as for your model. If $\rho \propto r^{-1}$ in shells, this would imply that annuli with a given thickness $\Delta r$ contain similar quantities of mass! $\Delta M = 2\pi r z\rho\,\Delta r$ (where $z$ is a thickness for the disk). So, rather than having a galaxy whose azimuthally integrated luminosity decreased with distance from the centre, your Galaxy, without dark matter, would have to have constant integrated luminosity, as a function of radius as you move away from the centre (or to satisfy my critics below, if you are observing the Galaxy face on, the surface brightness of light would fall as just $r^{-1}$). And, of course without defining some sort of cut-off radius, the total mass of your Galaxy would soon become large! However, if you assume that most of this matter is dark then indeed you might be able to explain the rotation curve of the Galaxy using such a density law!
Below I show an example (using surface brightness) for M31 (taken from Corteau et al. (2012)), using various luminosity indicators, on which I have marked a $\rho \propto r^{-1}$ dependence. A $r^{-1}$ does work reasonably in the inner part of the disk (it is actually a bit shallower than that because of the bulge), but at some point $r> 10$ kpc, the luminous matter just runs out and the observed intensity distribution becomes steeper than $r^{-1}$.

In fact the most commonly used prescription for dark matter is the Navarro, Frenk & White dark matter profile, 
$$ \rho(r) = \frac{\rho_0 R_s}{r(1 + r/R_s)^2}, $$
where $R_s$ is a length scale (of order 15 kpc for the Milky Way and M31). When $r< R_s$, this does scale as $1/r$ and does explain the flat rotation curve!
i.e. Your analysis that a $\rho \propto 1/r$ relationship leads to a flat rotation curve is roughly correct. However, the fact that intensity in our Galaxy (and others) falls off more steeply than $r^{-1}$ in the outer parts of the Galaxy leads to the conclusion that the matter is ... dark! There is an additional problem with the normalisation as well. Even in the inner parts of the disk, the mass implied by the luminous matter is insufficient (by factors of a few) to explain the rotation speeds.
So now to answer the last part of your question - how do we know that $\rho$ (of luminous matter) does not fall as $1/r$ in the disk. This is just a question of counting up stars and estimating the contribution of gas from HI surveys (and dust, though this is negligible). There is no single source of this information (though here is an example I pick at random that uses SDSS number counts), it is agglomerated from many different surveys at different wavelengths and built up to give a coherent picture. The underlying assumptions are that we understand the types and mixture of stars that make up the overall stellar populations. Our understanding could be incorrect, but the way in which it would need to be incorrect to explain rotation curves is to have lots (and I mean orders of magnitude) more dim stars that contribute mass but no light at large radii (i.e. dark matter, though baryonic, which doesn't help you with other pieces of evidence for dark matter).
For example, the luminosity of the M31 data shown above drops off steeper than $r^{-1}$. If you were to extrapolate a $r^{-1}$ relationship, then to ensure that the mass did go as $r^{-1}$ you would need the mass to luminosity ratio to increase by some factors of ten. To do this would require orders of magnitude more faint stars to bright stars than are observed in the local disk.
