You're right - and this confused me a lot, too, because in particular, $r$ may not be a proper function of $m$, i.e. it may (and often is) be multivalued (one-to-many), in that there are multiple radii at which the same mass $m$ would exist, rendering the usual idea of integration problematic.
And what I came to is this, which amounts to a long comment on the calculus, as that's really where I feel the problem lies: how the calculus - math - is often taught, versus how it ends up working in practice. More specifically, the problem is the integral notation, and it hides what is really going on. You see, in a way, either you can say this notation
$$\int r^2\ dm$$
is a lie (note the lack of bounds, too!), or you can say that the way we typically told we should write integrals
$$\int_{a}^{b} f(x)\ dx$$
is a kind of lie, and its this latter one that I'd like to talk about here. But in either case, there's some lying going on. And what I want to do is motivate here, with a long shpiel, another notation, that we could use to make clear what is really going on.
In particular, the "$d$" thing is not actually simply something that designates "what variable you integrate with respect to". That interpretation is too simplistic and doesn't cover all the use cases, as you've just encountered.
It's hard to give a formalism for just what the "$d$" thing is, and there are quite a variety of formalisms available - for example, one of these is what is called differential forms, another is called non-standard analysis, a third is called measure theory - so without going into too many details, what I would say is to stick to an intuitive definition only for now, which after all is important because otherwise the formalism will seem like dry rules:
$dx$ (and $dm$, etc.) are increments, or functions thereof.
You see, an integral
$$\int_{a}^{b} f(x)\ dx$$
really is not too much different from a sum, i.e. from this:
$$\sum_{n=a}^{b} f(n)$$
That's why we use that funny symbol $\int$. It's an old script and italicized "s", for "sum". Just as $\sum$ is the Greek letter sigma, a Greek version of "s", again, "sum". If you look at some historical English writings, you may encounter the $\int$ symbol used as a letter.
The difference is that, while a sum adds up for each integer from $a$ to $b$ inclusive, an integral adds up for each real number from $a$ to $b$ inclusive. It doesn't add $f(x)$ at each $x \in [a, b]$, rather it adds $f(x)\ dx$ at each $x \in [a, b]$. $dx$ is a part of the thing being added up, not some auxiliary thereto.
The reason it's there is not to "just indicate what variable we integrate with respect to". You've just seen how this interpretation is too narrow and even leads one into trouble in practice. (For example, in integrals with multiple variables $dx$ and $dy$) Rather, to understand the need for it, let's take a closer look at this idea of "summing up at every real number".
The chief difference between the real numbers and whole numbers, for our purposes, is that in any interval of the latter, there are only finitely many, but in any interval of the former, there are uncountably infinitely many. And while we can add up a finite, or even countable, number of real summands to get a finite real result, we cannot add up an uncountable number of such summands, and get a finite result, unless all but finitely many of them are zero. But that, of course, would be no different than a usual sum, so it's kind of trivial.
Hence, to get something non-trivial, we must create a third option: what if we add up a bunch of things that are so tiny, that they are smaller than any real number, yet not zero? Such things are not real numbers - hence all the different formalisms I just mentioned - but we will in effect add them up and pretend the output we get is a real number. That's why we use an $\int$, not a $\sum$ - the latter only takes real numbers in, the former takes these other things in and cranks real numbers out. But what are they?
Well, to make it more natural to where this comes from, suppose I write a sum as
$$\sum_{n=a}^{b}\ f(n)\ \Delta n$$
where this is not a different notation. Instead, $\Delta n$ is just another numerical factor we've added in to the usual summation. What it is is the size of the increment as $n$ jumps from one whole number to the next whole number. We just don't typically write it, because it is $1$. But note that if I put the increment of another variable that was not changing, i.e.
$$\sum_{n=a}^{b}\ f(n)\ \Delta m$$
I'd get a sum of 0, because those increments are 0. Hence it seems "$\Delta(...)$" "selects" what variable we are integrating with respect to, but it isn't actually what determines it: what determines it is what's under the sum symbol.
In
$$\int_{a}^{b} f(x)\ dx$$
$dx$ is the increment as $x$ jumps from one real number to an adjacent real number - except there is no such thing, so we have to come up with some more complex formalism to try and pretend there is in some way, one of the most elementary being the Riemann (or Darboux, as you point out) sums, where the limit allows us to shrink a finite increment down. We use this increment as it provides a convenient damping factor. But there's no reason we could not use something else, provided it were also "suitably small" - without getting too deep into formalisms.
And so, what my point here at the end is is, is that we really should notate the variable of integration explicitly, i.e. we should not write
$$\int_{a}^{b}\ f(x)\ dx$$
but we should really write
$$\int_{x=a}^{b}\ f(x)\ dx$$
which makes the variable explicit, just as with sums. Now there's no problem! We could change out $dx$ with anything else, or even get rid of it altogether (in which case the integral will be infinite unless $f(x)$ is zero at all but a countable number of points).
In the case of
$$\int r^2\ dm$$
what we are integrating with respect to is NOT the variable $m$, but instead a point, $P$, within the region occupied by the object, $O$, a set of points in $\mathbb{R}^3$. Hence, we should really write
$$\int_{P \in O} r^2\ dm$$
Now $dm$ is an infinitely small amount, that has nothing to do with the variable of integration. The variable of integration is really $P$, a point on the object. $dm$ is the point-sized amount of mass "just around" $P$, that would be zero but isn't quite. So we're adding up all the tiny, uncountably many moments of inertia $dm r^2$ contributed by each little bit of mass around each point $P$ in the object. And we add up all those little tiny masses at each point in $A$. If you want,
$$\int_{P \in O} [r(P)]^2\ dm(P)$$
would be even more explicit, as $dm$ is different at each particular point.
So you aren't "integrating with respect to mass". You're integrating with respect to a point within the object. What you are integrating, though, involves infinitely small masses at each point therein.
And
$$\int_{0}^{M} (...)$$
notation? That's silly. That's what happens when you take the "$d$ means 'what to integrate with respect to'" business too far, from this perspective.