So, here's the thing. The chemistry that underlies molar mass ratios dates back at least to 1805. We've known that if you divide by a certain "relative mass" number you can get whole-number ratios for atoms in a pile of stuff, for that long. It took us about 60 more years to get a handle on how large atoms were with the estimations of Loschmidt, who worked out that atoms are much smaller than the wavelengths of visible light -- too small to ever "see". This gave a rough count of how many atoms there were in a confined space, too -- but we weren't able to connect these two different quantities (atomic relative masses, count of atoms) together to figure out the mass of a single atom until some work done by Einstein on diffusion in Brownian motion (1905) and some concrete numbers could finally be rolled in with Millikan's oil-drop experiment (1910).
So due to history and convenience, the chemists are basically at the level of saying, "okay, we have N grams of this stuff, our mass spectrometer says that it's M grams per mole, so we've got N/M moles, that includes N/M moles of nitrogen and 15 N/M moles of hydrogen due to the known atomic composition, ..." and so on. You never have to add the uncertainty in Avogadro's number to these calculations; the "size" of a mole isn't important. It's only important when you start to want to know things that are "beyond" historical chemistry approaches, like counting actual numbers of atoms.
With all that said, you'll be heart-warmed to know that there is a unit revision being considered by the SI organization, and one of the proposals is to fix the number of atoms in 1 mole. But of course they will still use as a guideline that "1 mole of carbon-12 weighs exactly 12 grams"; it will just transition from what is now "exactly" to what will be "almost exactly."