There's more gravitational force in our galaxy (and others) than can be explained by counting stars made of ordinary matter. So why not lots of dark planetary systems (i.e., without stars) made of ordinary matter? Why must we assume some undiscovered and unexplained form of matter?
There is a very precise reason why dark planets made of 'ordinary matter' (baryons - particles made up of 3 quarks) cannot be the dark matter. It turns out that the amount of baryons can be measured in two different ways in cosmology:
- By measuring present-day abundances of some light elements (esp deuterium) which are very sensitive to the baryon amount.
- By measuring the distribution of the hot and cold spots in the Cosmic Microwave background (CMB), radiation left over from the early universe that we observe today.
These two methods agree spectacularly, and both indicate that baryons are 5% of the total stuff (energy/matter) in the universe. Meanwhile, various measures of gravitational clustering (gravitational lensing, rotation of stars around galaxies, etc etc) all indicate that total matter comprises 25% of the total. (The remaining 75% is in the infamous dark energy which is irrelevant for this particular question).
Since 5% is much less than 25%, and since the errors on both of these measurements are rather small, we infer that most of the matter, about 4/5 ths (that is, 20% out of 25%) is 'dark' and NOT made up of baryons.
As above answers have mentioned most of the ordinary matter has been considered as candidates and we are fairly certain that there has to be some sort of "dark" matter at work.
Firstly, we take on the phenomenon of gravitational lensing. A very famous example is the Bullet cluster where you can clearly observe the effects of a compact mass acting as an optical lens. A mass of such magnitude can not conveniently be a cluster of brown dwarves and most black holes we observe have many objects orbiting and a particle jet accompanying. Besides, a black hole of small magnitude would not be enough to explain the magnitude of the gravitational lensing in question.
Secondly, much more complicated computations made with general relativity principles need a lot more mass to account for the current shape and angular speed of galaxies. It may be easy to say that GR is wrong(which you shouldn't say to the face of a physicist), even though GR is incomplete in the sense that it is not a theory of everything, it still explains most of the gravitational phenomena pretty well. Besides, the fact that there is gravitational lensing means that there really IS some sort of mass or gravitational pull(or more like space time curvature anomaly) in certain parts of our galaxy, and the universe.
Even though dark matter seems so mysterious, we can guess most of its properties from our "lack of knowledge". Firstly, since it is "invisible" in all spectrum of light, we can assume that it does not interact via electromagnetic force. Since it is a fountain of gravitational force, we can say that it, surprise surprise, interacts via gravitational force. Calculations for weak&strong forces are pretty complicated and very indirect so I'm just going to say that most of the currently proposed dark matters interact via strong and weak forces as well.
A planetary system without a star is only possible if the star died out to be a neutron star, black hole, or a white dwarf, all of which are detectable in some way.
Besides, there would have to be an unrealistic amount of these "faded" star systems to even account for the lost mass. And assuming that such planetary systems are fairly rare, we would rather consider an unusual form of matter which is quite possible and plausible if found.
All the matter that we do know to exist (called Baryonic matter) emits some kind of electromagnetic radiation at some frequency. Sometimes it's measured in infrared radiation, because matter, no matter how cold, will still radiate some amount of heat. To the best of our knowledge, it's not actually possible to cool any matter to absolute zero, and it's certainly not happening naturally. I believe the coolest matter known to exist in the universe is around 3 degrees Kelvin. Other wavelengths can determine the exact nature of the matter in question, and its temperature. For example, ionized oxygen glows in visible light at a certain wavelength - that's one of the things that make emission nebulae visible.
So, after measuring all that matter in other galaxies, we've noticed that there's simply not enough mass to keep them from falling apart (namely, stars within galaxies are moving faster than the escape velocity of the gravitational pull of the center of mass of the galaxy) - especially when we've tried to make computer models of galaxies. That was the first clue that there was something going on that we didn't fully understand. As others have described, other methods of determining the total mass of a galaxy have shown similar results.
As a result, there's only one possible explanation for this phenomenon: there must be some kind of matter (and all matter has mass) that we can't detect. In fact, the amount of mass that can't be accounted for in this way is about 95% of the mass of any given galaxy. Of course, that's a pretty big deal.
By the way, we've been trying to work out this very large discrepancy since 1933. The difference between now and then is that accuracy of our measurements of velocity of stars in other galaxies has improved, as well as our ability to measure other phenomena happening within galaxies and galaxy clusters. The more accurate we've become in our measurements, the more this glaring discrepancy has become evident.
The possibility of large dark objects made of normal baryonic matter has been considered. These are called MACHOs.
However, there are various reasons to think that most of the dark matter can't be in the form of MACHOs. From the above wikipedia article (which links to some relevant journal articles):
The Big Bang as it is currently understood simply couldn't produce enough baryons without causing major problems in the observed elemental abundances, including the abundance of deuterium. Furthermore, separate observations of baryon acoustic oscillations, both in the cosmic microwave background and large-scale structure of galaxies, set limits on the total baryon-to-total matter ratio. These observations show that a large fraction of non-baryonic matter is necessary regardless of the presence or absence of MACHOs.
The major missing link is the existence of a class of particles, which aside from gravity are sufficiently weakly interacting to fill the need. We already know of one kind of particle which has most of these characteristics, neutrinos. In this case physicists and cosmologists have theoretical reasons to believe the upper bound on the amount of mass in neutrinos is too small. But the mere existence of one class of weakly interacting particles should make the possibilty of another appear less novel.
You can estimate total mass using gravitational lensing and compare it to the estimated mass of a galaxy by adding up the masses of all the visible stars and making generous estimates of other kinds of ordinary matter that could be there. There is still a big discrepancy.
If the extra mass were ordinary matter, it would clump and it would interact with photons, both of which would lead to a much less diffuse distribution than the one that must be assumed to get the galaxy dynamics right.
As an expansion on the other answers, there is also a role that astrophysical simulations have played in ruling out known particles from explanations of dark matter.
First, a clarification: there is a distinction to be made between the non-baryonic matter addressed in most answers and "undiscovered and unexplained forms of matter." Reasons that most of dark matter must be non-baryonic include arguments involving the CMB and Big Bang nucleosynthesis, as mentioned by others. However, non-baryonic (but known matter) such as the known flavors of neutrinos seemed like fair candidates for dark matter even after baryonic matter was largely ruled out. Later, neutrinos were ruled out by other sources of evidence.
One of the early sources of such evidence ruling out neutrinos was actually computer simulations of large-scale structure formation. In an interesting historical essay written by Simon White (and posted today, 6/19/18 on arXiv):
... the large voids [found in simulations of neutrino-dominated universes] in the galaxy distribution were incompatible even with the relatively meagre observational data available in 1983. This discrepancy led to the abandoning of the known neutrinos as potential dark matter candidates, even though it would be another two decades before they were finally excluded by experimental upper limits on their masses. The demonstration that no known particle can account for the dark matter remains one of the most significant contributions of computer simulations to astrophysics and cosmology.
See the essay here.