Three reasons:

1. As you correctly point out, black dwarfs are "hypothetical objects". There has been insufficient time since the first stars were born for white dwarfs to cool below about 3000 K. i.e. Whilst there are faint white dwarfs with luminosities below a few $10^{-5} L_{\odot}$, they are not invisible.

2. Microlensing experiments rule out "[massive compact halo objects][1]", like cold white dwarfs or black holes as a significant contributor to dark matter.

3. Most of the dark matter needs to be non baryonic and to interact very weakly with normal matter in order to form the structures that we see today in the universe; and to reconcile the inferred primordial abundances of helium, deuterium and lithium with the total amount of matter deduced to be in galaxies and clusters of galaxies. Cold white dwarfs are baryonic, so cannot represent the bulk of dark matter.


  [1]: https://en.m.wikipedia.org/wiki/Massive_compact_halo_object