Three reasons:

1. As you correctly point out, black dwarfs are "hypothetical objects". There has been insufficient time since the first stars were born for white dwarfs to cool below about 3000 K. i.e. Whilst there are faint white dwarfs with luminosities below a few $10^{-5} L_{\odot}$, they are not invisible.

2. Microlensing experiments rule out "massive compact halo objects", like cold white dwarfs or black holes as a significant contributor to dark matter.

3. Most of the dark matter needs to be non baryonic and to interact very weakly with normal matter in order to form the structures that we see today in the universe; and to reconcile the inferred primordial abundances of helium, deuterium and lithium with the total amount of matter deduced to be in galaxies and clusters of galaxies. Cold white dwarfs are baryonic, so cannot represent the bulk of dark matter.

Three reasons:

1. As you correctly point out, black dwarfs are "hypothetical objects". There has been insufficient time since the first stars were born for white dwarfs to cool below about 3000 K. i.e. Whilst there are probably some old, faint (massive) white dwarfs with luminosities below $10^{-5} L_{\odot}$, they are not completely invisible and there also seems no reason why their progenitor stars should have been more widely distributed in a halo around the galaxy than the stars we see today.

2. Microlensing experiments rule out "massive compact halo objects" with $10^{-6}<M/M_\odot<100$, like very cold white dwarfs or black holes, as a significant contributor to dark matter (see for example here).

3. Most of the dark matter needs to be non baryonic and to interact very weakly with normal matter in order to form the structures in the galaxy clusters and in the microwave background that we see today; and to reconcile the inferred primordial abundances of helium, deuterium and lithium with the total amount of matter deduced to be in galaxies and clusters of galaxies. Cold white dwarfs are baryonic, so cannot represent the bulk of dark matter.