It's important to keep in mind that the International System of Units (SI) was created for business and industry and their impact on humans, and that remains its primarily purpose. The candela is a good example of that. There's nothing fundamental about the candela with regard to physics. There is however something fundamental about it in terms of human physiology. As a side note, sound engineers have concepts analogous to the candela with regard to sound. These sound units based on the behavior of the typical human ear haven't made it into the SI, but that's primarily because sight is more important than is sound for most humans.
While the SI exists primarily to make our everyday life better, it is based primarily on quantum physics. (This role will be strengthened in a couple of years when the BIPM accepts the proposed redefinition of the SI.) The mole played an important role in relating mass at the atomic/molecular level to mass in the macroscopic world of everyday life. This will no longer be the case in a couple of years. The mole will be an appendage. Like the candela, the mole is not fundamental.
That leaves five SI base units that might have a basis in physics as denoting fundamental dimensionful quantities: time, mass, length, current, and temperature. Temperature is a measure of energy per unit component per degree of freedom. Temperature isn't quite fundamental. Current (or more fundamentally, charge; the SI uses current rather than charge because current is more easily and more accurately measured at a macroscopic scale) similarly falls by the wayside as being a fundamental dimensionful quantity.
That leaves just time, mass, and length as possibly being fundamental dimensionful quantities -- and there are some physicists who debate even this.
