My understanding is that all specular reflection is due to Fresnel reflection at the interface which is a broadband effect.
A classic example might be a white reflection on a polished ball (e.g. pool ball, or apple, etc.). The highlight is almost always white since it is a broadband reflection of the light source, regardless of the colour of the ball. The pigment or colour of the object plays no role in the specular highlight.....
The diffuse reflection of the object can have colour since this is due to absorption of the pigment in the object.
So what about metals? For example, a polished gold sphere. Is the specular highlight white or gold in colour?