Black and white matters. But why and how? I know black conducts heat while white reflects it.
But they are colors after all.
If a metal is painted black, it conducts more heat or at a rapid speed than it would do before it was coated.
But, as far as I know, colors don't have any special "substance" in them, which might trigger the sudden absorption of heat or reflection of the same.
What is the physics behind this? Are colors by themselves, some catalyst kinda thing? 
 A: 
But, as far as I know, colors don't have any special "substance" in them, which might trigger the sudden absorption of heat or reflection of the same.

This statement appears to be the crux of your confusion, and it is false.
When you paint an object black, or white, or any other color, you coat that object with a thin layer of a substance which, in fact, absorbs particular wavelengths of electromagnetic radiation, and reflects others.  The color you see is a consequence of the coating doing this, and which color is dependent entirely on the chemical composition of the paint.
Now, heat is not electromagnetic radiation.  Heat is random molecular motion, which can both cause, and be caused by, EM radiation.  All wavelengths of EM radiation can transfer heat, but some are better at it than others.  Infrared light is sometimes called heat radiation because it is particularly good at transferring heat within our familiar Earth-surface environment, because many substances found in this environment strongly absorb it.  In particular, paint which is black (== paint which strongly absorbs all EM wavelengths corresponding to "visible light") will probably also absorb infrared light.
And finally, heat doesn't have to be transferred by electromagnetic radiation.  Heat can also transfer by conduction between two objects in direct contact: molecules of the hotter object randomly bump into molecules of the colder object and transfer some of their energy.1  This process is not dependent on what wavelengths of EM radiation are absorbed by the colder object.  This is the primary way paint heats up a painted object, and this is why the underlying color of the object doesn't matter.
1 technically this process does also involve exchange of "virtual" photons, but this is a detail which is usually ignored for macroscopic objects.
A: 
I know black conducts heat while white reflects it.

The correct term is "black absorbs light while white reflects it".
We have named colors of  light we see in the visible spectrum .
White reflects most of the energy falling from the visible spectrum, black absorbs it. When the energy of light is absorbed it turns into heat . Any material painted black will absorb this heat further  and its temperature will be raised but it will depend on the material how far the heat is transferred. If it is metal painted black, metal is a good conductor of heat and will distribute the energy fast on the whole body.

But they are colors after all.

They change the surface properties of materials on which they are painted thus changing the ability of absorption and emission of radiation.
The energy coming from the sun covers a much larger  electromagnetic spectrum than the visible. The visible  has  about half  of the energy coming from the sun on the surface, as seen in the link.
So a metal door in the sun will transfer the heat of the visible spectrum to the interior if painted black, will reflect it back and keep the interior cooler if painted white. It is a good  reason for painting  roofs and walls white in hot countries. A white car is also better in hot countries for this reason .
It is not always sure that the color properties ( absorption/reflection)  are followed by the invisible part of the sun spectrum, infrared or  ultraviolet. Each paint has to be studied as far as its response to the impinging radiation to be used efficiently for thermal protection.
A: I believe that there are some incomplete/incorrect assumptions in this question:  the bulk thermal conductivity of a metal will not be affected by a surface coating; it's response to radiative heating will be affected though.
This part of the difference in thermal response to radiative heating is based on the equality of emissivity and absorbtivity:  white materials absorb little, and thus emit little radiation, conversely darker materials absorb and emit more electromagnetic radiation.  You can see that this must be the case by considering putting the objects inside a "black-body cavity" held at a fixed temperature.  Once the object reaches the equilibrium temperature -- for each amount of energy absorbed from the black body radiation filling the oven, and equal amount of energy has to be emitted.  Thus absorbtivity=emissivity.
A: You have it backwards. You're coming from the point of view that being black makes something good at absorbing radiation and being white makes it bad and asking why this should be so. It's exactly the other way around: being good at absorbing radiation (in particular, visible light) makes something black; being bad at absorbing radiation (i.e., good at reflecting it) makes something white.
Now, you can ask what makes something good or bad at absorbing radiation but that's a whole 'nother question.
A: 
I know black conducts heat while white reflects it.

Black objects don't "conduct heat". Black objects absorb incoming radiation in the visible range. Similarly, white objects don't reflect heat. They diffusely reflect incoming visible radiation.

But they are colors after all.

Yes and no. Whether black or white are "colors" depends very much on what you mean by color. I'll leave that debate for a different question. For the purpose of this question, it's better to look at black and white as shades of gray rather than as colors like red and blue.

What is the physics behind this?

This is the question I'll address in detail. The answer lies in the concepts of emissivity, absorptivity, reflectivity, and transmissivity.


*

*Emissivity is the ability of an object to emit thermal radiation, relative to that of an ideal black body.

*Absorptivity is the fraction of incoming radiation absorbed by an object.

*Reflectivity is the fraction of incoming radiation reflected by an object.

*Transmissivity is the fraction of incoming radiation that passes through an object


The latter three (absorptivity, reflectivity, and transmissivity) completely enumerate what happens to incoming radiation. They add to 1 (or to 100% if you want at those as percentages). For the rest of this answer, I'll assume opaque objects, where transmissivity is zero. Incoming light for opaque objects is either absorbed or reflected, in the ratio determined by the object's absorptivity and reflectivity (which add to one).
Reflectivity and absorptivity explains in part why black objects get hotter than do white ones. A perfectly black object absorbs all incoming visible radiation, while a perfectly white object reflects all incoming visible radiation. As there is no such thing as a perfectly black or perfectly white object, all objects absorb incoming visible radiation to some extent. However, black objects absorb considerably more incoming visible radiation than do white ones.
The flip side of the coin is emissivity. An object will eventually come to thermal equilibrium, with the energy absorbed from incoming radiation being equal to the energy emitted as outgoing radiation. The outgoing radiation is a function of the object's emissivity $\epsilon$, it's temperature $T$, and it's surface area $A$, dictated by the Stefan Boltzmann equation $E_{\text{out}} = A \epsilon \sigma T^4$ where $\sigma$ is the Stefan Boltzmann constant. The incoming radiation is a function of the incoming energy flux $\phi$, the object's absorptivity $\alpha$ and it's cross section to the incoming radiation $A_c$: $E_{\text{in}} = A_c \alpha \phi$. Equating and solving for temperature yields $T= \left( \frac {\alpha}{\epsilon} \frac{A}{A_c} \frac{\phi}{\sigma} \right)^{1/4}$.
Note that only the first factor in the above, $\frac {\alpha}{\epsilon}$ depends on composition. The other two factors represent geometry ($\frac A {A_c}$) and incoming energy ($\frac {\phi}{\sigma}$). Per Kirchhoff's Radiation Law, emissivity and absorptivity at any given frequency are equal. For an ideal gray body, both absorptivity and emissivity  are constant, independent of frequency and temperature. The ratio $\frac {\alpha}{\epsilon}$ is one for a perfect gray body. All perfect gray bodies with the same geometry and subject to the same incoming radiation will eventually reach the same equilibrium temperature.
So we need something else to explain why black objects get hotter than do white ones. The answer lies in the fact that absorptivity and emissivity are frequency and temperature dependent for real objects. Ideal gray bodies don't exist. They're nice approximations if applicable. "Black" and "white" refer to the reflectivity (and hence absorptivity) in the visible range. An object that is white visibly can be very black in the thermal infrared. An object that is visibly white but thermally black won't heat up as much as will an object that is visibly and thermally black.
