According to particle physics , every fundamental force has its force carrier particle. Photon is a force carrier particle of electromagnetic force but how does force gets transmitted when a iron is brought near magnet quantum mechanically ?
The detailed mechanism is described by the very complex Quantum ElectroDynamics theory (QED for short).
Now I looked for a relatively simple explanation or overview of this and found nothing I liked (or feel you'd be happy with). However I did find this very short segment of an interview with Richard Feynman (who was one of the key developers of QED) about this question and why it's so very difficult to explain to a non-expert. It's worth saying that if this guy can't explain it simply, I'd not even try - he knew more about this that I ever will and was much better at explaining it to non-experts. The short version is that it's hard to explain to people because there's no everyday, common sense knowledge they'll have that they could relate to QED.
This video may give you some sense of what's going on with magnetism and it's the closest thing I could find that attempts to explain things without burying someone in several large volumes of complex mathematics.
Why does magnet starts applying force when it is brought near iron but does not apply any force on any other substance ?
There's always an electromagnetic field, it's simply that some substances interact less with it than others. Some have no or practically no interaction with EM fields whereas some do.
Just a classical description of magnetism is quite involved and you can start here on WIkipedia.
The behavior of e.g. a compass near iron can be described using classical theories of magnetism without needing to get deep into quantum theory.
Why does magnets attract and repel each other from point of view of quantum and particle physics ?
Again this is the very complex theory of QED and it's not practical to try and explain it in a brief answer like this.
And why do they get attracted to iron objects from the point of view of quantum physics ?
Again it's simply that the ever-present EM field has a stronger interaction with some materials than others.