I have just started reading nuclear physics.I know that the sum of masses of the quarks is less than the proton or neutron itself as a whole . But why is it that the sum of the masses of the nucleons(protons and neutrons) is more than the nucleus itself? whats the difference between the two cases?
The difference comes from the kind of force that holds the constituents together. The force on the quarks in the nucleon is the color force, (one of the four fundamental forces), between nucleons, it is the residual color force, which appears as the strong force that binds the protons and neutrons in a nucleus.
The color force and the fact that the coupling constant is 1 in the color interactions between quarks and gluons is what creates the nucleons, protons and neutrons, with the exchange of gluons in a "bag" where a sea of quarks antiquarks and gluons binds everything together to the mass of the nucleon. Within a nucleon the color force acts as a spring , binding the constituents together. The invariant mass of the whole depends on the sum of the four vectors of the numerous temporary constituents and thus to a very small degree on the masses of the three constituent quarks which define the baryon quantum number.
The force between protons and neutrons is a spill over force, similar to the van der Waals forces that hold molecules in structures , it is attractive still but it is short range . It has been modeled by Yukawa as pion exchange with an exponentially falling effect. When the protons/neutrons are close enough, within the range of the Yukawa force, they can be bound: As it is attractive it has energy levels in which protons and neutrons can be bound giving up energy with the special relativity effect of the sum of the bound masses to be less than the free ones.
The sketch is an attempt to show one of many forms the gluon interaction between nucleons could take, this one involving up-antiup pair production and annililation and producing a π- bridging the nucleons.
(Feynman diagrams for strong interactions have to be taken with care, as with a coupling constant of 1 , perturbative expansions, which is what Feynman diagrams are about, do not converge. New methods of calculations have been found.)