A question in Serway's book asked the density of proton and to compare it to osmium's density. I found 2.88*10^16 kg/m^3. Isn't it so much dense ? I mean, 10^16. What is the reason behind it's density's bigness ?
As noted in a comment "why" questions are impossible to answer in physics , but if you ask how protons (and neutrons) can be so dense, the answer is found in the characteristics of QCD (the theory that describes how quarks are bound together by gluon exchange to form nucleons and also other hadrons).
It has already been noted that the QCD force is very strong compared to the electromagnetic force. The QCD force is more than just strong, however. It has a property that no other elementary force possesses. It actually becomes stronger if you try to move two quarks further apart than about 1 fm. No other force of nature has this property. Because of this it is incorrect to refer to it as short ranged. That designation is reserved for forces that fall off exponentially as distance increases. To say that a force is long ranged means that it falls off inversely with $r^2$.
The QCD force is neither long or short ranged. It becomes so strong when a quark is pulled from a nucleon that a quark/antiquark pair is created. The antiquark bonds with the removed quark to form a meson while the quark from the pair joins with the remaining quarks to restore the original nucleon. That is how nucleons can be so dense and quarks can never exist as free particles outside hadrons. BTW, most of the mass of nucleons arises from these strong binding forces rather than from quarks themselves which have relatively low masses.
Well you're not comparing similar things, so you can't expect similar results.
The density of osmium ( or any conventional solid material ) is governed mainly by the way it is bound together. The forces governing that are much weaker than than the extremely strong forces that bind the parts of a proton together.
A proton is, we believe, made up of three much smaller particles called a quarks. Quarks are elementary particles and while it's very hard to describe them in a small note, there is a forced between quarks that is much, much stronger than the electromagnetic force due to the electric charges they carry. This forces keeps the proton very, very small.
But those forces are very short range and they have no effects at all at the kind of distances than electrons orbit nuclei. It is the way electrons act in a solid that keep it bound together, and those forces are very weak, as we're really dealing with atoms ( very large compared to a proton ) bound together weakly.
An atom is just a very compact nucleus surrounded by a bunch of electrons.
So on one scale you have a small number of very strongly bound quarks in a proton that keep it small, and on the other a much weaker force binding atoms together.
Strictly speaking it is not possible to say a proton has a definite size, but we can give an average size as a rough measure. The same is strictly speaking true for atoms, but we have an easier time defining a way to make a consistent size measurement for them. There are in fact many ways to define the size of an object, but in terms of orders of magnitude they tend to give similar results.