Because surfaces aren't smooth on the atomic scale, when you touch two surfaces together only the high points (asperities) on the surfaces make contact so the real area of contact is much smaller than the apparent area of contact. I'm guessing you know this, hence your comment in your second paragraph:
and there would be more peaks
The reason why the force is approximately independant of area is that as you increase the force you get elastic deformation of the asperities and the real area of contact per asperity increases. If you increase the area you increase the number of points of contact but you also decrease the force per point of contact so the asperities deform less and the real area of contact per asperity goes down.
The real area per asperity is roughly proportional to pressure, i.e. $F/A$, and the number of asperties in contact is proportional to area $A$. So when you multiply these together you find the real area of contact, and hence the friction, is just proportional to the applied force.
For relatively soft materials like rubber the approximation doesn't hold, and their frictional behaviour is a lot more complex.