Newton's third law of motion says that the internal forces in a system cancel out each other --- what is the system here? what can be considered as a system? and why do the forces cancel out each other's effects?
Consider a system of two cars colliding. The system consists of two cars. That's what Newton means.
Internal forces are any forces due to bodies inside the system, hence the name. Although for each car, there is a net force due to the other car (as they just collided with each other and there is a contact force), for the system as a whole, nothing really acts on it.
If this is the case, momentum within that system is conserved.
If there is an external force (a force by anything that is outside the system acting on bodies in it), momentum conservation law fails.
what is the system here? what can be considered as a system?
A system can be anything.
For example, if we consider the Earth and Moon as one system, then their mutual gravitational attractions are internal forces, which cancel themselves out. If we considered only the Moon as the system, then the gravitational force from the Earth is not internal anymore.
and why do the forces cancel out each other's effects?
Because, by Newton's 3rd law, whenever there is a force present then there is also always a reaction force of the same size but opposite. Naturally, these cancel each other out if they are both acting on a system.
Those two forces do not act on the same bodies, though. If you kick a bowling ball, then the ball feels your kicking force, while your foot feels the ball's reactions force. But if you consider both ball-and-you as one system, then both forces are included in the system. Each object feels a non-zero net force, making them accelerate (the ball starts moving, while your foot is slowed down). But the system as a whole doesn't feel any net force. It doesn't accelerate. The two parts that the system consists of (ball and you) are accelerating, but the centre-of-mass of this ball-and-you system doesn't accelerate at all.
There are many general misconceptions about Newton's 3rd law.Read the statement of 3rd law in simple terms:-
"If a particle A exerts a force F on a particle B,then particle B would exert a force --F on A".
We don't know why particle A would exert a force and in counter attack particle B would exert a equal and opposite force.It is a law and we have to follow it just as we follow definitions in mathematics.
So,this law never talks about systems,it talks about particles. You can take anything as a system and a force which is applied by particle of the system on other particle is internal (for multi particle system). It is a consequence of 3rd law of motion that internal forces cancel out in the system. A force applied by an agent other than the system is external force for the system.I am saying again that internal forces cancel each other in a multi particle system as a consequence of 3rd law and we have to follow it. A research is being done in LHC for the proof of statement of 3rd law on gravitons.