The reflection could viewed as a two step process. The incident wave causes the electrons in the silver to vibrate like in an antenna. Though by vibrating they also emit the same light. So it's the electrons at the surface of the silver that reflects the incoming wave. As you mentioned the wave is part electric and part magnetic, but these cannot be taken apart they are each others cause and effect without one the other wouldn't be there either, and therefore it must reflect both parts.
That silver (and all metals) don't distort is due to that they are also very good conductors. This prevents the electromagnetic waves from entering the object. The boundary conditions which must hold (from being an conductor) result in the perfect reflection and that the resulting angle is equal to the incident angle.
Similar boundary conditions are there for non-conducting materials like plastic and glass. These similar conditions result in reflection of glass and the shine/reflection on other smooth surfaces (though there can be other causes too). Also snell's law would follow from these boundary conditions.
In contrast to conducting materials it is possible to for electromagnetic waves to enter non-conduction objects. As a consequence part of the incoming wave is transmitted into the material. The propagation or dampening of the wave through the material is largely depended on the properties of the material. Some material like glass hardly dampen the wave and you can see through them, while others like most plastics dampen them and thus are opaque.