According to the Third Newton's law of motion:   

> For every action there is an equal and opposite reaction.


So, I understand that if I hit a brick wall with $50\, \mathrm{lbs}$ of force, the brick wall also hits me with $50\, \mathrm{lbs}$ of force (usually painfully). In this instance, Newton's third law makes sense. What I'm confused about is, if I hit a patch of drywall with $50\, \mathrm{lbs}$ of force, it's probably going to break, and due to the lack of pain in my hand, I can tell it did not hit me back with $50\, \mathrm{lbs}$ of force. 

How does Newton's third law apply to situations when one object or the other is destroyed? It certainly seems like at that point it is incapable of delivering the full force of my blow back to me. What happens with the energy?