For every action there is an equal and opposite re-action.
                             -3rd law of motion

So, I understand that if I hit a brick wall with 50 lbs of force, the brick wall also hits me with 50 lbs of force (usually painfully). In this instance, newtons third law makes sense. What I'm confused about is, if I hit a patch of drywall with 50 lbs of force, it's probably going to break, and due to the lack of pain in my hand, I can tell it did not hit me back with 50 lbs of force. How does newtons third law apply to situations when one or the other object is destroyed? It certain seems like at that point it is incapable of delivering the full force of my blow back to me, so what happens with the energy?