What makes you think that the maximum force you applied to the dry wall was anything like the maximum force you applied to the brick? It certainly wasn't. The dry wall gave way much before you were able to attain the same force as applied to the brick. Try punching the air and see how much force you are able to apply. The experimental evidence that the force you applied to the brick (and it applied to you) was higher than the force you applied to the dry wall was your injured hand.

What makes you think that the maximum force you applied to the dry wall was anything like the maximum force you applied to the brick? It certainly wasn't. The dry wall gave way well before you were able to attain the same force as applied to the brick. Try punching the air and see how much force you are able to apply. The experimental evidence that the force you applied to the brick (and it applied to you) was higher than the force you applied to the dry wall was your injured hand.