If we ignore reverberation, sound intensity follows the inverse-square law and falls off with the square of distance. Since distance does not depend on direction, if both of you are talking at roughly the same level than the speech "signal" at your ears will be roughly the same. The noise, however, is closer, and hence more intense, to the person in the noisy room. This means the signal-to-noise ratio (SNR) is lower for the person closer to the noise source. Our ability to both hear sounds and understand speech depend critically on the SNR.
Consider the following worked example:
If both talkers are speaking such that the sound level 1 m away is 80 dB SPL, then the sound level 10 m away will be 60 dB SPL.
If the sound level of the noise in the room, 1 m from the center of the room is 80 dB SPL, then the sound level 10 m away will be 60 dB SPL.
Assuming the two talkers are 10 m apart and one is 1 m from the center of the room and the other is 10 m from the center of the room, then at each listeners ears the signal level will be 60 dB SPL. The noise level at ears of the listener near the center will be 80 dB SPL, giving a SNR of -20 dB which will mean the speech is inaudible and unintelligible. The noise level at the ears of the listener far from the center of the room will be 60 dB SPL and the SNR will be 0 dB which will make the speech both clearly audible and nearly perfectly intelligible.