There are different sources of noise, and many different contexts that noise occurs, and when one says there is, "no such thing as a perfect one-way valve", one needs to be more specific about the particular context one is referring to. If one uses a one way valve where the input is a high pressure system, and the valve is isolating the high pressure from the low pressure system (the output side), then the proper context of a valve not performing perfectly is that there is a leakage of high pressure to low pressure and not really that there is low pressure leaking into the high pressure (although there are those who would probably argue its a matter of convention, much like their is an argument about the conventional flow of charge and electrons).
Noise in a thermodynamic system is closely linked to the concept of entropy which relates to the possible configurations a system can be in based on its macroscopic state. If we wanted to think in terms of a leaking valve, there are real mechanical issues such as back flow and the related concept of back pressure.
In any case, if we recognize the idea that molecules are what is flowing in a mechanical system, then there are configurations that are consistent with the macroscopic state of the system where some molecules are on the opposite side of a valve. Through any multitude of mechanisms, there is some small probability that some molecules will find themselves on the wrong side of the valve. Again, this is best understood in terms of entropy. As the disorder increases (entropy increases), energy must be added back into the system to return the entropy to its beginning value. The best way to understand this is by studying the Carnot cycle in thermodynamics.
Noise in information theory (and effectively electronics) is understood best through the works of Claude Shannon and other pioneers. It is important to understand that noise is usually a term associated with unwanted effects.
As far as to the fundamental nature of noise, we could certainly argue that we accept noise as fundamental, but why do we do so? The only real statement we have is the second law of thermodynamics and that is a postulate (or axiom in modern terms). This is something we hold true in principle, and we hold it to be true because it has never been violated, however it also closely related to the central limit theorem which has been proven to be true mathematically.