I absolutely agree with Erlend's answer (RE the noise floor of a telephone application.) I also agree that a noise floor of 20 dB is remarkably low. The noise floor on my professional sound level meters are 14 dB and 5.5 dB, respectively. The latter is a special low-noise system.
However, here's an attempt to answer your question about why a reference may list "very silent room" as more energetic than "leaf noise". Outdoor environments represent a closer approximation of free field conditions, where sounds propagate outward without reflection.
Enclosed rooms are usually representative of the reverberant field, with reflections off walls and objects in the room. Acoustic energy stays in the room longer and results in higher values of measured pressure for the same source power.
For example: when I stop typing, one-second equivalent sound pressure levels ($L_{Aeq, 1s}$) in the center of this room are between 30 - 40 dB. Most of the energy is coming from my computer's fan and the ballast of the fluorescent light. Outside on a still day these sources would contribute far less energy to a measurement at their same distance.
This may provide an explanation for why a handbook would list even a "very silent" room as being more energetic than an outdoor environment with some motion.
Frankly, I think these kinds of generalizations are far too broad, so I try to avoid them in my work. The prediction of ambient level is dependent on both the form and material structure of a space as well as the sources within/nearby. The sheer variety of combinations possible suggests that measurements are highly preferable to reliance on listed values. This includes an awareness of noise floor limitations, as Erlend's answer emphasizes.