For example, if you had a thick piece of glass and it had a kind of reservoir inside with half water and half space allowing the water to move but not escape from the inside of the glass, and then you boiled the entire piece, would the water inside evaporate?
No, water doesn't need oxygen to evaporate. All condensed matter (solids, liquids) tends to evaporate spontaneously, although the rate may be very low. Equilibrium is reached when the pressure in the gas phase equals the so-called vapor pressure. Here's what the vapor pressure looks like as a function of temperature for water:
Note that atmospheric pressure, or 760 mmHg, corresponds to the boiling temperature, 100°C. This isn't a coincidence; it's how the boiling point is defined: the temperature where the vapor pressure equals the ambient pressure. Bubbles form during boiling because the evaporative pressure is large enough to push the liquid out of the way.
So for your thought experiment, if you start with a closed compartment half-filled with water, you should expect the pressure of the water vapor above the liquid to increase to about 0.03 atmospheres at room temperature. If you then heat the water, some additional liquid will evaporate, and the vapor pressure will increase as shown in the plot above. Because liquid is much denser than gas, most of the water will remain, though, at least at 100°C. This is the difference between your closed compartment and an open pan of water, for example, where the saturated vapor will tend to convect away and liquid will continue to boil until the pan is dry.