In a previous question about layered green houses (like matryoshka dolls) How hot would it get if you put a greenhouse inside of a greenhouse inside of a greenhouse (like those Russian matryoshka dolls)? we came to the conclusion that the inner greenhouse(s) would stay at the same temp as the outer or perhaps be colder? If I understand correctly this is because the outer layers basically attenuate the light, "So for all intents and purposes the smaller greenhouse may be considered to be directly exposed to sunlight (albeit with enhanced reflection since the sunlight has to pass through two glass walls)." And the inner room would exchange heat back and forth with the outer room tending to bring it into thermal equilibrium - well what if we halt that trend as much as possible by making every other layer of the doll a vacuum insulator? (assuming the whole thing doesn't blow up due to the pressure differences lol, maybe you'd have to use a different insulating strategy or thick thick glass, but that would attenuate more light so I thought it would be easier to answer using the assumption that it just doesn't blow up) The way I see it is there are two variables for the inner green house, how much energy goes in, and how much energy goes out, if in is greater than out the thing accumulates energy and heats up. So to sum it up - is it physically possible to increase 'in' and decrease 'out' to the point of accumulating energy?
The principle of multi-layered insulation is well established in the context of radiative shielding - it even has a name: "Mulit-Layer Shielding"
Usually the idea is that if you have an object that is subject to radiative cooling, then you can reduce the amount of cooling by adding one or more layers of material; the material will reach some intermediate temperature because it will re-emit part of its heat towards the source, with only a fraction going to the "universe".
The same kind of thing works when you have multi-layered green houses. The glass gets warm, and becomes a secondary emitter of radiation. By using multiple layers, you can reduce the degree of emission - because the outermost layer of glass is the one emitting radiation to the universe, and it is "more insulated" from the inner layers.
With perfect insulation (only radiation transport) between the layers, you can definitely reach a higher internal temperature in the core of your greenhouse. Exactly how hot it can get depends on the geometry - the fastest way to increase the temperature is of course to increase the apparent size of the heat source (the sun) using mirrors; that will be much more effective than reducing the outflow of heat.
It might be worth doing a simplistic calculation. Let's assume your "glass" is perfectly transparent to wavelengths shorter than 800 nm, and perfectly absorbing for wavelengths longer than that. You may know that emissivity and absorbtion go hand in hand - so if it's a perfect absorber, it's also a perfect emitter at these wavelengths.
Now we can model the flux of the sun as a source of blackbody radiation; the outermost layer of glass will absorb all the IR, and let the rest of the sunlight through... all the way to the core. Each of the N layers will have a temperature $T_i$, which is somewhere between the temperature of the outermost layer and the inner "core" (which we will assume to be a perfect black body).
If we write the integral of the Planck law for all the heat from an emitter of temperature $T$ in the range shorter than 800 nm as $P_s(T)$, and the integral of the "longer" component as $P_l(T)$, then we can write a series of coupled equations for the heat balance (assuming the input from the sun is $H$ per unit area). Note - we are assuming that all the radiation from the core below 800 nm goes "right through" for simplicity. We can also write the heat from the sun in terms of the component above 800 nm, $H_l$, and the component below 800 nm, $H_s$.
Just a reminder - if you have a single layer of glass, the equations would be
$$H = P_l(T_1) + P_s(T_c)\\ H_s + P_l(T_1) = P_s(T_c) + P_l(T_c)$$
In these, the first is the heat balance between universe and glass; the second is the balance between the glass and the core ($H_s$ is the component of sunlight hitting the core, and $P_l$ is the component of heat radiation due to the glass; the two terms on the right make up the heat flux away from the core).
Two equations, two unknowns: you can solve this numerically for the temperatures. Every time you add another layer, you add one more equation and one more unknown. I am not sure if there is an analytical way to solve these equations (there is for the case where you have constant emissivity over the entire spectrum, but that's not the case here - see the above link on MLI). I may try to run a simple simulation if I find the time to see if there is a limit to how hot the core can get. Clearly, it cannot get hotter than the sun - that would violate thermodynamics. With the materials in all layers having the same properties, there is probably some relationship between the cutoff wavelength, the number of layers, and the temperature reached.