I remember it being quite hot in the last green house I went inside, and I'm imagining how hot you can make it has something to do with the difference between the temperature inside and outside, because that would determine the conduction of heat through the wall no? So if you construct a smaller green house inside of an existing larger green house, then it should get hotter in the smaller one, and if you keep nesting the green houses I'm imagining at some point you'd just melt your inner building materials and have a hot blob of plastic, which might burst into flames and bring the whole thing down... how many layers would that take?
A greenhouse works by letting in visible light, which are absorbed by plants (and any other absorbing surfaces) inside, while blocking the IR radiation re-emitted by those plants. Suppose you had another smaller greenhouse inside the bigger one. Then what radiation passed through the bigger greenhouse shall also pass through the smaller one (whatever fraction is incident on it; to be sure there will be some reflection too). Radiation re-emitted by plants in the bigger greenhouse will not get through into the smaller greenhouse for the same reason that they cannot escape out of the bigger one. So for all intents and purposes the smaller greenhouse may be considered to be directly exposed to sunlight (albeit with enhanced reflection since the sunlight has to pass through two glass walls). Therefore if there are absorbing surfaces inside the smaller greenhouse too, then the temperature there can be no greater than that inside the bigger greenhouse. The argument may be extended to nested greenhouses, and the temperature inside any of the greenhouses would not be greater than that inside the biggest greenhouse.
The answer by @Deep gets it correctly that the inner green house would get somewhat smaller amount of energy than the outer green house, so there is no gain... there is even loss of efficiency. The answer however gets incorrectly a few points about the greenhouses, which it is necessary to clarify:
Black body radiation
Greenhouse is heated by Sun, which can be considered as a black body with temperature of about $6000K$. The plants, earth and other object inside the greenhouse absorb this radiation and reemit it. This radiation can be also considered as black-body radiation, but at the temperature inside the greenhouse, which is much lower than that of the Sun - about $300K$ (mainly in the infrared). It is this difference in temperatures that results in the net heating (not blocking the infrared radiation - in fact, some materials transparent in visible spectrum could be blocking the ultraviolet just as well).
The energy transfer mechanism that is blocked in the greenhouse is convection - without the greenhouse the heated earth and plants would heat the air around them, which would rise, carrying with it the energy. Greenhouse blocks air flow, keeping the hot air inside.
Why does not temperature keep rising?
What limits the highest temperature in the greenhouse is inevitable convection losses, but also the thermal conductance through the earth and the greenhouse walls.
The point of having an inner greenhouse may be very valuable as frost protection. It would take longer to cool down.
Also it works to keep moisture in just like a lidded seed tray.
My greenhouse has an 80L water butt inside to act as a thermal mass during the winter.
I believe in Alaska they put a wall of water butts on the north side to extend the seasons.
Deep's answer is correct.
I would just like to add a scheme to point out that, since there will always be some reflection at the glass/air interface, putting another greenhouse inside the first one could actually decrease the quantity of energy entering the system, thus making the whole thing colder.
Neither of them take into account a small heater inside the inner Green house. We have a 14x20 hoop house inside a 35x60 timber frame Green house we are building over top the hoop house. It has been heated and will still be. We are hoping to grow orchids inside it like vanilla beans. In theory the heat from the inner one will help heat the outer one in the winter.