What is wrong with my thinking?
There are a couple of points where you make key errors. The first (and least important) is here:
Suppose the efficiency was maximum so that exactly $\frac{1}{2}mv^2$ of chemical energy was just converted to this kinetic energy.
This is not possible for a rocket that begins at rest because it does not simultaneously conserve momentum. For a rocket at rest, at that moment, all of the power goes into the KE of the exhaust, not the rocket.
Even assuming no waste of any kind, the only time that 100% of the KE goes into the rocket is at the moment that the speed of the rocket equals the exhaust velocity. In general, for a constant $\Delta PE$ of the chemical potential energy in the fuel the $\Delta KE$ of the rocket depends strongly on the speed of the rocket.
In this frame, the rocket stays at rest but now the whole universe, say with mass 𝑀≫𝑚, is now moving towards me with energy $\frac{1}{2}Mv^2$. Energy was clearly not conserved in this frame. Presumable this is because it is an accelerating frame, and energy is only conserved in inertial frames. But my intuition says that as the acceleration approaches zero, the results should be the same as in an inertial frame. However, it is easy to check that this huge energy violation persists for arbitrarily small accelarations and at arbitrarily short times.
In the rocket frame the universe is subject to a fictitious gravitational acceleration $g$. So if the universe falls through this fictitious gravity field for a time $t$ then the velocity of the universe is $v=gt $. So we can write the KE of the universe as $\frac{1}{2}M(gt)^2$. If we expand that to first order in $t$ we get $0+O(t^2)$ and if we expand that to first order in $g$ we get $0+O(g^2)$. Therefore your intuition is correct.
Since you didn’t post your math it is impossible to know what led to the mistake, but at least in this case you are correct to intuit that as the time or the acceleration go to 0 so does the energy violation.
Be aware, if the acceleration is constant in time then it is possible to make a potential energy in this frame and recover conservation of energy. The universe falls through the gravitational field losing PE and gaining KE, thus conserving energy. That only works if the acceleration is constant.