As Guillermo Angeris correctly pointed out, this is essentially a numerical roundoff problem, not a physical situation.
As a physical example, there are sungrazing comets that get very close to to Sun, yet they maintain their original elliptical (or hyperbolic) orbit, without the orbit precessing a full third of a circle as you seem to be seeing.
Computationally, there are a few interesting issues. As Kyle pointed out in a comment, many integration schemes are indeed unreliable in that roundoff error (which is always present in floating-point computations) can accumulate in a runaway feedback. Indeed I often advise using leapfrog methods over Euler (used by Box2D) or even Runge-Kutta (see for instance What is the correct way of integrating in astronomy simulations? over at the Computational Science Stackexchange).
However, I suspect your problem is even simpler, in the sense that even an unstable numerical scheme should work for one or two orbits. Given that everything is going wrong in just one pass, it seems that your timesteps are simply too large. A brief glimpse at the Box2D documentation suggests you don't change the timestep mid-simulation, so I presume you are just using a good value to simulate the whole process in reasonable time. The problem is that when gravitating bodies get close in their orbit, they move quickly, sometimes very quickly. The way the code works is it updates each object's position and velocity at each timestep, where the new velocity is determined by the force. As far as I can tell, this is done in line 206 of b2Island.cpp (v. 2.2.1):
v += h * (b->m_gravityScale * gravity + b->m_invMass * b->m_force);
Without looking at your code, I am guessing you simply calculate the gravitational force the body should feel at that moment, and have the simulation chug away. The problem is this moves the orbiting object in a straight line for the next timestep, and that straight line takes it too far away from the gravitating mass for that mass to properly curve its orbit into a closed ellipse. The quick schematic below shows the blue object moving to the tip of the red arrow, rather than staying on the path.
Physically, your timestep should be smaller than any timescale you encounter in the problem. Now for an orbit conserving angular momentum, the product of the orbiting body's mass, tangential velocity $v$, and distance from the other object $r$ should be constant: $v \sim 1/r$. At the same time, the acceleration $a$ it feels is given by Newton's law of gravity: $a \sim 1/r^2$. So one natural timescale in this problem is
$$ t \sim \frac{v}{a} \sim \frac{1/r}{1/r^2} \sim r $$
(omitting dimensional constants), which goes to show that if your timescale is just barely small enough and then you tweak the orbit so as to half the periapsis distance (distance of closest approach), then you would expect to need timesteps at least twice as small in order to preserve the integrity of the simulation.
