What is the optimal burn direction to lower periapsis of hyperbolic orbit? I am an engineering student who is interested in orbital mechanics.  I am doing some self study before taking some orbital mechanics courses next year.  I was learning about various orbit types (elliptical, parabolic, hyperbolic, etc.) and the effects of burning in various directions.  I have found a lot of good information about about how to manipulate elliptical orbits (raising/lowering apo/periapsis, changing inclinations, etc.).
However, I haven't found a lot of information on manipulating a hyperbolic trajectory.  I have found a lot of good information like this in calculating various parameters (impact parameter, turning angle, etc.) but little specifics on how to change one.   
For example, say you were on a hyperbolic flyby like in this example.  However, you wanted to lower your radius at periapsis by a couple hundred km for some reason (take some measurements, increase turning angle etc.).  What would be the most efficient direction to burn?  I could see doing it 2 different ways but unsure of which would be more efficient. You could burn retrograde lowering your velocity magnitude which would pull you closer to the planet.  Or you could burn perpendicular to your current velocity vector in the direction of the planet changing your approach angle?  Perhaps some combination of the 2?
Anyone know how to determine what would be the most optimal?
 A: If you look at this problem in 2D you have the following parameters at some instant which describe your trajectory (position and velocity) around a celestial body with gravitational parameter $\mu$, radius $r$, radial velocity $v_r$ and tangential velocity $v_t$. There are also a few others, but these do not really matter in this problem, due to symmetry.
You can calculate the radius of your periapsis by using the equations for the semi-major axis and the eccentricity, which when expressed in $\mu$, $r$, $v_r$ and $v_t$ look like
$$
a = \frac{\mu r}{2\mu - \left(v_r^2 + v_t^2\right) r}, \tag{1}
$$
$$
e = \sqrt{1 + \frac{\left(v_r^2 + v_t^2\right) r}{\mu} \left(\frac{v_t^2 r}{\mu} - 2\right)}, \tag{2}
$$
$$
r_{pe} = a (1 - e), \tag{3}
$$
with $a$ the semi-major axis, $e$ the eccentricity and $r_{pe}$ the periapsis.
Now if you calculate the total time derivative of the periapsis it should be zero, if no other external force is applied besides Newtonian gravity, because without perturbation each orbital element should stay constant,
$$
\frac{d p_{pe}}{dt} = \frac{\partial r_{pe}}{\partial r} v_r + \frac{\partial r_{pe}}{\partial v_r} \dot{v}_r + \frac{\partial r_{pe}}{\partial v_t} \dot{v}_t = 0, \tag{4}
$$
where $\dot{v}_r$ and $\dot{v}_t$ are the time derivatives of $v_r$ and $v_t$ respectively, which is the same as the vector components of the net acceleration.
If you now apply an additional force/acceleration by burning the engines at an angle $\phi$ relative to the tangential direction, as illustrated in the image below, equation $(4)$ will now not necessary be equal to zero.

The magnitude of the additional acceleration is $f$. When applying this acceleration and using that equation $(4)$ is zero the time derivative of $r_{pe}$ becomes,
$$
\frac{d p_{pe,f}}{dt} = \frac{\partial r_{pe}}{\partial r} v_r + \frac{\partial r_{pe}}{\partial v_r} \left(\dot{v}_r - f \sin\phi\right) + \frac{\partial r_{pe}}{\partial v_t} \left(\dot{v}_t + f \cos\phi\right) = f \left(\frac{\partial r_{pe}}{\partial v_t} \cos\phi - \frac{\partial r_{pe}}{\partial v_r} \sin\phi\right). \tag{5}
$$
You want to know for which angle $\phi$ the value for the time derivative of $r_{pe,f}$ becomes the largest. This can be done by differentiating it with respect to $\phi$ and solve for it when you set the resulting equation equal to zero.
$$
\frac{\partial}{\partial \phi}\left(\frac{d p_{pe,f}}{dt}\right) = f \left(-\frac{\partial r_{pe}}{\partial v_t} \sin\phi - \frac{\partial r_{pe}}{\partial v_r} \cos\phi\right) = 0, \tag{6}
$$
solving for $\phi$ yields,
$$
\phi = \tan^{-1}\left(\frac{-\frac{\partial r_{pe}}{\partial v_r}}{\frac{\partial r_{pe}}{\partial v_t}}\right). \tag{7}
$$
The only messy part of this solution is calculating the the partial derivatives of $r_{pe}$.
When I try to solve it for your example I get an angle of -3.2544°, so very close to the tangential direction, which decreases the angular momentum of the orbit, but also close to the perpendicular to the current velocity because the radial velocity is larger than the tangential velocity.
