If you look at this problem in 2D you have the following parameters at some instant which describe your trajectory (position and velocity) around a celestial body with gravitational parameter $\mu$:, radius $r$, radial velocity $\dot{r}$$v_r$ and angulartangential velocity $\omega$$v_t$. There are also a few others, but these do not really matter in this problem, due to symmetry.
You can calculate the radius of your periapsisperiapsis by using conservation of specific angular momentumthe equations for the semi-major axis and orbital energythe eccentricity, which when expressed in $\mu$, $r$, $v_r$ and $v_t$ look like
$$ a = \frac{\mu r}{2\mu - \left(v_r^2 + v_t^2\right) r}, \tag{1} $$
$$ e = \sqrt{1 + \frac{\left(v_r^2 + v_t^2\right) r}{\mu} \left(\frac{v_t^2 r}{\mu} - 2\right)}, \tag{2} $$
$$ r_{pe} = a (1 - e), \tag{3} $$
with $a$ the semi-major axis, $e$ the eccentricity and $r_{pe}$ the periapsis. $$ \epsilon=\frac{\omega^2r^2+\dot{r}^2}{2}-\frac{\mu}{r}=\frac{\omega_p^2r_p^2}{2}-\frac{\mu}{r_p} $$ $$ h=\omega r^2=\omega_pr_p^2 $$ From here
Now if you can derive an expression forcalculate the radiustotal time derivative of the periapsis it should be zero, if no other external force is applied besides Newtonian gravity, because without perturbation each orbital element should stay constant,
$$ \frac{d p_{pe}}{dt} = \frac{\partial r_{pe}}{\partial r} v_r + \frac{\partial r_{pe}}{\partial v_r} \dot{v}_r + \frac{\partial r_{pe}}{\partial v_t} \dot{v}_t = 0, \tag{4} $$
where $r_p$: $$ \omega_p=\frac{h}{r_p^2} $$ $$ \epsilon=\frac{\left(\frac{h}{r_p^2}\right)^2r_p^2}{2}-\frac{\mu}{r_p}=\frac{h^2}{2r_p^2}-\frac{\mu}{r_p} $$ $$ \epsilon r_p^2=\frac{h^2}{2}-\mu r_p\rightarrow \epsilon r_p^2+\mu r_p-\frac{h^2}{2}=0 $$ $$ r_p=\frac{-\mu\pm\sqrt{\mu^2+2\epsilon h^2}}{2\epsilon} $$ These two solutions correspond with apoapsis$\dot{v}_r$ and periapsis (the to point in$\dot{v}_t$ are the orbit attime derivatives of $v_r$ and $v_t$ respectively, which the radial velocity is zero). And you might suspect that the 'plus' solution might correspond with apoapsis and 'minus' with periapsissame as the vector components of the net acceleration. However
If you now apply an additional force/acceleration by burning the opposite is true, since forengines at an elliptical orbitangle $\epsilon$ is negative$\phi$ relative to the tangential direction, as illustrated in the image below, equation (it$(4)$ will also hold for trajectories with higher eccentricity), so: $$ r_p=\frac{\sqrt{\mu^2+2\epsilon h^2}-\mu}{2\epsilon}=\frac{\sqrt{\mu^2+\left(\omega^2r^2+\dot{r}^2-\frac{2\mu}{r}\right) \omega^2r^4}-\mu}{\omega^2r^2+\dot{r}^2-\frac{2\mu}{r}} $$ To find what wouldnow not necessary be the best angle to burnequal to lower your periapsis the most you could use a fixed amountzero.
The magnitude of the additional acceleration is $\Delta v$$f$. When applying this acceleration and seeusing that equation $(4)$ is zero the time derivative of $r_{pe}$ becomes,
$$ \frac{d p_{pe,f}}{dt} = \frac{\partial r_{pe}}{\partial r} v_r + \frac{\partial r_{pe}}{\partial v_r} \left(\dot{v}_r - f \sin\phi\right) + \frac{\partial r_{pe}}{\partial v_t} \left(\dot{v}_t + f \cos\phi\right) = f \left(\frac{\partial r_{pe}}{\partial v_t} \cos\phi - \frac{\partial r_{pe}}{\partial v_r} \sin\phi\right). \tag{5} $$
You want to know for which angle $\phi$ would lower the periapsisvalue for the mosttime derivative of ($\frac{\delta}{\delta\phi}\Delta r_p=0$): $$ \Delta r_p=\frac{\sqrt{\mu^2+\left(\omega^2r^2+\dot{r}^2-\frac{2\mu}{r}\right) \omega^2r^4}-\mu}{\omega^2r^2+\dot{r}^2-\frac{2\mu}{r}} - \frac{\sqrt{\mu^2+\left((\omega r+\Delta v\sin{\phi})^2+(\dot{r}+\Delta v\cos{\phi})^2-\frac{2\mu}{r}\right) (\omega r+\Delta v\sin{\phi})^2r^2}-\mu}{(\omega r+\Delta v\sin{\phi})^2+(\dot{r}+\Delta v\cos{\phi})^2-\frac{2\mu}{r}} $$ I will leave deriving this equation$r_{pe,f}$ becomes the largest. This can be done by differentiating it with respect to $\phi$ and solve for it when you set the resulting equation equal to zero.
But I suspect that lowering the specific angular momentum will have the greatest impact$$ \frac{\partial}{\partial \phi}\left(\frac{d p_{pe,f}}{dt}\right) = f \left(-\frac{\partial r_{pe}}{\partial v_t} \sin\phi - \frac{\partial r_{pe}}{\partial v_r} \cos\phi\right) = 0, \tag{6} $$
solving for $\phi$ yields, so burning in
$$ \phi = \tan^{-1}\left(\frac{-\frac{\partial r_{pe}}{\partial v_r}}{\frac{\partial r_{pe}}{\partial v_t}}\right). \tag{7} $$
The only messy part of this solution is calculating the radial direction, especially when are far awaythe partial derivatives of $r_{pe}$.
Edit:
To answer your second question in When I try to solve it for your example I get an angle of (first) comment-3. Adding2544°, so very close to the radial parttangential direction, which decreases the angular momentum of the orbit, but also close to $\omega^2r^2$ can be done using the fact thatperpendicular to the current velocity because the radial velocity component ca be expressed as $v_\theta=\omega r$, so:
$$
v_\theta+\Delta{v_\theta}=\omega r+\Delta{v}\sin{\phi}\rightarrow \left(v_\theta+\Delta{v_\theta}\right)^2=\left(\omega r+\Delta{v}\sin{\phi}\right)^2
$$
However your equation can be simplified to this as well:
$$
\left(\omega+\frac{\Delta{v}\sin{\phi}}{r}\right)^2 r^2=\left(\left(\omega+\frac{\Delta{v}\sin{\phi}}{r}\right) r\right)^2=\left(\omega+\frac{\Delta{v}\sin{\phi}}{r}\right)^2 r^2
$$is larger than the tangential velocity.