Black holes have conservative potentials. Why don't things "swing" in and out? Black holes attract objects via gravity, which is a conservative force. Thinking from conservation of energy, It seems like for an object moving toward them, they should be able to "swing" into and out of a black Hole's event horizon no matter how strong the gravitational attraction is inside so long as it didn't collide with the singularity. Where am I going wrong?
Further comments:
@Javier mentioned that if a potential has greater than 1/r^2 dependence, it would overcome the angular momentum term and things would fall into the object producing the potential. I was confused because gravity only has 1/r scaling, but this comment by Javier was just to mention that having object collide like this was possible even with classical mechanics (though not via gravity). I was also confused by his figure appearing to put r = 0 on the time axis, but this just relates to his comment about space and time switching places.
 A: First note that even in classical mechanics, any potential that goes to $-\infty$ faster than $1/r^2$ will be stronger than the angular momentum barrier, and you will hit the center. But this isn't the reason you can't escape a black hole.
In the context of general relativity (which is the only way to make sense of black holes), gravity is not exactly a force. The usual name for what's going on is "curvature of spacetime". In essence, gravity changes how spacetime works, in such a way that in familiar circumstances it looks just like a force. And when you have something orbiting outside a black hole, you can pretend that there is a force with a conservative potential and proceed much like you do in Newtonian mechanics.
But when you go inside the event horizon, things get weird. You can't pretend gravity is just a force anymore, because a black hole really messes with spacetime. The details of this are quite hard to explain to someone without a background in general relativity; one popular way of putting it is to say that inside a black hole, time and space switch places. In other words, going forwards in time (which is what everything must do) means going towards $r=0$.
This is clearest if you look at a Penrose diagram:

Essentially, the singularity doesn't really correspond to a point in space. What the diagram is saying is that the singularity behaves more like a point in time; it's the future of everything that goes inside the black hole.
A: 
Thinking from conservation of energy, It seems like for an object moving toward them, they should be able to "swing" into and out of a black Hole's event horizon no matter how strong the gravitational attraction is inside so long as it didn't collide with the singularity. ...
@Javier mentioned that if a potential has greater than 1/r^2 dependence, it would overcome the angular momentum term and things would fall into the object producing the potential. I was confused because gravity only has 1/r scaling,

For a Schwarzschild black hole of mass $M$, the effective potential for an orbit of a test particle of specific angular momentum $\ell$ is (in units of $G = c = 1$):
$$V_\text{eff} =  \underbrace{-\frac{M}{r} + \frac{\ell^2}{2r^2}}_\text{Newtonian form} - \frac{M\ell^2}{r^3}\text{,}$$
and the orbit of a massive particle is characterized by a constant specific energy
$$\mathcal{E} = \frac{1}{2}\left(\frac{\mathrm{d}r}{\mathrm{d}\tau}\right)^2 + V_\text{eff}\text{.}$$
The $r^{-3}$ relativistic correction the Newtonian form can be be interpreted as a perturbation responsible for e.g. the precession of Mercury, although in GTR both the meaning of Schwarzschild radial coordinate $r$ and proper time $\tau$ are ultimately non-Newtonian.
In any case, you can see that at the Schwarzschild radius $r = 2M$, the centrifugal potential and the relativistic correction cancel out each other. Since it's the centrifugal potential that's responsible for orbits "swinging out" in Newtonian gravity, we can't expect it to save us here. Rather, the attractive $r^{-3}$ term is dominant at smaller radial coordinates.
For more complicated black holes, there is usually no single effective potential, and gravity can't be said to be a conservative force in anything like the Newtonian sense. However, an event horizon still prevents a particle from escaping for other black holes as well, analogously to @Javier's answer (although they would have more complicated Penrose diagrams).
