Gravitational Assists: star-hopping - what's the limit? By dipping deep into a star's gravitational well, one can take advantage of the Oberth effect and of gravity assists. From the Oberth effect, one can control large changes in direction from relatively small expenditures of fuel (or perhaps even with a solar sail).  From suitably aimed gravity assists, one can make large changes in direction and also potentially capture some of the star's velocity for free. 
Stars in a galaxy average about one light-year apart and their relative velocities are about 0.01% the speed of light (30 km/s). Probably it would take many thousands of years between stellar encounters, but it seems like a robotic craft could guide itself from one suitable star to the next suitable star gaining speed almost indefinitely in the process.  (Though this seems in violation of the second law of thermodynamics).

I'm just wondering what the limits to this process might be.  As the speed of the spacecraft increases, the encounters with stars will become increasingly brief and the momentum transfer from a solar sail or from a gravitational assist will become vanishingly small.  Perhaps aiming at white-dwarfs or neutron stars eventually becomes the best strategy. Perhaps one can never achieve speeds more than a few times the rms relative velocity of the stars? 
A limiting strategy for gravitational assists might be to use black holes which have particularly deep gravitational wells.  Here however there are the issues of tidal forces possibly destroying the spacecraft and gravitational waves radiating away the valuable energy/momentum and defeating the whole purpose. Perhaps this works better with a rotating black hole?
 A: In principle, a  single triple black hole system could be used to accelerate a probe to relativistic velocities.
Consider the following configuration: two black holes (of comparable masses), let us denote them $A$ and $B$, are closely orbiting each other on nearly circular orbits, while the third black hole, $C$ orbits around the first two at much greater distance and thus with lower velocity (see the image):

A probe starts from vicinity of $C$ towards $A$ and $B$ and after circling one of this pair flies back toward $C$, circles it performing almost perfect turnover and flies back again towards $A$ and  $B$ and so forth. During each time the trajectory passes near the component of pair $A$ and $B$ that has instantaneous velocity component directed toward $C$. As a result of each flyby near $A$ or $B$ the probe gathers speed. If all velocities remain nonrelativistic, the speed gain is approximately double the instantaneous velocity component of either $A$ or $B$ toward $C$. Taking into account that usually instantaneous velocity $\mathbf{v}_A$ or $\mathbf{v}_B $ would be at some angle from direction to $C$, on average after large number $N$ of iterations the probe's velocity could be of order $v_\text{probe}\sim N u$ where $u$ is an average orbit velocity of a pair $A$ and $B$.
As probe's velocity become comparable with the speed of light, relativistic laws of velocity addition must be taken into account, so of course the probe could never go faster than the speed of light, but (in principle) could approach it after sufficiently large number of iterations. For a triple black hole system described here, similar trajectories could be constructed for photons, they would of course be always travelling at the speed of light but as a result of this gravitational assists photons would gain energy, extracting gravitational/kinetic energy from the system in, these trajectories could be seen as ultimate limit of gravitational assits.
If instead of black holes we use other bodies such as white dwarfs, neutron stars or even ordinary stars, then the type of gravitational assist described here would have a builtin speed limit in the form of escape velocity on the surface of a body. The lowest of such velocities would determine the maximum velocity of a probe that completes almost complete turnover near such body (for bodies like neutron stars, that need GR to describe orbits around them, the relationship between surface escape velocity and orbital velocity is nontrivial). This limit explains why this type of maneuver would not work for small planets: escape velocity of e.g. Mercury is $4.25\,\text{km/s}$, while Mercury's orbit velocity is $47.4\,\text{km/s}$, so Mercury cannot turn around a probe so that it can fully benefit from its large orbit velocity.
Of course, there are a lot of technical issues that could complicate and impose limitations for such type of travel: tidal accelerations on a probe near a black hole or neutron star could be very large, the maneuvering must be very precise, especially as velocities grow, etc.

Another type of “limit” for gravitational assists has been  considered by Freeman Dyson in a paper

*

*Dyson, Freeman. Gravitational machines. in Interstellar Communication, edited by AGW Cameron, (Benjamin Press, New York, 1963) (1963), free pdf.

This is a limit not in terms of velocity but in terms of scale: if some civilization surrounds e.g. a pair of white dwarfs orbiting each other by streams of masses extracting gravitational and kinetic energy via gravitational assists, then potentially the power extracted could exceed solar luminosity by several orders of magnitude.
A: As an addendum to AVS's excellent answer:
Is this against thermodynamics? No, because you are playing Maxwell's demon! An inert object thrown into space with randomly moving stars will have its velocity vector randomized and end up with an average kinetic energy KE given by the virial theorem: $2\langle \text{KE}\rangle + \langle \text{PE}\rangle = 0 $ where PE is the potential energy in the galaxy. Since spacecraft usually are less massive than stars they will tend to move much faster, but for this to really happen you need to wait for a bunch of galactic relaxation times (many trillions of years). You are likely to eventually end up with galactic escape velocity, several hundred km/s.
A spacecraft that steers expends a tiny amount of energy to place itself in naturally unlikely situations in position-velocity space so that it gets consistent gravity assists (most of this energy is engine steering, a tiny fraction is measuring the environment and calculating where to go).
This also shows another limit to speeding up this way. After you leave a star along a certain velocity vector, you can steer into a widening cone around it. That means you can choose which star to hit next, but only within a certain range and velocity. The further away the better the boost can be. However, as you speed up the angular turning becomes smaller, and the choice of next star becomes more constrained or you have to wait longer for the next boost.
Also, the optimum relative velocity to get a maximal velocity boost is $\sqrt{GM/r_{min}}$ where $r_{min}$ is the closest acceptable approach. So basically you want to point at the closest star with that relative velocity, but as you speed up you are further and further into the tail region of the 3D velocity distribution. If that distribution is a 3D Gaussian the density of suitable stars declines as $\propto \exp(-v^2/2\sigma^2)$ (further reduced by the lesser angular steering ability). For each boost the time until the next one grows as $\propto \exp(v^2/2\sigma^2)/(1+v)$ (the $1/(1+v)$ factor is due to faster transits) making the growth very slow.
Hence super-dense objects start to look tempting for getting very high speeds.
