I don't think there is a lower bound.
Set $x$ horizontal, $y$ vertical, $(0,0)$ at the center of the circle.
We'll find $x$ and $y$ for the point of impact. Say the ball takes time $t$ to cross the cylinder. It falls a distance $\frac{1}{2} g t^2$, so $y = -\frac{1}{2}gt^2$. We also have $x = vt - R$, or $t = (x+R)/v$ . Using the fact that for a circle $dy/dx = -x/y$, the slope of the wall where it hits is $\frac{2x}{gt^2} = \frac{2xv^2}{g(x+R)^2}$.
The y-velocity of the ball at impact is $-gt$. The ball's trajectory's slope at impact is $v_y/v = \frac{-gt}{v} = -\frac{g(x+R)}{v^2}$.
Multiplying the slopes gives $\frac{-2x}{x+R}$.
Since $0<x<R$ for large $v$, we see that the slopes multiply to something $0>product>-1$. That is, the slope of the ball's trajectory is not quite perpendicular to the wall. Instead, it's a little shallower than that. That means the ball bounces back up off the wall at a slightly higher angle than it hit the wall, and therefore bounces right back out almost the way it came, but slightly higher, in the limit of high $v$. Having the ball come off the wall at a higher angle clearly makes it bounce out of the cylinder in this limit. That means there is no minimum time, since the time inside shrinks as $v$ increases.