Peter Schor's comment that one of the three roots of the cubic must be negative is correct. In general, you have to select the root that is physically relevent. You can reject the negative root, and for the Schwarzschild geometry, any root smaller than the photon sphere radius at $1.5 r_s$.
Also, I believe that you have an error in your equation for $R$, since as written, it does not have the correct solution at the critical impact parameter, that allows the closest approach where it is still possible for a photon to escape.
I assume that you derived the equation by solving for $\frac{dr}{dt}=0$ at closest approach, and using normalized radius and impact parameter as follows, where $r_s$ is the Schwarzschild radius.
$$R = \frac{r}{r_s}$$
$$B = \frac{b}{r_s}$$
I believe that the equation for $R$ should be as follows.
$$R^3 - B^2(R-1) = 0$$
Solving this equation for the critical impact parameter, $B=3 \sqrt{3}/2$, results in a double root at $R = 3/2$, and a negative root at $R = -3$. Solving the equation at a slightly larger impact parameter, say $B=3$, gives roots at about 2.227, 1.185, and -3.411. Clearly only the larger radius can be a solution. The solution at $B=5$ is about 4.394.