The theoretical formula expresses the fraction of events that decays to the final state you are looking for (numerator) with respect to all possible final states (denominator).
The experimental formula needs to be interpreted in a special way : you have to consider as numerator $N_{obs}/\epsilon$ (it is not "understandable" if you put "directly" $\epsilon$ at the denominator), and as denominator the total number of produced events (whatever is the decay).
$N_{obs}$ represents the number of produced events with the final state. Since you wish to relate this to the number of "expected" events without decay ($N_{BB}$), you need to divide $N_{obs}$ by the selection efficiency of your analysis : $\epsilon$. This will transform the numerator into the corresponding number of events before the decay. This selection efficiency is obtained from Monte-Carlo simulation of your signal sample.
$N_{BB}$ is obtained either from a combination of several analyses, or by simulation.