Recently I came across the terms signal efficiency and background rejection (for example in this paper), but I couldn't find an understandable definitions to those terms and how they are used. Specifically, I couldn't find a mathematical definition.
I'd appreciate an explanation with an example or a reference.
This answer is framed in terms of particle physics, but the basic idea can be generalized to any analysis where the presence or absence of the thing you are trying to study must be deduced from the raw data.
Many particle physics detectors collect data in a very general sense, and to study something in particular you need to select from all the myriad recorded events that those that correspond to the physics you want to study.
That is, you are going to partition the whole data set into those events that you believe represent your physics and those that do not. Obviously you can make two classes of errors in that partition:1
The term signal efficiency is usually defined as the fraction of the desired events that you actually get (i.e. it goes down as the false negative rate goes up). The term background rejection is usually defined as the fraction of events that don't belong in your sample that are excluded from your sample (i.e. it goes down as the false positive rate goes up).
1 And it is generally the case that the harder you work to prevent one class of errors the more of the other class you are going to get, so there is a trade-off here.
