For my Bachelor's thesis, I've created a neural network with the task of classifying FCNC tz-production events. It was trained on data from a Monte-Carlo simulation, and tries to output 0 when encountering a background event, and 1 when encountering a signal event. Obviously, its output is continuous between 0 and 1, so some kind of classification threshold is needed above which the event is classified as "signal". During training, this was set to 0.5, but in the validation phase, it makes sense to adjust this parameter in order to ensure maximum significance before feeding the real data into the network.
I tried doing this by looking at the network's response to all events in the validation dataset, and creating a histogram for them. Here is said histogram:
Now, I continued by trying to maximize $S/\sqrt{B}$. I did this by starting at the right-most bin and adding the bins together, for signal and background seperately. With each iteration, I then calculate $S/\sqrt{B}$, where $S$ is the current sum of the signal bins and $B$ is the sum of the background bins. The idea is that the highest $S/\sqrt{B}$ obtained in this procedure gives you the optimal classification threshold, which would then be the left edge of the bins added together.
However, doing this, I only ever get a supposedly optimal threshold directly on the left edge of the right-most bin. I've tried various different histogram step sizes, all leading to the same result of the right-most bin alone having the maximum $S/\sqrt{B}$. This seems to make sense, because the right-most bin holds very many signal events (in the case of 100 total bins, the last bin holds more than half of all signal events) while there are very few background events in it.
But setting my threshold at 0.99 seems ridiculous, and I'm certain this is not the correct way to go about this. I also tried to use the improved "Asimov Z" instead of $S/\sqrt{B}$, which looks like this:
$$ Z=\sqrt{2((S+B)\ln(1+S/B)-S)} $$
But the results were the same.
