This won't be a complete answer, because I'm not familiar with Brown-York, but:
The Komar mass applies to stationary spacetimes.
The ADM mass is for asymptotically flat spacetimes, and includes gravitational radiation going to null infinity.
Bondi mass is similar to ADM, but doesn't include the radiation.
Komar and ADM were later shown to be consistent in the cases where both of them apply:
The paper by Sheykhi that you link to is about black holes in anti-de Sitter space, which isn't asymptotically flat. In a spacetime that's not asymptotically flat, it's not at all obvious what would be meant by the mass. The way we normally get a description of the mass of an isolated object is by going far away, where the Newtonian limit applies. For example, you can have something in orbit around a black hole, at a large enough distance that the fact that it's a black hole doesn't matter. Then ordinary newtonian gravity tells you the mass of the object it's orbiting around. In a spacetime that's not asymptotically flat, you don't have this kind of natural definition.