You can't argue with a black object and a white object alone, as I think you partially understand in trying to build your thought experiment. You need a little bit more to define things properly. See whether the following helps.
Imagine a black object at a temperature $T_0$ and a white object also at $T_0$ inside a perfectly isolating box full of blackbody radiation at some higher temperature $T_1>T_0$ (i.e. without the black and white objects, this radiation is in thermodynamic equilibrium).
To understand exactly what would happen, you would have to describe the "colour" of your objects with emissivity curves that show emissivity as a detailed function of frequency. So your "black" and "white" would need to be defined in much more detail. You would also have to define the surface areas of the two objects and what they are made of (i.e. define their heat capacities). But all of this only effects the dynamics of how the system reaches its final state, i.e. these details only influence how the system evolves. What it evolves to is the same no matter what the details: the box would end up with everything at the same temperature such that the total system energy is, naturally, what it was at the beginning of the thought experiment. "Blacker" as opposed to "Whiter in this context roughly means "able to interact, per unit surface area, with radiation more swiftly": the blacker object's temperature will converge to that of the radiation more swiftly than does that of the whiter object, but asymptotically the white object "catches up". Blacker objects absorb more of their incident radiation its true, but they also emit more powerfully than a whiter object at the same temperature. The one concept emissivity describes the transfer in both directions. Think of emissivity as being a fractional factor applied to the Stefan-Boltzmann constant for the surface as well as being the fraction of incident light absorbed by the surface relative to a perfect blackbody radiator.
This description is altogether analogous to that of the situation where $T_0<T_1$. Begin with $T_0=T_1$, and you've got thermodynamic equilibrium from the beginning. Nothing happens, of course.
Maybe the following will help thinking about what is a really quite a complex question: it would be a fantastic last question for an undergrad thermodynamics exam BTW: You can abstract detail away by saying lets define object $A$ to be blacker than object $B$ if, when both objects are made of the same material, are the same size and shape, the temperature of $A$ converges to the final thermodynamic equilibrium temperature more swiftly than that of $B$ when they are both compared in the box-radiation-object thought experiment above.
Thinking about this now, I am not sure whether the above definition would hold for every beginning temperature of the radiation. Maybe there are pairs of surfaces whose relative blackness is different at different beginning temperatures such that $A$ is blacker than $B$ with some beginning temperature whilst the order swaps at a different beginning temperature. I think it is unlikely, but that is probably a different question altogether.
By the way, which pub do you drink in? I might come along.