You are quite right one ought not to assume that, and indeed to prove it takes a little care. Here it is. I will offer first a simple argument for the case where a body is totally enclosed in a cavity, and then a more general argument.
Simple argument when a body is totally enclosed
We treat the case of a convex body at temperature $T_1$ surrounded by walls at temperature $T_2$. (Convex means the body's shape does not dip inward at any point in such a way that emitted radiation could be reabsorbed.)
I take as starting point that the total power emitted by the body is
$$
P_{\rm out} = \epsilon A \sigma T_1^4
$$
where $A$ is its surface area and $\epsilon$ the emissivity. Meanwhile radiation is falling on the body from the surrounding walls. If the body were at the same temperature as the walls then we know it would receive the same power in as it is giving out, because then everything would be in equilibrium. So clearly it receives from the walls $\epsilon A \sigma T_2^4$.
Now using that $\epsilon = \alpha$ where $\alpha$ is the absorption coefficient, we find that the power absorbed by such a body as it absorbs thermal radiation emitted by the surrounding walls is
$$
P_{\rm in} = \alpha A \sigma T_2
$$
and this will be still be true, for a given shape and a given $\alpha$, even when the body is at some other temperature.
More general argument
Now suppose more generally that two bodies are exchanging energy by radiation,
and neither need necessarily enclose the other. Let
$$
\begin{array}{rcl}
F_{12} &=& \text{fraction of energy leaving 1 which arrives at 2}\\
F_{21} &=& \text{fraction of energy leaving 2 which arrives at 1}
\end{array}
$$
We take the bodies to be of fixed shape, so these are functions of only the
geometry and the angular distributions of emission.
Total energy leaving 1 per unit time is
$$
A_1 J_1
$$
where $A_1$ = surface area and $J_1$ = flux averaged over the surface area.
Hence total arriving at 2 is
$$
A_1 J_1 F_{12}
$$
and total arriving at 1 is
$$
A_2 J_2 F_{21}.
$$
Now consider the case where the situation is thermal equilibrium. There may be other bodies nearby, but we can argue from the principle of detailed balance that the exchange between any pair is balanced:
$$
A_1 J_1 F_{12} = A_2 J_2 F_{21}.
$$
Now take the case where both bodies are black, and we still have thermal equilibrium. In this case $J_1 = J_2$ so have found that in thermal equilibrium,
$$
A_1 F_{12} = A_2 F_{21}.
$$
This is called the shape factor reciprocity relation. This is a remarkable result, which applies for any geometry. It would be quite difficult to derive it by considering the emitted radiation as a function of angle from each surface, and all the lines of sight from one body to the other, etc.
Now that we have found how the geometry works, we can now go back and consider non-black bodies and situations other than equilibrium. The shape factors $F_{12}$ and $F_{21}$ and areas don't change, as long as each body on its own is emitting thermal radiation (i.e. it is not a laser which has just been switched on or something like that). So the result still applies. The non-blackness comes in to the relationship between the flux $J$ and temperature $T$ for each body. The non-equilibrium means we will not necessarily find $A_1 J_1 F_{12} = A_2 J_2 F_{21}$ any more. But the wonderful thing is that if the emission from each body on its own is thermal, then it has the same geometric pattern as it would have in overall thermal equilibrium, so we can still use
$$
F_{21} = \frac{A_1}{A_2} F_{12}
$$
so the power exchanged is
$$
P_{12} = A_1 F_{12} (J_1 - J_2).
$$
I still find it amazing even after having proved it!
Finally, if 2 surrounds 1 then we have $F_{12} = 1$ and hence the same result as given by the simple argument.