Your problem is that light can do 3 things when hits a wall surfice: it can pass through (transparency), absorbed (black body) or re-radiated (reflection). You overlook the latter. That is, you overlook that there is no any limit on the direction that atom can reflect the photon. What is the chance to re-emit into the original direction? It is basically zero. Can you talk about the transparency if there is no chance to pass your photons through?
Actually, I once had the same misunderstanding/question in a physics lesson. I asked "Why do we see a gas, illuminated by distant stars in space, instead of the stars themselves? The gas should entail some delay, like transparent body, but it should re-emit the photon afterwards and let it go through." The teacher explained that the photons hit the gas atoms indeed but then they are emitted into all directions. That is why you do not see the empty space but start seeing it once cloud is placed there. Empty space does not radiate but cloud does.
Same thing happens with reflection. The inciting photons are reflected into all directions from the surfice but you cannot see them deep in the material. Since fraction of them is absorbed, they all are absorbed after some re-emissions-absorbtions in the depth of the matter. Here, by "absorbtion" I mean black-body absorbtion. The photons disappear accelerating the thermal motion of the atoms. Only those which are reflected back into the air immediately stay visible. Therefore the most objects you see are neither transparent nor black. They reflect the inciting light.
I also cannot cannot explain is why most reflections are mirror-like, which means that if they fall at angle $\alpha$, they are re-emitted at angle $-\alpha$, despite they are re-emitted in all directions arbitrarily, as I told you. Feynman tried to explain this popularly, but I failed.
So, anyway, the brick can either absorb the photons the black-body way but reflects a fair amount of photons from the surface before this happens. There is almost 0 chance for the light to pass the brick or any other object because of absorbtion and reflection into all directions. You need to assimilate the reflection process into your picture.