why can't we see our image on a simple brick wall? We can see our image in a mirror due to the reflection of light. We also know that a red color brick wall also reflects light. So, why can't we see our image there?
 A: Let's first of all negelect the differences in absorption between the miror and the red brick wall. The main difference between the brick wall and a mirror is the flatness of the surface. The morphology of the surface will determine whether you will observe specular or diffuse reflection. 
I've depicted both situations in the figure below. You can easily see that the specular reflection will result in a reflected image that is representative for the original object (this is the case for the flat mirror). On the other hand the diffuse reflection will distort the image in such a way that light is scattered in all directions and all "information" about the original object is lost. In the case of the brick wall the image is distorted so much that the only thing you see is the color of the brick.

You could wonder how flat the surface has to be in order for specular reflection to occur. The answer to that question is that it depends on the wavelength of the light you wish to reflect. Telescope builders know this all to well. Whereas deviations from a perfect surface for mirrors in optical telescopes ($\lambda \sim$ a few 100 nm) are of the order of nanometres, the deviations for radio telescopes ($\lambda \sim$ a few metres) can be of the order of centimetres. This is why radiotelescopes look like a metal grid (e.g. the Westerbork Synthesis Radio Telescope)  but still act as a perfect mirror for radiowaves. 
A: For an image to be reflected back the whole wavefront must back scatter elastically, keeping its direction and phases. This is where the "flatness" discussed by others comes in. 
On a rough surface part of the light may scatter elastically, but it becomes a point source at each bump, losing the phase of the wavefront, i.e. coherence. An addition of millions of such point sources is the result, the loss of coherence and image information; the color is a matter of what wavelenghts the wall has absorbed.
As others have stated the wavelength of the light defines how flat the surface has to be.
A: It is because the brick wall has a rough surface and therefore shows a diffused reflection.
A: It is because the surface of a brick wall is rough and a ray of light needs a smooth surface to reflect.       Therefore we do not see our reflection on a brick wall 
