There are a few things to clarify, this is a QM explanation:
you are talking about visible light, so that is what I am going to talk about
there are two types of reflection, specular (mirror), and diffuse.
Now when we see an object, it is because the photons from the object hit our eyes.
When a photon interacts with an atom, three things can happen:
elastic scattering, the photon keeps its energy and phase and changes angle
inelastic scattering, the photon keeps part of its energy and changes angle
absorption, the photon gives all its energy to the atom
Now you would think that when you see an object it is because of reflection. That is not correct.
When you see an object, it is because of:
reflection, that is 1., elastic scattering
absorption and re-emission, 3.
Now it is the ratio of the three, that is different.
For a shiny, mirror like object, like metals, most of the photons get reflected (elastically scattered). This is usually too the case when we talk about specular (mirror) reflection, when all photons keep their energy level, phase, and relative angles.
Do you see the texture of the mirror? No. Why? Because:
metals (except for a few) do not have their own color. They reflect the visible photons in their original color.
metals lattice structure on their surface is usually extremely flat (polished), so they create a specular reflection, causing the photons' relative angle to be kept.
For a non-shiny object, like asphalt in your case, we talk about diffuse reflection. It is diffuse, because the angle of the photons' is not kept. In the case of asphalt, most of the photons are absorbed and re-emitted, and very little are reflected. The problem with absorption and re-emission is, that the angle will be random. That is why asphalt creates a diffuse reflection, the photons' angle is random.
Do you see the texture of asphalt? Yes. Why? It is because:
asphalt has a random, non-flat lattice structure on the surface, and even if you make asphalt completely flat on the macro level, with an asphalt drum roller, the lattice structure on the QM level will still be non-flat, random, and it will create a diffuse reflection. What you truly see on the asphalt, is macro level structure. It is because the mechanical structure of the ingredients causes asphalt to brake into pieces on the top layer. So you are basically talking about seeing a macro level structure.
the reason you see the macro level structure, is because asphalt does re-emit some visible light, and the different parts of the macro structure will re-emit different amounts of photons. Though the wavelength of the photons (re-emitted) will be almost the same, you will see less or more depending on the structure, and the angle it is relative to the lightsource.
So in your case, with asphalt, most of the visible photons are absorbed and some (4%) are re-emitted, and very little are reflected (elastically scattered).
It is very important to understand that there are two structures you can talk about:
micro level, this is at the QM level, and it causes a diffuse reflection for asphalt. This means that asphalt cannot create a mirror reflection. The angle of the re-emitted photons will be random.
macro level, that is at the level of the pieces, ingredients of the asphalt. That creates a visible macro texture.
I assume that you are talking about this macro level texture. This is caused by a combination of the micro level diffuse reflection, and the macro level reflection.
Now, why does the macro level structure of asphalt create a visible texture?
It is because:
the photons intensity changes as the macro structure surface relative angle changes. There are sides of a gravel piece that re-emit more photons with more intensity in our eyes' direction, and there are other sides that re-emit with less intensity. As our eyes see the difference in intensity, we see texture. This is most of the texture we see with asphalt. This is intensity.
different areas of the macro structure might re-emit different wavelength photons, and our eyes see that as texture too. This happens to very little ratio of photons. This is shades of color.
So basically the answer to your question is that the texture is created by intensity changes and shades at the macro level and diffuse reflection and non-flat random surface at the micro level.
With the superblack (or vantablack), there are practically no visible photons re-emitted or reflected.
So it is not reflection. It is re-emission and reflection. These special black coatings make almost no visible photons to be re-emitted or reflected. What happens to the photons? They are all absorbed by the coating. And are not re-emitted (or at least not at visible wavelengths).