The three-dimensional nature of a crystalline solid leads to a directional electron population. If a valence band maximum and conduction band minimum do not occur in the same electron-momentum direction, light absorption that would otherwise occur at the bandgap energy will be very unlikely. This is because the electron that hops up in energy (taking up the photon's energy to do so) must also change its momentum to complete that transition. Photons have enough energy, but do NOT contain enough momentum to make that quantum jump occur. So, the
direct gap material has a higher photon absorption rate, because the 'easy' short-hop transition is not excluded by conservation of momentum.
There will be absorption due to more complex interactions, involving a third particle (usually a phonon being absorbed or generated), but those occur at a
very low rate compared to the simpler direct electron promotion.