As a starting point, I suggest that you take a look at the Wikipedia page on heat transfer physics, phonon scattering and Umklapp scattering.
In general the thermal conductivity $\kappa$ associated to some carrier satisfies
$$\kappa = \frac 1 3 n c_v u \lambda \tag{1}\label{1}$$
where $n$ is the carrier number density, $c_v$ is the heat capacity per carrier, $u$ is the carrier speed and $\lambda$ is the mean free path. The mean free path is related to the scattering relaxation time of the carrier $\tau$ by
$$\lambda = u \tau \tag{2}\label{2}$$
You can interpret $\tau$ as the average time between two successive "collisions" (scattering events) of the carrier. Therefore, from \ref{1} and \ref{2}, you can immediately see that in an infinite perfect harmonic crystal, in which no scattering takes place, $\kappa$ would be infinite, since there would be no scattering and therefore $\tau = \infty$ (in a finite crystal, you would still have scattering from the boundaries).
In a realistic physical systems, phonons are scattered by other phonons, electrons, defects (impurities) and boundaries. To take these effects into account, you have to leave the harmonic approximation and consider also anharmonic terms in the Hamiltonian. It was shown by Pierls [a] that it is the anharmonicity, together with the discrete nature of the crystal lattice, that generates thermal resistance.
In particular, when talking about phonon-phonon and phonon-electron scattering, a distinction is usually made between "normal" processes, which conserve the total wave vector, and "Umklapp" processes (or U-processes), where the total wave vector is changed by a reciprocal lattice vector. To know more about this distinction and its (debatable) utility, you can take a look here.
All these events contribute to the relaxation time $\tau$, and their contributions are taken into account through Matthiessen's rule, which states that the total relaxation time $\tau$ can be calculated as
$$\frac 1 \tau = \sum_i \frac 1 {\tau_i}$$
where $\tau_i$ are the relaxation times associated to the possible different scattering events. So this is how, "mathematically", scattering events influence the thermal conductivity of a material.
How does this phonon picture explain the fact that when we heat a bad conductor the heat propagates gradually from the hotter to the cooler end? If they are delocalized collective excitations, shouldn't they heat up all parts of the substance at the same time?
I think that here the answer is simply that you need some time for the "delocalized collective excitation" you are talking about to set in. This must be true even for a perfect crystal, even if its infinite thermal conductivity would seem to suggest the opposite, otherwise we would have instant propagation of a signal (the vibration of the atoms). You think about the phonons as "delocalized collective excitation", but in reality they are much more similar to wave packets arising from a superposition of these collective excitation (the normal modes of the crystal). Maybe I am not being 100% precise in my terminology here, but I hope that I managed to convey the general meaning of what I have in mind.
References
[a] R. Peierls, “Zur kinetischen Theorie der Wärmeleitung in Kristallen” ("On the Kinetic Theory of Thermal Conduction in Crystals")
Ann. Phys. 395, 1055–1101 (1929)