The scattering rates for different processes (impurities, phonon-phonon, etc.) are typically calculated using Fermi's golden rule and then summed up to get a total scattering rate. (Summing them together goes by the fancy name of Matthiessen's rule.) The relaxation time is the reciprocal of this total scattering rate.
The question is then, what perturbing Hamiltonian do I put into Fermi's golden rule? (The unperturbed Hamiltonian is taken to be the quadratic terms in the interatomic potential, which lead to phonons looking like uncoupled simple harmonic oscillators. See Ashcroft and Mermin Eqs. 22.9-11.) If you want phonon-phonon scattering due to three-phonon interactions, then you take the cubic terms in the interatomic potential as the perturbing Hamiltonian. (See Ashcroft and Mermin Eqs. 22.5-7). If you want four-phonon interactions, then you take the quadratic terms. Etc. In general, the interatomic potential is quite complicated and is calculated numerically, but if you're only looking for an estimate, you can sometimes use a simple potential like the Lennard-Jones potential (Ashcroft and Mermin Eq. 20.2).
There is an extra complication in that not all phonon-phonon scattering events disrupt heat flow, and how to best account for this in the relaxation rate is still a matter of debate. However, if you're looking for an estimate, you don't need to worry about normal vs umklapp scattering. (That said, in some cases normal vs umklapp scattering is very important. See, for example, second sound.)
If you're interested in impurity scattering, then your perturbing potential is that of a different atom.
If you're interested in scattering from surfaces, you typically take a simpler approach --- like that phonons are scattered randomly by the surface, and the rate this happens at is determined by how frequently phonons run into the surface. (This requires knowing the dimensions of the structure and how fast the phonons travel.)
If you want actual, rough equations for different scattering rates, wikipedia is a good place to start. Those equations aren't enough to do an accurate calculation, but they give you an idea of how the rates scale with frequency.