This is actually more of a linguistic question than a physics one. And, like most linguistic questions, there are no normative rules describing when the term is used. We try to capture the use of these terms after the fact, but there's no prescribed rule.
The best way to approach this is from the etymology of degree. We find it is used to describe steps of a process. So if a subject permits division into steps, "degree" is often a word that follows shortly behind.
One conjecture I have seen is that "degree" often appears when the most natural division is simply too big to be used. For example, the natural unit for angles is, well, full circles (what we now call 360 degrees). However, this is too large to be generally applicable, so we map a uniform scale to this to describe smaller divisions. In the case of angles, maps of the heavens (1 rotation per year) across the 365 days quickly turns to a much more convenient 360 degrees. In the case of Celsius, treating 0 as "coldest water" and 1 as "hottest water" gets divided up into 100 even divisions. One might extend this conjecture to suggest that Kelvin doesn't have "degree" used with it because the divisions were already tied to small things (degrees Centigrade), and did not need to undergo this uniform division process again. (though I will note that "degrees Kelvin" is not an uncommon phrasing)
This is not a solid rule, but this conjecture does suggest why there doesn't seem to be a solid rule. It suggests historical abilities to work with these quantities drives the linguistics.
