Conservation of angular momentum is closely related to the fact that there is no privileged spatial direction.

Let's start with your intuition for linear momentum:

> A single particle keeps drifting in a direction unless forced to
> change. And a isolated group of particles too keep drifting as a whole
> in an average direction unless forced to change. The law for an
> isolated group of particles is exactly of the same form as that for a
> single particle.

Another way to say this is: if I drift along a free particle, with the same constant velocity it has, it appears motionless to me. This is because there is no privileged position in the universe, so as long as I am at rest, it does not make a difference if my "absolute position" changes or not. There is no "absolute" position to start with. 

This does not depends on whether there is one or several particles, so similarly if I drift along a free system of particles, the center of mass of the system will appear motionless to me.

When it comes to angular momentum, the fact that there is no privilege position takes another twist, precisely because angular momentum is defined with respect to a specific point in space.  

> A particle cannot keep moving in a circle on it's own, whereas a
> group of particles keep rotating as a whole

A group of particles *does not keep rotating*. See the case described in  [this question](https://physics.stackexchange.com/q/62470/109928): we have two particles (astronauts there) linked by a taut rope and spinning around their centre of mass. When they let go of the rope, they each follow a straight trajectory: observed from an inertial (non-rotating) frame at rest with respect to their center of mass, they do not have any angular speed anymore. Angular momentum is conserved *because they move away from the center of mass*, not because they keep moving in circle.

> The law for a single particle leads to a law that is of a different form for the group.

That's because the single particle is a solid object: its whole mass distribution is maintained by cohesive forces. If you abstractly divide it into different parts, each one of these parts would follow a straight line and move away from the center of rotation, if it was not bound to the other parts. Exactly as the astronauts did: they kept rotating as long as they formed a bound system.

So the law of angular momentum conservation that indeed is the same for a single particle and a group is not the law you are thinking of: it is not a law about keeping moving in circle.



 that you could imagine It is not at all case is a degenerate one: the particle being identified with its center of mass, which is also its center of rotation, 


It is the very same thing for angular momentum.

Consider a spinning particle. If I spin along with it (I put myself in its axis of rotation and observe it while rotating at the same constant angular speed as it is), it will appear to me to be not rotating at all. This is because there is no privileged direction in the universe, so as long as I am regularly spinning it does not make a difference (in some sense, see just below) whether I rotate or not: there is no absolute reference direction. 

There is actually something new though: because a rotating reference frame is not inertial (precisely because it posits a privileged spatial axis by rotating round it, see how this is related to the first part of our discussion), I can now observe centrifugal forces around me. 

Now let's see how it works for a system of particles: it is the same idea. This time I will take a specific example, the one from [this question](https://physics.stackexchange.com/q/62470/109928): we have two particles (astronauts there) linked by a taut rope and spinning around their centre of mass. How do we intuitively see the conservation of angular momentum after the link is broken?

Well let's spin along them, while they are still connected: we observe that they are not spinning at all, but that they are subject to centrifugal forces (which keep the rop taut). When they let go of the rope, the centrifugal forces push them apart, away from their center of mass, but, more importantly: in a straight line. There is still no spinning, no global change of direction for the overall system. 

We say the angular momentum is being conserved, but it amounts to say that if there is no privileged direction to start with, there is no reason for such a direction to show up at any later time in the evolution of the system.