Angles are a peculiar case, and don't really fit the pattern of other physical quantities. But the SI brochure isn't giving a complete picture of what's really going on here. It's quite complicated.
Physical quantities are characterised by their symmetries. The physical quantities exist in their own space, and there are symmetry operations that map them on to one another while preserving, in some sense, the physics. We can build a coordinate system for these physical quantities by picking a 'standard' example, and then specifying any other instance of the physical quantity by listing a set of symmetry transformations needed to get to it from the 'standard' value. The symmetry transformations can be parameterised by various sorts of 'numbers' (that might or might not be the Reals).
This is called a 'homogeneous space' by the mathematicians. If there is a unique transformation to get to any other instance (so the coordinates are unambiguous and well-defined), then it is called a 'principal homogeneous space' or 'torsor'.
Many physical quantities have a scaling symmetry. So we can specify any other quantity by scaling the standard unit by a scale factor. This turns the symmetry group action on some abstract space of physical quantities into a vector space over the Reals, which is much easier to handle.
Angles don't have a scaling symmetry, they have a rotational symmetry, which is periodic. A rotation of $2\pi$ radians is the same as the identity, so there is a natural 'scale' already defined for them. The symmetry is already broken.
However, we can still find a mapping between the angles and the Reals, by wrapping the Real line around the unit circle infinitely often. And then we can apply scaling transformations to the Reals anyway, despite the fact that it breaks the symmetry. That's how we get 'degrees' and 'radians' as alternative angular 'units'.
This mapping uses the Lie algebra for the rotation group. It works a bit like a generalised logarithm. The rotation $\mathrm{exp}(i\theta)$ has logarithm $i\theta$ where we can treat $\theta$ as a Real number and $i$ acts like a 'unit' for rotation. In fact we can generalise this to more than two dimensions by setting $i$ to be a bivector, a quantity that specifies an oriented plane element with magnitude, and has the property that its square is -1. This is actually constructed as a ratio of two vectors pointing in different directions, and rather than just setting a size it also specifies the orientation of the plane in which the angles lie.
As a historical note, this was how Hamilton invented the Quaternions - as the ratio of a pair of vectors in different directions. He had hoped to be able to use them to create an algebra for 3D vectors in the same way Complex numbers worked for 2D vectors, but he was never able to get it to work properly because bivectors don't work the same way as vectors. (Complex numbers aren't vectors, either, for the same reason.) The two can be unified in Clifford algebras, but that's another story.
So, there is an analogy here between scaling transformations and rotations, in that one can be implemented by multiplying by $\mathrm{exp}(\lambda)$ and the other is implemented by multiplying by $\mathrm{exp}(i\theta)$. Multiplication is turned into addition, and a change of unit becomes an additive offset. The choice of unit in these coordinates is effectively a choice of origin.
Thus we can see in several ways that the algebra of linear units is not the same as the algebra of angles. We can't use the same "scale a standard unit" trick to represent them. However, this does not mean that angles are Real numbers, either. We can represent them as unit Complex numbers in 2D, or as unit Quaternions in 3D, but using the multiplication operation to combine them instead of addition. (Or we can take the logarithm to get an additive operation, at the cost of making the mapping periodic and thus ambiguous.) But whatever we do, we're not getting the same sort of object as the dimensionless ratio of two scalar lengths - a Real number.
This isn't quite the end of the story. If we consider an angular velocity, we find that the original $0=2\pi$ identification no longer applies. $2\pi$ radians per second is physically different from 0 radians per second. We have to work now in the Lie algebra, which can now be thought of as the tangent space to the rotation group at the identity. Angular velocity is thus properly a bivector, although in 3D we can use the dual to it and treat it as a vector.
Now angular velocity does have the sort of scaling symmetry that other physical quantities do, and thus the angular units in a compound unit like radians per second can more easily be considered to be a 'dimensionful' unit.
As a final twist on the entire topic, the translation, rotation, and dilation transformations can all be considered as special cases of the same group. A translation is 'a rotation about a point at infinity', pointing off perpendicular to the direction of movement. Consider the limit of rotations about a point as that point is moved further away and the angle shrunk proportionately. It is also 'a dilation centred on a point at infinity', this time the infinite point is off in the same direction as the translation. We have a single transformation group covering all cases, and so there is the potential to unify angles and lengths as essentially the same thing.
This is the point of view of the Klein geometries. Euclidean geometry is a special case of a Klein geometry, but its metric is degenerate, which is what stops us directly comparing lengths and angles. However, we can start to see how they fit together if we look at other Klein geometries like the Elliptic geometry. This is the geometry of a sphere with antipodal points identified, or equivalently, the geometry of lines through the origin. Now we discover that lengths on the surface of the sphere are in fact angles, measured from the centre. So treating this angle as a bivector as before, representing the plane of the angle, we find one component spanning the plane is embedded in the geometry, and the other is directed out of the geometry towards the origin. The angle is a ratio of the two lengths.
Now when we switch to the Kleinian picture for Euclidean geometry, we flatten the sphere out into a plane. This is the picture we normally use for projective geometry, where we represent points of the Euclidean plane as a homogeneous triple $(x,y,w)$ and normalise $w$ to get points of the plane $(x/w,y/w)$. This $w$ is measuring the distance from the plane to the origin, and gives the entire geometry a natural scale. And now we can see that what we consider to be 'lengths' are in fact angles measured from this projective origin, where these angles are dimensionless ratios of the length in the geometry and the 'distance' from plane to origin in the $w$ direction perpendicular to the geometry.
Thus, lengths are really just angles measured at right-angles to reality!
We can also interpret the other dimensionful quantities as behaving the way they do because they, too, are essentially lengths. Space and time are of course unified in special relativity. Mass is equivalent to energy, which in quantum mechanics is related to frequency in time. So mass is really just a reciprocal length. And electric charge can be interpreted geometrically as a sort of 'angular momentum' in an internal dimension of space. So fundamentally, all the physical quantities have a geometrical interpretation, and they are all measured by means of angles. The degenerate Euclidean metric causes most of these angles to behave like the Reals, the Lie algebra of the rotations, with a scaling symmetry. But the algebra of non-degenerate angles is quite different, and so the 'unit' construction cannot be applied in the same way.
The SI brochure is simplifying a very complicated issue. Dimensions and units are a lot more complicated than most people think!
