From Noether's theorem we understand that conservation of angular momentum means that the laws of nature have rotational symmetry. From cosmology we understand that the universe has no center. But rotation makes no sense without a center. How can we claim rotational symmetry for something that has no center?
The laws of nature satisfy rotational symmetry about every point (or in other words, the general law of conservation of angular momentum holds everywhere in space). So you can pick any origin you'd like for your coordinate system, and rotate about that origin any way you like, and the laws remain the same.
But rotation makes no sense without a center. How can we claim rotational symmetry for something that has no center?
An easy counter example is an infinite flat 2D plane. An infinite plane has no specific center, but it has rotational symmetry. You can rotate about any point.
If an object has rotational symmetry and lacks translational symmetry then it will have an identifiable center.
I think you may be bringing in a cosmological argument that doesn't really belong here:
First of all, Noether's theorem can be applied when the system of interest has rotational symmetry about one special point only: e.g., a particle moving in an external potential that depends only on the radial coordinate relative to that point. Such a system is not rotationally symmetric around any other point. And as you say, angular momentum is always conserved only relative to the special point for which a Noether invariance has been identified.
A special case is an isolated system (no external forces), which indeed shows a conserved angular momentum no matter what reference point you choose. This is true even if the objects in the system aren't distributed with any particular symmetry whatsoever. The symmetry that matters for Noether's theorem is really the invariance of the equations of motion under rotations, not mundane things like the mass distribution or the force vectors between the objects.
Now when people say "the Universe has rotational symmetry," they might mean different things. For example, the cosmologists you refer to may actually be talking about the commonly made approximation that the mass distribution of the Universe is isotropic at large distance scales. They furthermore assume that this symmetry is valid around every point. This approximation needs to be made if you wish to obtain analytically solvable equations from Einstein's theory of general relativity.
But that's actually quite unrelated to the symmetry that goes into Noether's theorem. The latter is not so much a cosmological statement, but a statement about the equations of motion that govern everything in the Universe, together with the fact that the Universe is by definition an isolated system. The cosmological models in current use are (highly symmetric) solutions to those equations of motion, but the symmetries Noether talks about are inherent in the equations of motions themselves -- even if they could also be solved by arbitrarily asymmetric mass distributions.