Is it sloppy to say that the isotropy of space leads to conservation of angular momentum? Sometimes it is said that the isotropy of space leads to conservation of angular momentum. But the derivation of conservation of angular momentum follows not from isotropy of space but that of the action. 
We know that the space around us is not isotropic but angular momentum is still conserved if the action has rotational symmetry. 

Is it, therefore, sloppy to say that the isotropy of space leads to conservation of angular momentum?

 A: The terms "isotropy of space" and "rotational invariance of the action" are taken as synonymous, inasmuch as one implies the other in general relativity. The observed anisotropies of space are taken as being due to the presence of matter that takes part in the exchange of angular momentum, so it would be more accurate to say that "if space is isotropic in the absence of matter/energy, then angular momentum is conserved."
In a little more detail, when the stress-energy tensor is zero (absence of matter/energy), the curvature is zero. In terms of the Einstein field equations:
\begin{align}
R_{\mu\nu} - \frac{1}{2} R g_{\mu\nu} + \Lambda g_{\mu\nu} &= \frac{8\pi G}{c^4} T_{\mu\nu} = 0 \Rightarrow \\
R_{\mu\nu} & = 0,
\end{align}
that is, the Ricci curvature tensor is zero. If we assume that the Christoffel symbols, are metric compatible and torsion free, they have this form:
$$\Gamma^i_{\hphantom{i}kl} = \frac{1}{2}g^{im} \left(\frac{\partial g_{mk}}{\partial x^l} + \frac{\partial g_{ml}}{\partial x^k} - \frac{\partial g_{kl}}{\partial x^m}\right).$$
Because the Ricci curvature is defined by the Christoffel symbols as
$$R_{\mu\nu} = \partial_\rho\Gamma^\rho_{\hphantom{\rho}\nu\mu} - \partial_\nu\Gamma^\rho_{\hphantom{\rho}\rho\mu} + \Gamma^\rho_{\hphantom{\rho}\rho\lambda} \Gamma^\lambda_{\hphantom{\lambda}\nu\mu} - \Gamma^\rho_{\hphantom{\rho}\nu\lambda} \Gamma^\lambda_{\hphantom{\lambda}\rho\mu},$$
$R_{\mu\nu} = 0$ implies a number of things, but the relevant implication to your question is that the metric is rotation invariant. A rotation invariant metric is synonymous with isotropy of space.
Admittedly, requiring $T^{\mu\nu}=0$ is too strong of a condition. It is sufficient that $T^{\mu\nu}$ is invariant under spatial rotations (isotropy of stress/energy) to get isotropy of space.
The connection to the action is that the Einstein field equations are derived from finding the stationary point of the Einstein Hilbert action added to the action that describes the rest of the physics with respect to the metric tensor, $g_{\mu\nu}$.
A: Sure, I guess so. But in GR, rotationally invariant metrics have Killing fields that generate the symmetry rotation, and the inner product of geodesic four-velocities and these Killing fields is conserved, allowing us to define a conserved "angular momentum" which really does follow directly from the isotropy of space itself, without needing to specify an action (as long as you're okay with assuming that free-falling particles move along geodesics).
A: @tparker is right in that generally the isometries of space time (i.e., a Killing vector field, or a Lie derivative of the metric being zero) lead to a symmetry of the stress energy tensor (it's Lie derivative wrt to the same field being zero), and moreover that you can define a conserved quantity in terms of the stress energy tensor. Thus energy is conserved with timelike Killing fields, momentum with spacelike ones, and angular momentum with Killing fields obeying a rotational group (isotropy). 
As discussed in the comments to the first answer, but not fully exposed, three statements can be made: 1) the self consistency of the EFE imposes symmetries on the 'matter energy' which can form and maintain that spacetime; 2) if you ignore part of the setting (eg, the square walls) you are ignoring a 
N integral part of how the field is defined, and you won't get the conservation law that otherwise holds; and 3) clearly things are not that simple, and in fact the positing of a 'matter-energy' or stress energy tensor symmetry does not lead to, necessarily, to a spacetime symmetry. 
The equation are nonlinear and the correspondence from 'matter energy symmetries' to spacetime symmetries do not follow. There is a long history of research in that area with counterexamples and various results for conditions when they are equivalent; more generally there is a range of possible definitions of the 'matter energy symmetries' where the equivalence holds in some cases more and others less, but there is no one to one relation. See for instance a paper around 1993 describing and obtaining results of the implications of various definitions. There's been kinetic theory matter symmetries, matter  collineations, perfect fluid model symmetries, electromagnetic symmetries, and others. Part of the problem is of course that fact that in GR the gravitational energy (and momentum and stress) is part of the spacetime geometry, and one can never separate those invariantly unless one imposes the spacetime isometries first. See the paper at https://link.springer.com/article/10.1007%2FBF00672696. You can google a lot of papers on the topic, going back to the early 70. The intuitive idea that a 'matter energy' symmetry generally leads to a spacetime isometry has never been proven, but certain relationships have. 
When you see the matter distribution to be isotropic, and then assume the spacetime is isotropic, as far as I know, has never been proven. I don't believe it's been disproven either.   
