Is there a physical interpretation to invariant random matrix ensembles? Disclaimer. I am a graduate student in pure mathematics, so my knowledge of physics more advanced than basic 1st/2nd year undergraduate physics is very limited. I welcome corrections on any misconceptions present in my question.

Background.
From readings I have done online (mainly on Wikipedia and online lecture notes),
I understand that according to the theory of quantum mechanics,
the possible energy levels of a quantum system are sometimes described using the eigenvalues of a Hermitian operator $H$ called the Hamiltonian operator on a (possibly infinite dimensional) Hilbert space,
which is (for convenience) sometimes approximated using a large $n\times n$ Hermitian matrix $\widehat{H}_n$ (i.e., with $n\gg0$).
Furthermore,
I understand that for many very complicated and rapidly fluctuating systems
(such as heavy atomic nuclei),
one is often mostly interested in a generic or typical hermitian operator,
which can be modelled by defining our approximation $\widehat{H}_n$ as a random hermitian matrix.
According to what I have written so far,
I understand at least partially the interest of studying the spectrum of large Hermitian matrices for applications in physics.
However,
when applications in physics are mentioned in random matrix theory,
there is usually a lot of emphasis on the invariant random matrix ensembles,
which are ensembles of random matrices $M$ whose distributions are invariant under conjugation by matrices from one of the classical matrix Lie groups.
For example,
the random matrix $M$ is said to belong to a unitary ensemble if the probability distribution of $M$ is equal to the probability distribution of $UMU^*$ for every unitary matrix $U$.
This leads me to the following question:

Question. Is there a physical reason why physicists are especially interested in the invariant ensembles?
While researching this,
I came across the following paragraph in a paper which seems to address this question

Physically, an invariant random matrix ensemble describes extended (but phase-randomized) states, where the localization effects are negligible. In contrast to that any non-invariant ensemble accounts for a sort of structure of eigenfunctions (e.g. localization) in a given basis which may be not the case in a different rotated basis (remember about the extended states in the tight-binding model which are the linear combinations of states localized at a given
site).

but given my lack of knowledge of physics jargon,
I don't quite understand what is meant by "localization effects are negligible".
 A: I'm not sure whether this is correct, but I think the physical motivation for the random matrix ensembles to be related to symmetry. In physics you often have, for example, rotational symmetry, so that your potential energy or action or something is invariant when your position vector undergoes a rotation: $\vec{r} \rightarrow \mathbf{R} \vec{r}$. This requirement, that no direction is special, means that your quantity should only depend on the magnitude of position (or velocity, or whatever).
In the same way, if your quantum theory is random, it shouldn't privilege any state $|\psi \rangle$. Over any other state. Thus, your ensemble of matrices should be invariant under unitary transformations (the equivalent of rotations, in the sense that the magnitude is not changing) $|\psi \rangle \rightarrow \hat{U} |\psi \rangle$. However, if we think of that unitary transformation as acting on the Hamiltonian rather than the state, then our ensemble should satisfy
$$P(\hat{U}^\dagger \hat{H} \hat{U})=P(\hat{H}).$$
You might ask why we should be restricted to unitary transformations. In the classical picture, we stick to rotations because it is often the case that direction doesn't matter but magnitude does (for example, for the kinetic energy as a function of velocity). Here, though, I think we just stick to unitary transformations because quantum states are normalized, so it doesn't usually make sense to talk about changing their magnitude.
A: Here's one example of the use of random matrix theory in physics, more specifically in inflationary models where many fields contribute to inflation (reference).
In this context, random matrix theory is used to maintain a high level of generality: Lacking (a way of obtaining) complete information about the inflationary potential $V(\phi_i)$, one wants to find out what happens if one tries to assume only a few properties of the potential which each field is subjected to. It can then be informative to ask oneself what the generic properties of potentials obeying these conditions are.
As it turns out, the basic assumptions that seem reasonable to impose in any realistic theory (of course, this is up or debate) restrict the Hessian matrix associated with the potential of the fields to a well known ensemble of random matrices, the Gaussian Orthogonal Ensemble.  One can then use random matrix theory to study the generic features of these potentials, finding out how much exactly we can say about a broad class of possible inflationary potentials.
A: A random scattering matrix must (in a large number of scenarios) obey a number of physical constraints. Principle of these are conservation of energy, reciprocity and time reversal symmetry. In a quantum context you also have conservation of probability (i.e. normalisation of the wavefunction). These impose some necessary symmetries onto the scattering matrix (see e.g. doi.org/10.1103/PhysRevResearch.3.013129)
A: I’m not really a physicist, or even proficient in math, but I think I see why you’re asking the question. Correct me if I’m wrong, but it seems like you’re hinting at the fact that outside of just mathematical theory, controlled lab environment, simulations.. it doesn’t seem useful to work with invariant random matrixes. (Things tend to be variant in nature and there are tons or external factors).
My answer is this, because I can think of nothing else: quantum logic gates.
In computing, when we generate several versions of a randomized environment to gather data from, we can choose some values to remain constant between the environments. I don’t know all the terminology y’all are throwing around, but this seems like an example of that, Randomly generated values are a common use for quantum computing. Of course, I could be completely wrong here!
A: An additional use of random matrix theory appears in quantum gravity. Random matrix theory can be used to analyze the transition amplitude between two states and also to find a corresponding phase-diagram. Here Two-matrix model with ABAB interaction  or here
The phase diagram of the ABAB model  you can find a connection to dynamical triangulations, which is based on Regge calculus. It has an extension to higher dimensions Colored tensor models, which can be used in 4-d quantum gravity, thus its highly relevant!
A: I think that it is worth thinking about the concept of « uniformity ». What makes the uniform measure on a finite set special? It is the only (probability) one that is invariant under every symmetry of the space, that is, under every bijection of the finite set.
Now, it seems natural to think that the good group of symmetries of a Hilbert space is the group of unitaries. So, assuming that it is worth considering random matrices, and assuming that it is worth considering probability measures that are as « uniform » as it is possible, it is natural to consider invariant unitary ensembles.
