Invariance of commutator relations under change of basis

Question

Consider the following:

Let for operators $\hat A$ and $\hat B$ the following commutation relation holds:

$$[\hat A,\hat B]=\hat C \tag{1}$$

and now we know that this relation holds,

$$[\hat A',\hat B']=\hat C' \tag{2}$$

where,

$$\hat A'=T_{_{X\leftarrow Y}}A$$

$$\hat B'=T_{_{X\leftarrow Y}}B$$

$$\hat C'=T_{_{X\leftarrow Y}}C$$

(Since commutation relations do not depend on a change of basis )

Now, it's easy to prove this for a Unitary transformation $U$ (which is nothing but a change of basis) as follows:

$$[\hat A',\hat B']=\hat A'\hat B'-\hat B'\hat A'=UAU^{\dagger}UBU^{\dagger}-UBU^{\dagger}UAU^{\dagger}=U(AB-BA)U^{\dagger}=UCU^{\dagger}=C'$$

where now, (I think this is where the problem is$^1$ )

$$\hat A'=UAU^{\dagger}$$

$$\hat B'=UBU^{\dagger}$$

$$\hat C'=UCU^{\dagger}$$

Here U (which is nothing but a transformation/transition matrix) is getting applied on both ends since A is an operator here on a ket of Hilbert space as opposed to the transition matrix which just gets pre-multiplied.$^{2}$

When I try to prove $(2)$ from $(1)$ for my previous case involving transition matrix this happens;

$$[\hat A',\hat B']=\hat A'\hat B'-\hat B'\hat A'=T_{_{X\leftarrow Y}}AT_{_{X\leftarrow Y}}B-T_{_{X\leftarrow Y}}B T_{_{X\leftarrow Y}}A$$

I am stuck here and I think the resolution is related to $^{1}$ and $^{2}$

I think we should be able to prove the above relation or does change of basis (not changing commutator relations) only work for Hilbert spaces where $T$ (transformation matrices which are always unitary for orthonormal basis change for Hilbert space) on an operator behave as U()U^{\dagger} and not for a general change of basis like I tried
above for vector spaces which are .....um...non-Hilbert?

Answer 1

Summing over repeated indices, entries in the original commutator relation satisfy $C_{ij}=A_{ik}B_{kj}-A_{kj}B_{ik}$. The most general linear transformation of operators is $A^\prime_{ij}=X_{ijmn}A_{mn}$, and you're welcome to determine the condition on $X$ equivalent to $X_{ijmn}(A_{mr}B_{rn}-A_{rn}B_{mr})=A_{ik}B_{kj}-A_{kj}B_{ik}$. But as @fqq and @sslucifer note, if we want every vector to transform viz. $v^\prime=Tv$ we need $TAv=A^\prime v^\prime=A^\prime Tv$, so $A^\prime=TAT^{-1}$. (The condition $T^{-1}=T^\dagger$ preserves the inner product, but not all bases of interest are orthonormal.) This is the case $X_{ijmn}=T_{im}(T^{-1})_{nj}$, which you can verify works out. So depending on your perspective, you can see this transformation as multiplying $A$ by either one or two factors.