It is a straightforward evaluation evincing the power, charm and efficiency of matrix mechanics; I have no idea why the author feels inclined to work in the coordinate representation — let me know if you did. In any case, everything I do below concerns operators, so I drop all the superfluous carets: they are implied to hat everything here!
The fundamental adjoint action Hadamard identity one always uses is
$$
e^{A} B e^{-A} \equiv \operatorname{Ad}_{e^A} ~ B = e^{\operatorname{ad}_A} B\equiv e^{[A,\bullet ] } B= B+ [A,B]+ \frac{1}{2} [A, [A,B]]+...
$$
but (oh, the joy!) only the linear terms in A will survive in our manipulations, since the arguments thereof are always linear, by $[q,p]=i$.
Just compute away:
$$\bbox[yellow]{
e^{-\frac{\omega^2}{4} [q^2,\bullet]} ~ q =q, \qquad e^{-\frac{\omega^2}{4} [q^2,\bullet]} ~ p = p -i\frac{\omega^2}{2} q \\
e^{-\frac{1}{2} [p^2,\bullet]}~ p = p , \qquad
e^{-\frac{1}{2} [p^2,\bullet]} ~q = q + iq \qquad}~.
$$
Consequently,
$$
T p T^{-1}= e^{-\frac{\omega^2}{4} [q^2,\bullet]} e^{-\frac{1}{2} [p^2,\bullet]}e^{-\frac{\omega^2}{4} [q^2,\bullet]} ~ p \\
\qquad \qquad = (1+\omega^2/2) p -i(\omega^2/2)(2+\omega^2/2)q ~~, \\T q T^{-1}= e^{-\frac{\omega^2}{4} [q^2,\bullet]} e^{-\frac{1}{2} [p^2,\bullet]}e^{-\frac{\omega^2}{4} [q^2,\bullet]} ~ q \\ = i p +(1+\omega^2/2)q ~~.
$$
This is your target vector equation if only you reverse the order of p and q in your operator vectors. *I believe the correct equation you meant to write is actually $\hat{T} \left(\begin{matrix}\hat{q} \\ \hat{p} \end{matrix}\right)\hat{T}^{-1} = M \left(\begin{matrix}\hat{q} \\ \hat{p} \end{matrix}\right) $ .