Why $3$-dim isotropic harmonic oscillator's symmetry is not $O(6)$ but $U(3)$? 3D harmonic oscillator's Hamiltonian is 
$$H=\sum_{i=1}^3p_i^2+q_i^2$$
Why all textbooks say that its symmetry is $U(3)$. But I think it's $O(6)$. Because the rotation of $6$ coordinates in phase space is invariant.
 A: The statement holds both in the classical and the quantum case:
Clasical mechanics
The symmetry group of the function $H:\mathbb{R}^3\times\mathbb{R}^3\rightarrow\mathbb{R}$ given by $H(q,p)=\sum_{i=1}^3(q_i^2+p_i^2)$ is indeed $O(6)$, as you guessed. However, when talking about symmetries an extra condition is needed: they should preserve the symplectic structure. 
To understand what is meant by this, remember that Hamilton's equations are $\dot{q_i}=\partial H/\partial p_i$ and $\dot{p_i}=-\partial H/\partial q_i$. They may be written as
\begin{equation}
  \left(\begin{array}{c}\dot{q}\\\dot{p}\end{array}\right)=
  \left(\begin{array}{cc}0 & I \\ -I & 0\end{array}\right)
  \left(\begin{array}{cc}\frac{\partial H}{\partial q} \\
  \frac{\partial H}{\partial p}\end{array}\right)
\end{equation}
where $I$ is the $3\times 3$ identity matrix. Thus, if we want the form of Hamilton's equations to be invariant under a transformation
\begin{equation}
  \left(\begin{array}{c}q\\p\end{array}\right)\mapsto
  M\left(\begin{array}{cc}q \\ p\end{array}\right)
\end{equation}
it should satisfy $M^T \Omega M = \Omega$ where
\begin{equation}
  \Omega = \left(\begin{array}{cc}0 & I \\ -I & 0\end{array}\right).
\end{equation}
The group of such matrices $M$ is the symplectic group $Sp(6,\mathbb{R})$. The group of symmetries should now be the intersection of $O(6)$ and $Sp(6,\mathbb{R})$. Using the general property $U(n)=O(2n)\cap Sp(2n,\mathbb{R})$ we get that the group we are looking for is $U(3)$.
Quantum mechanics
An explanation of why the symmetry group can't be $O(6)$ (in a more general case) can be found in the answer to this question, as pointed out in the comments.
Now we want to see that the symmetry group is $U(3)$. We require the invariance of $H$ and of commutation relations $[q_i,q_j]=[p_i,p_j]=0$, $[q_i,p_j]=i\delta_{ij}$, which can be written collectively as $[(q,p)^T,(q,p)]=i\Omega$. Therefore the commutator preserving transformations are $Sp(2,\mathbb{R})$ and by the same reasoning as before the group of symmetries is $U(3)$.
A: The canonical references for this are:


*

*Fradkin, D. M. "Three-dimensional isotropic harmonic oscillator and SU3." American Journal of Physics 33.3 (1965): 207-211, 

*Harvey, Malcolm. "The Nuclear SU3 Model." Advances in nuclear physics. Springer US, 1968. 67-182


In short, the symmetry group of the $n$-dimensional harmonic oscillator is best understood by expressing the momenta and positions using the creation and destruction operators $\hat a_k^\dagger$ and $\hat a_k$, with $k=1,\ldots,n$.  This makes it clear that the symmetry group is not limited to real transformations and to "point" transformations mixing positions and moments separately.
Any complex $n\times n$ transformation $U$ sending $\hat a_k\to U\hat a_k$ must also send $\hat a_k^\dagger\to \hat a_k^\dagger U^\dagger$.  Thus 
$H=\sum_k \hat a_k^\dagger \hat a_k$ invariant if $U^\dagger U$ is the unit matrix, whence $U$ is an $n\times n$ unitary matrix.
