What's the physical significance of the off-diagonal element in the matrix of moment of inertia In classical mechanics about rotation of rigid object, the general problem is to study the rotation on a given axis so we need to figure out the moment of inertia around some axes. In 3-dimensional case, we have a matrix (i.e. moment of inertia tensor)
$$
  I = 
\left(
\begin{matrix}
I_{xx} & I_{xy} & I_{xz}\\
I_{xy} & I_{yy} & I_{yz}\\
I_{zx} & I_{zy} & I_{zz}
\end{matrix}
\right)
$$
I am curious what's the physical significance of the matrix element. I guess the moment of inertia in element $ij$ is the moment of inertia when the object is rotating about the axis $ij$. For example, $I_{xy}$ is the moment of inertia when the object is rotating about $xy$ axis and $I_{yy}$ is the moment of inertia when the object is rotating about $y$ axis, is that correct?
When I read further into the text, it then introduce the a method to diagonalize the moment of inertia tensor such that the non-vanishing elements only appears on diagonal. In that case, the text call the diagonal elements as the principal moment of inertia, my question is what's the physical significance of the principal moment of inertia?
 A: Your guess is wrong, because first of all, there is no "xy"-axis, there's only an "xy"-plane, the plane perpendicular to the z-direction.
As you note, your text describes how the matrix can be made diagonal. Not sure how well the text explains this, but the way to get the matrix to be diagonal is to go to a new coordinate system: You started out with $x$, $y$ and $z$ in "some" way. But your initial $x, y$ and $z$ might not be the "natural" axis of your object. If you go to new axes $x', y'$ and $z'$ such that the tensor is diagonal, you are now in a coordinate system where the object will rotate freely only around those new axes. That's why they are called the principal axes. 
The nice thing about a diagonal matrix is that matrix-vector products in them are very easy to calculate. For example, if you rotate with angular velocity $\omega$ around an axis $\vec \omega = (\omega_x, \omega_y, \omega_y)$, then the rotational energy of that is 
$$\frac{1}{2} \vec{\omega}^T \cdot I \cdot \vec{\omega}$$
If $I$ is diagonal, this simply becomes
$$\frac{1}{2} \omega_x^2 I_{xx} + \frac{1}{2} \omega_y^2 I_{yy} + \frac{1}{2} \omega_z^2 I_{zz}$$
It is, however, correct that $I_{yy}$ is the moment of inertia for rotation around the $y$ axis. The off-diagonal elements would come into play if you don't go to the coordinate system that makes $I$ diagonal and then look at rotations around axes different from the coordinate axes. For example 
$\vec \omega = (\omega, \omega, 0)$. There, the rotational energy would be 
$$\frac{\omega^2}{2}(I_{xx} + I_{yy} + 2 I_{xy})$$.
So if you want to compute the rotational energy in a coordinate system where $I$ is not diagonal, you get all those pesky off-diagonal matrix elements in there, cluttering up your expression, whereas you can get rid of all of them if you transform the coordinate system so that they are $0$. 
This diagonalization, btw, is something that comes up very often in all fields that deal with matrices, just because it makes dealing with those matrices so much easier, and the remaining diagonal entries (the "eigenvalues" of that matrix) contain a lot of useful information about the nature of the matrix.
A: For a non-spherical object, there is a unique direction along which the object is "longest", that is, to have the smallest moment of inertia if rotated about an axis with that direction.  The material of the object are as close to that axis as can be, compared to other directions.  
There's another direction perpendicular to that about which the moment of inertia is maximum.   
Then  finally we have an intermediate amount of moment of inertia in a third direction perpendicular to the previous two.  I lied; that "intermediate" moment of inertia may be the same as the minimum one or maximum one, in which case you have some freedom to pick an arbitrary angle for one axis, but never mind this detail for present purposes.
A spherical object, of course, has the same moment of inertia about any axis, so is boring.  You have freedom to pick axes however you like, but never mind that special case either, since it's not interesting.
For the non-special case, we have the unique directions for minimum, maximum, and intermediate moments of inertia.  We could name these directions, the 'principal axes', with letters like, oh maybe: 'X', 'Y', and 'Z' and thus have the tensor 
$$
  I = 
\left(
\begin{matrix}
I_{xx} & 0 & 0\\
0 & I_{yy} & 0\\
0&0 & I_{zz}
\end{matrix}
\right)
$$
These three numbers are physically meaningful, giving a general overall measure of size and mass distribution of the object.
But maybe the object is positioned at some crazy angle with respect to things we care about, like our nice level tabletop, our local notion of 'east' and 'north'.  So we must rotate the object and its various physical vectors and tensors (and spinors if it's a fermion).  An arbitrary rotation is described by three angles (e.g. Euler angles).  The fully general $I$ tensor then has six independent quantities.  We see nine components, but they count as six due to always being a symmetric tensor.  
The physical significance of the off-diagonal components is that you're using a coordinate system not aligned with the principal directions of the object.  They tell us nothing interesting about the object itself.
A: The fundamental equation that contains $I$ is:
$\vec{L} = I \vec{\omega}$
where $\vec{L}$ is the angular momentum vector, and $\vec{\omega}$ is the angular velocity vector. The fact that $I$ is a tensor (and not just a scalar) means that $\vec{L}$ and $\vec{\omega}$ do not necessarily point in the same direction.
When you want a particular matrix representation of the moment of inertia tensor, you may get off-diagonal elements. However, since by its very definition, any matrix representation of $I$ is a symmetric matrix. Any real, symmetric matrix can always be diagonalized by orthogonal matrices. What this physically means is that, for at least one choice of $x$, $y$, $z$ axes, the representation of $I$ will be perfectly diagonal, with all the off-diagonal elements zero.
However...
This does not at all mean that the off-diagonal elements are not important. And I will not escape the question of their meaning.
There is also the equation describing the dynamics of the whole business:
$\frac{d \vec{L}}{ dt} = \vec{\tau}$
This is analogous to $\frac{d\vec{p}}{dt} = \vec{F}$ which describes linear motion; here of course $\tau$ is the torque, while $\vec{L}$ is the angular momentum. 
Substituting $\vec{L} = I \vec{\omega}$ and assuming a time-invariant moment of inertia tensor, this becomes:
$I \frac{d\vec{\omega}}{dt} = \vec{\tau}$
Now, the torque you are applying does not really have to coincide with a vector that diagonalizes your $I$ (such vectors are called principal axes). For instance, take the wheel of your car. It is driven by a rod, going through its axis. Let us take that to be the $z$-axis. So, in this case, $\tau_z$ is non-zero, while $\tau_x$ and $\tau_y$ are zero.
The above equation is actually three equations, let me show the third one explicitly:
$I_{zx} \frac{d\omega_x}{dt} + I_{zy} \frac{d\omega_y}{dt} + I_{zz} \frac{d\omega_z}{dt} = \tau_z$
Now, if the off-diagonal elements $I_{zx}$ and $I_{zy}$ are zero (assuming zero initial conditions) your wheel will just acquire $\omega_z$, that is, it will start rotating nicely about the axis you are applying the torque on. 
However, if they are not zero... $\omega_x$ and $\omega_y$ will not remain zero. That is, the wheel will tend to start to rotate around the other axes as well! So, you will not get a rotation around the $z$-axis, and the wheel will tend to wobble! This is the condition where you take your car to the shop, and have the wheels balanced!
So, the off-diagonal elements really mean that any attempt to rotate the body by applying a torque about a given axis will not result in a rotation about just that axis and there will be rotation around the other axes as well. 
Note that if, for instance, we have $I_{xy} \neq 0$ and $I_{xz} = I_{yz} = 0$, a torque around the $z$ axis will result in a balanced rotation, while a torque around the $x$ or $y$ axes will not...
