Why is the inner product not invariant under general coordinate transformations? This came up in some of my reading (Introduction to Tensor Calculus by Kees Dullemond & Kasper Peeters, page 15).


*

*Why is the inner product not invariant under general coordinate transformations?

*and I don't understand how the author transposed "A".

 A: The old definition is not a scalar invariant. The new one is.
In a Galilean rectangular reference frame it does remain invariant. In others it needs to be multiplied by the metric. The product you stated as the definition is not a scalar invariant, it needs to be a contraction of covariant and contravariant vectors, or include the metric.
A: First its important to be clear on what an inner product is. We can define it as a map that grabs to elements of a Vector Space and maps it into a field ie. $< \cdot , \cdot > : V\ x\ V \rightarrow R$. In your case we are interested in the real number field. Here you can see that your text refers to two different inner products and they both have this property so we can call both of these inner products although they may have very different properties. 
The "old" inner product you refer to if you actually perform the transformation you don't observe an invariance. $$ < \vec a , \vec b > \ = \ a^{\mu} b^{\mu} $$  then $$( a^{\mu} b^{\mu})' = \frac{\partial (x')^{\mu}}{\partial x^i} \ \frac{\partial (x')^{\mu}}{\partial x^i} \ a^{i} b^{i} $$ which as you can see is not the same as our initial expresion. On the other hand our second definition does achieve our goal of being invariant. If now we use $$ < \vec a , \vec b > \ = \ a^{\mu} b_{\mu} = g^{\mu \nu} \ a_{\nu} \ b_{\mu}$$ then $$(a^{\mu} \ b_{\mu})' = (g^{\mu \nu} \ a_{\nu} \ b_{\mu})' =\frac{\partial (x')^{\mu}}{\partial x^i} \ \frac{\partial (x')^{\nu}}{\partial x^{j}} \frac{\partial x^{j}}{\partial (x')^{\nu}}  \frac{\partial x^{i}}{\partial (x')^{\mu}} g^{i j} a_{j} b_{i} =a^{i} b_{i}$$ which leaves us with the same thing we had before, therefore we show that this "new" inner product is invariant under said transformations. 
So in the end both are valid definitions for inner products but the latter is very useful for our purposes in physics. 
A: While the concept itself makes sense, I agree that within that notation the transpose is very unclear.
Note that $\langle a', b'\rangle$ is in fact equal to $(a')^Tb'$ since we need to multiply the vectors and thus we want the first as a row and the second as a column vector. Substituting in our transform $A$, we see that $\langle a', b' \rangle=\left(Aa\right)^TAb = aA^TAb$ and thus the dot product is only independent of coordinate system if $A^TA = I$ which is the conclusion in the booklet. If you rewrite $aA^TAb$ in their index notation you can see where the confusion comes from.
