Quantum non-unitary transformation? Let us say that I apply a non-unitary transformation $\def\ket#1{| #1\rangle} \def\braket#1#2{\langle #1|#2\rangle} \hat A$ to the ket's:
$$\ket{\psi} \rightarrow \hat A \ket{\psi}$$
$$\ket{\phi}\rightarrow \hat A \ket{\phi}$$
Clearly in this case the probability:
$$P=|\braket{\phi}{\psi}|^2$$
Will change. What physically is going on here? i.e. why for unitary operators we can perform such a transformation but for non-unitary operators we can't?
 A: I see two sides to your question, in addition to what was already pointed out in comments:


*

*For a transformation to preserve the scalar product for any pair of kets $|\psi\rangle$, $|\phi\rangle$ in the Hilbert space, it is sufficient that it be an orthogonal transformation: Say $U$ is such a transformation. If $\lbrace |\psi_n\rangle \rbrace_n$ is an orthonormal basis set, then $U$ must be such that
$$
\langle \psi_m | U^\dagger U | \psi_n \rangle = \langle \psi_m | \psi_n \rangle \equiv \langle \psi_m | I | \psi_n \rangle = \delta_{mn},\;\;\;\forall\;m,n
$$
which means
$$
U^\dagger U = I
$$
and so $U$ is orthogonal. Note that unitary transformations also satisfy $U U^\dagger = I$.

*When you apply a non-orthogonal or non-unitary transformation $V$, the scalar product of the transformed kets amounts to 
$$
\langle \phi | V^\dagger V | \psi \rangle \neq \langle \phi | \psi \rangle
$$
But note that this is actually a matrix element for the hermitian operator, and so potentially the observable, $V^\dagger V$. One ubiquitous example: the annihilation operator $\hat a$ for a many-particle system. It is not hermitian, does not preserve the scalar product, but the scalar product of the transformed kets gives a matrix element of the occupation number, which is not the corresponding amplitude:
$$
\langle \phi | {\hat a}^\dagger {\hat a} | \psi \rangle \neq \langle \phi | \psi \rangle
$$
A: A nonunitary operator can be thought of as the sum of unitary and antiunitary operators. So let me let $T = aU + bA$, a, b = normalization constants, and we evaluate 
$$
\langle T\phi|T\psi\rangle = \langle\phi|T^\dagger T|\psi\rangle.
$$ 
The product is $T^\dagger T = U^\dagger U + U^\dagger A + A^\dagger U + A^\dagger A$. The unitary operator is easy $U^\dagger U = 1$. For $A^\dagger A$ we have
$$
\langle\phi|A^\dagger A|\psi\rangle = \langle\psi|\phi\rangle.
$$
For $A^\dagger U$ we have 
$$
\langle\phi|A^\dagger U|\psi\rangle = \overline{\langle\phi|AU|\psi\rangle}
$$
and similarly
$$
\langle\phi|U^\dagger A|\psi\rangle = \overline{\langle\psi |UA^\dagger|\phi\rangle} = \langle\psi |AU|\phi\rangle
$$
The probability will then be a rather long summ
$$
P = \langle T\phi|T\psi\rangle = |a|^2|\langle\psi|\phi\rangle|^2  + b^2|\langle\phi|\psi\rangle|^2 = (|a|^2 + |b|^2)|\langle\psi|\phi\rangle|^2
$$
where $|a|^2 + |b|^2 = 1$.
