Normalization of photon wavefunction and beam splitters I am  having trouble with the normalization of photon wave functions, when passed through a beam splitter.
Let me define the single photon state as
$$|1\rangle = \int \text{d}\omega \phi(\omega) a^\dagger(\omega)|0\rangle$$
and the two photon state as
$$|2\rangle = \int \text{d}\omega_1\text{d}\omega_2\phi(\omega_1)\phi(\omega_2) a^\dagger(\omega_1)a^\dagger(\omega_2)|0\rangle$$
These states are normalized if $\int \text{d}\omega |\phi(\omega)|^2 = 1$.
I have used the operator $a^\dagger$ to denote the creation operator in the first of my two modes. The operator for the second mode is $b^\dagger$. Let me insert the two photon state into the first port of a beam splitter which takes $a \rightarrow \frac{a-b}{\sqrt{2}}$. My output state is then
$$|\psi\rangle = \frac{1}{2}\int \text{d}\omega_1\text{d}\omega_2 \phi(\omega_1)\phi(\omega_2) [a^\dagger(\omega_1)-b^\dagger(\omega_1)][a^\dagger(\omega_2)-b^\dagger(\omega_2)]|0,0\rangle$$
I can compute the amplitude of the various output events. The amplitude for both $|2,0\rangle$ and $|0,2\rangle$ is $\frac{1}{2}$. The amplitude for $|1,1\rangle$ comes out to be $-1$. The state does not seem to be normalized. Why not?
Further details
I compute amplitudes in the following way. 
$$\langle 2,0|\psi\rangle = \frac{1}{2}\int \text{d}\omega_1^\prime\text{d}\omega_2^\prime\phi^*(\omega_1^\prime)\phi^*(\omega_2^\prime) \int \text{d}\omega_1\text{d}\omega_2 \phi(\omega_1)\phi(\omega_2)\langle 0,0 | a(\omega_1^\prime)a(\omega_2^\prime)  [a^\dagger(\omega_1)-b^\dagger(\omega_1)][a^\dagger(\omega_2)-b^\dagger(\omega_2)]|0,0\rangle$$
Only the term corresponding to two photons in the first mode survives. 
$$\langle 0,0 |a(\omega_1^\prime)a(\omega_2^\prime) a^\dagger(\omega_1)a^\dagger(\omega_2)|0,0\rangle = \delta(\omega_1^\prime-\omega_1)\delta(\omega_2^\prime-\omega_2)$$
Therefore the above integrals collapse to
$$\frac{1}{2}\int \text{d}\omega_1\text{d}\omega_2 |\phi(\omega_1)|^2|\phi(\omega_2)|^2 = \frac{1}{2} $$
Similarly
$$\langle 1,1|\psi\rangle = \frac{1}{2}\int \text{d}\omega_1^\prime\text{d}\omega_2^\prime\phi^*(\omega_1^\prime)\phi^*(\omega_2^\prime) \int \text{d}\omega_1\text{d}\omega_2 \phi(\omega_1)\phi(\omega_2)\langle 0,0 | a(\omega_1^\prime)b(\omega_2^\prime)  [a^\dagger(\omega_1)-b^\dagger(\omega_1)][a^\dagger(\omega_2)-b^\dagger(\omega_2)]|0,0\rangle$$
This time two different terms survive, one corresponding to the $\omega_1$ photon jumping to the second channel, the other corresponding to the $\omega_2$ photon jumping to the second channel.
$$\langle 0,0 |a(\omega_1^\prime)b(\omega_2^\prime) a^\dagger(\omega_1)b^\dagger(\omega_2)|0,0\rangle = \delta(\omega_1^\prime-\omega_1)\delta(\omega_2^\prime-\omega_2)$$
$$\langle 0,0 |a(\omega_1^\prime)b(\omega_2^\prime) b^\dagger(\omega_1)a^\dagger(\omega_2)|0,0\rangle = \delta(\omega_1^\prime-\omega_2)\delta(\omega_2^\prime-\omega_1)$$
Therefore, the integrals collapse and we get
$$2\times \frac{-1}{2}\int \text{d}\omega_1\text{d}\omega_2 |\phi(\omega_1)|^2|\phi(\omega_2)|^2 = -1$$
I have added the amplitudes of these two terms rather than the probabilities because the two photons have an identical spectrum. Is this incorrect for some reason?
Further edited in details
I have used the following unitary for the transformation of the field operators.
$$\frac{1}{\sqrt{2}}\begin{bmatrix} 1 & -1 \\ 1 & \phantom{-}1 \end{bmatrix}$$ 
This transformation can be found in any quantum optics book such as Scully and Zubairy. 
The normalization of the $|2\rangle$ state follows
$$\langle 2|2\rangle = \int \text{d}\omega_1^\prime\text{d}\omega_2^\prime\phi^*(\omega_1^\prime)\phi^*(\omega_2^\prime) \int \text{d}\omega_1\text{d}\omega_2\phi(\omega_1)\phi(\omega_2) \langle 0| a(\omega_1^\prime)a(\omega_2^\prime)  a^\dagger(\omega_1)a^\dagger(\omega_2)|0\rangle$$
Now note that there are four operators, each corresponding to a different frequency. The crucial commutation relation between them is $[a(\omega_1),a^\dagger(\omega_2)] = \delta(\omega_1-\omega_2)$. The braket is non-zero if I equate $\omega_1^\prime$ and $\omega_1$ and similarly $\omega_2^\prime$ and $\omega_2$. The braket reduces to
$$ \langle 0| a(\omega_1) a^\dagger(\omega_1) a(\omega_2)  a^\dagger(\omega_2)|0\rangle \delta(\omega_1^\prime-\omega_1) \delta(\omega_2^\prime-\omega_2) = \delta(\omega_1^\prime-\omega_1) \delta(\omega_2^\prime-\omega_2)$$
Then the integral above collapses to
$$\int \text{d}\omega_1\text{d}\omega_2|\phi(\omega_1)|^2|\phi(\omega_2)|^2 = 1$$
 A: The two-photon input state as you have defined 
$$|\tilde{2}\rangle = \int \text{d}\omega_1\text{d}\omega_2\phi(\omega_1)\phi(\omega_2) a^\dagger(\omega_1)a^\dagger(\omega_2)|0\rangle$$
is not-normalized. Consider
$$\langle \tilde{2}|\tilde{2}\rangle = \int\int 
\text{d}\omega_1 \text{d}\omega_2
\text{d}\omega_1^\prime \text{d}\omega_2^\prime
\phi(\omega_1)\phi(\omega_2)
\phi(\omega_1^\prime)\phi(\omega_2^\prime)
\langle 0| a(\omega_1^\prime)a(\omega_2^\prime)
a^\dagger(\omega_1)a^\dagger(\omega_2)|0\rangle\\ 
\phantom{\langle 2|2\rangle}= \int\int 
\text{d}\omega_1 \text{d}\omega_2
\text{d}\omega_1^\prime \text{d}\omega_2^\prime
\phi(\omega_1)\phi(\omega_2)
\phi(\omega_1^\prime)\phi(\omega_2^\prime)\phantom{\langle 0| a(\omega_1^\prime)a(\omega_2^\prime)
a^\dagger(\omega_1)a^\dagger(\omega_2)|0\rangle}
\\\times\left(\delta(\omega_1^\prime-\omega_1)\delta(\omega_2^\prime-\omega_2)+
\delta(\omega_2^\prime-\omega_1) \delta(\omega_1^\prime-\omega_2)\right)\phantom{\langle 0| a(\omega_1^\prime)|0\rangle}\\
\phantom{\langle 2|2\rangle} = 2, 
\phantom{\int+\text{d}\omega_1 \text{d}\omega_2
\text{d}\omega_1^\prime \text{d}\omega_2^\prime
\phi(\omega_1)\phi(\omega_2)
\phi(\omega_1^\prime)\phi(\omega_2^\prime)
\langle 0| a(\omega_1^\prime)a(\omega_2^\prime)
a^\dagger(\omega_1)a^\dagger(\omega_2)|0\rangle}$$ 
where I have repeatedly used the Bosonic commutation relations to simplify (see calculations appended) the expression above, but the same can be accomplished using Wick's theorem .
The correctly normalized state is thus
$$
|2\rangle = \frac{1}{\sqrt{2}}\int \text{d}\omega_1\text{d}\omega_2\phi(\omega_1)\phi(\omega_2) a^\dagger(\omega_1)a^\dagger(\omega_2)|0\rangle.
$$
Using the correctly normalized state, we obtain the three amplitudes
$$
\langle 20|U_{\text{BS}}|20\rangle = \frac{1}{2} \\
\langle 11|U_{\text{BS}}|20\rangle = -\frac{1}{\sqrt{2}} \\
\langle 02|U_{\text{BS}}|20\rangle = \frac{1}{2},
$$
the sum of whose squares is $1$. The beamsplitter is safe again for quantum mechanics.

Appendix: Calculations for $\langle \tilde{2}|\tilde{2}\rangle = 2$
$$\langle \tilde{2}|\tilde{2}\rangle = \int\int 
\text{d}\omega_1 \text{d}\omega_2
\text{d}\omega_1^\prime \text{d}\omega_2^\prime
\phi(\omega_1)\phi(\omega_2)
\phi(\omega_1^\prime)\phi(\omega_2^\prime)
\langle 0| a(\omega_1^\prime)a(\omega_2^\prime)
a^\dagger(\omega_1)a^\dagger(\omega_2)|0\rangle\\ 
=  \int\int 
\text{d}\omega_1 \text{d}\omega_2
\text{d}\omega_1^\prime \text{d}\omega_2^\prime
\phi(\omega_1)\phi(\omega_2)
\phi(\omega_1^\prime)\phi(\omega_2^\prime)
\\\times
\left(\langle 0| a(\omega_1^\prime)
a^\dagger(\omega_1)a(\omega_2^\prime)a^\dagger(\omega_2)|0\rangle +\\ 
\langle 0| a(\omega_1^\prime)
\delta(\omega_2^\prime-\omega_1)a^\dagger(\omega_2)|0\rangle
\right), \\
=  \int\int 
\text{d}\omega_1 \text{d}\omega_2
\text{d}\omega_1^\prime \text{d}\omega_2^\prime
\phi(\omega_1)\phi(\omega_2)
\phi(\omega_1^\prime)\phi(\omega_2^\prime)
\\\times
\left(\langle 0| (a^\dagger(\omega_1)a(\omega_1^\prime)+ \delta(\omega_1^\prime-\omega_1)
(a^\dagger(\omega_2)a(\omega_2^\prime)+\delta(\omega_2^\prime-\omega_2))|0\rangle +\\ 
\langle 0|
\delta(\omega_2^\prime-\omega_1) (a^\dagger(\omega_2)a(\omega_1^\prime)+\delta(\omega_1^\prime-\omega_2)|0\rangle
\right)\\=  \int\int 
\text{d}\omega_1 \text{d}\omega_2
\text{d}\omega_1^\prime \text{d}\omega_2^\prime
\phi(\omega_1)\phi(\omega_2)
\phi(\omega_1^\prime)\phi(\omega_2^\prime)
\\\times\left(\delta(\omega_1^\prime-\omega_1)\delta(\omega_2^\prime-\omega_2)+
\delta(\omega_2^\prime-\omega_1) \delta(\omega_1^\prime-\omega_2)\right)\\
= 2.
$$
The correctly normalized state is thus:
$$
|2\rangle = \frac{1}{\sqrt{2}}|\tilde{2}\rangle = \frac{1}{\sqrt{2}}\int \text{d}\omega_1\text{d}\omega_2\phi(\omega_1)\phi(\omega_2) a^\dagger(\omega_1)a^\dagger(\omega_2)|0\rangle.
$$
NB: The correctly normalized $n$-photon state is
$$
|n\rangle = \frac{1}{\sqrt{n!}}\int 
\text{d}\omega_1\text{d}\omega_2\ldots\omega_n
\phi(\omega_1)\phi(\omega_2) \ldots \phi(\omega_n) 
a^\dagger(\omega_1)a^\dagger(\omega_2)\ldots a^\dagger(\omega_n)|0\rangle.
$$
A: I am not sure what you mean by "normalized" here. Usually it is a state with norm 1, i.e.
$$
\langle\psi|\psi\rangle = 1
$$
A multi-particle state as you defined is not normalized:
$$
\langle2|2\rangle = \langle0|aaa^\dagger a^\dagger|0\rangle = \langle0|a(a^\dagger a+1)a^\dagger|0\rangle = \langle0|(aa^\dagger + aa^\dagger)|0\rangle = 2\langle1|1\rangle
$$
To normalize, one divides a state by the square root of number of particles.
Now, lets have a look at the transformation you called $a\rightarrow\frac{a-b}{\sqrt{2}}$. This transformation may be written out as
$$
\hat{A} = \int d\omega \frac{a^\dagger(\omega)-b^\dagger(\omega)}{\sqrt{2}}a(\omega)
$$
i.e. it annihilates a particle in state $a$ and creates a new particle in the antisymmetric superposition of states $a$ and $b$. This operator is not unitary:
$$
\hat{A}^\dagger\hat{A} \neq Id
$$
Non-unitary operators are not norm-conserving, so if you let them act on a state, the result will have a different norm.
