Assume a Hilbert space that is (i) truncated to at most one photon, and (ii) is path-encoded such that $(1,0)^T$ and $(0,1)^T$ represent the photon in two separate optical modes, respectively. Here, these could be the upper $|u\rangle$ and lower $|l\rangle$ arms incident on a beam splitter. If this beam splitter is symmetric, it would be written as \begin{equation} \hat{B} = \frac{1}{\sqrt{2}} \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}. \end{equation}
My problem is with the representation of the vacuum $|0\rangle$. I could write it as $(0,0)^T$ and the effect of the beam splitter would indeed be to leave it unchanged. However this can't be a valid representation of the vacuum state since it's not normalized. I then thought of extending the Hilbert space by including an extra dimension at the beginning of the state vector such that $(a,b,c)^T$ would be equivalent to $a|0\rangle + b|u\rangle + c|l\rangle$.
The question is then, how would the beam splitter look like in this three-dimensional Hilbert space? In order to capture the fact that the beam splitter leaves the vacuum unchanged, I tried \begin{equation} \hat{B} = \frac{1}{\sqrt{2}} \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 1 \\ 0& 1 & -1 \end{pmatrix}, \end{equation} but this doesn't satisfy the unitarity of the beam splitter. What am I missing?
