Im learning about the variation method for solving quantum mechanics problems. The principle is that eigenfunctions minimize the expectation value of the Hamiltonian on that state (provided $\Psi$ is normalized):
with equality meaning $\Psi=\Psi_1$. Next eigenstates are found sequentially by noting that $E_n$ will still be a local minimum and $\Psi_n$ will be orthogonal to all previously found eigenstates.
The professor did the derivation for $E_1$ in class. His argument was: $\Psi_n$ will form a basis for any trial solution $\Psi$. Thus using fouriers trick we can find an expansion for $\Psi$ in the basis $\Psi_n$ with coefficients $b_n$. Then the energy expectation value for the trial is
Clearly, the smallest $E_n$ is $E_1$, so the smallest $|E|$ is $E_1$, and the coefficients are $|b_1|=1$ and $|b_n|=0$ for all other $n$.
But what happens if I take a trial wave function that cannot be expressed in this basis? For instance, say I have a square well from $0$ to $L$ and I have a text function which is normalized but nonzero at some location with $x>L$. Obviously this is a stupid trial function and it is not a possible solution. I will not be able to find $|E|$ to check if the functional is minimized. But what part of the math above will break down? Where does the variational principle say, for instance, that in addition to being normalizable, the test function must be zero in quantum mechanically disallowed regions?