What determines the Hilbert space of quantum mechanical system? Is there a systematic way to determine the Hilbert space of a given quantum mechanical problem? Given a problem, the first task is to determine the Hilbert space. Is it dictated by the boundary conditions of the problem? I'm particularly interested in free particle problem, the problem of a particle in a box.
 A: Reading the comments here, I am very much reminded of the "physics vs formalism" debate. The idea put forward by some of the comments seems to assume that physics is a process of following a recipe to come up with an answer, very much like the type of questions one would get in a physics course assignment: "given a Hamiltonian that looks like this ... where these quantities are in these Hilbert space ... please show that ..." If every problem that we want to solve in physics just comes down to plugging in the right numbers into our formalism, then it would all just be like solving assignments for a physics course.
Actual physics is not like that. It deals with real world situations where you need to decide what you want to do and how you want to model the situation. Imagine you see a new phenomenon, some radiation from some new material and you want to understand the mechanism. Perhaps the material contains some atoms that you suspect play a role in what you observe. You need to decide how many of the energy levels in that atom would contribute. That may help you to determine the size of your Hilbert space. If the result doesn't agree with what you see, you need to go back and change your model.
So, to answer the question of the OP. There is no recipe that gives a unambiguous method to determine the Hilbert space for actual physics problems. It would have been boring if that were the case. The idea is to use human ingenuity to come up with a model that describes the physical situation and then to modify the model until its predictions agree what your observations.
A: As others have indicated, this question is detailed, and I can't give a full answer, if a satisfactory answer does exist at all.
But as a start, you can say that for a specified measurement in the problem, the space must have a basis with at least one unit vector for each distinct classical value of the measurement, orthogonal to all the others.
That is to say if the problem has a measurement with n possible different classical values, there must be at least n vectors $|v_a\rangle : a \in 1..n$ in the space, with $\langle v_f|v_g\rangle = 0$ for all $f$,$g$.
So in your example there are infinitely many such vectors (the space has infinite dimension).
