Lets say I have a density matrix $\rho$ encoded into a physical system. Lets say I can have access to as many copy of $\rho$ as I want. I perform hmodyne detection on the copies in phase and out of phase and based on the measurement result always lies in a circle nemely $$P^2+X^2\leq r$$ where $P$ and $X$ are real values and correspond to measurement results out of phase and in phase respectively. Based on this measurment results what can I say about the dim of hilbert space corresponding to $\rho$? Is it possible to bound the dimention?
Basically, the average value of $P^2+X^2$ is the average value of the energy, hence of $4n+2$ (with the suitable normalization). Therefore, what you describe is known ans an energy test in the literature. Basically, if your state is essentially restricted to a superposition of the Fock states of $O$ to $d-1$ photons, we cans say that it is restricted into a subspace of dimension $d$.
The details of the energy test vary, and there is still ongoing research work to find the most efficient one. One recent example is in PRL 118 200501 / arXiv:1701.03393, by Anthony Leverrier which does not assume a product state and use heterodyne detection. Basically, if your state has a non-negligible support of Fock states of $d$ photons or more, it will have a non-negligible probability to have a value of $P^2+X^2$ to be higher than $r$ and to be detected by the energy test.
One way to compute this probability would be to use the explicit expression of the Fock states using Hilbert polynomials, but I don’t know a nice and simple expression from the top of my head.