How does one compute the state of a quantum system following imperfect measurement? Suppose I have a quantum system $S$ ("system") with Hamiltonian $H_S$ and initial density matrix $\rho_S(0)$.
I allow $S$ to interact with another system $P$ ("probe"), which has Hamiltonian $H_P$ and initial state $\rho_P(0)$, via an interaction Hamiltonian $H_I$.
Then I measure $P$ in the basis of operator $\hat{Q}$.
Suppose my classical readout device is imperfect: if $P$ is in state $\lvert q \rangle$ which is an eigenstate of $\hat{Q}$ with eigenvalue $q$, then my readout device spits out numbers $q_\text{readout}$ according to a statistical distribution which depends on $q$.
For example, we might have a case where the readout value is Gaussian distributed about $q$, i.e.
$$P(q_\text{readout} | q) = \frac{1}{\sqrt{2\pi \sigma^2}} \exp \left[ - \frac{(q_\text{readout} - q)^2}{2 \sigma^2} \right] \, .$$
Given the Hamiltonians $H_S$, $H_P$, $H_I$, the initial states $\rho_S$ and $\rho_P$, the function $P(q_\text{readout}|q)$, and a realized measured value $q_\text{readout}$, what concepts/approach does one use to find out the state of the combined system $S \otimes P$ after the measurement?
How does the result change if the measured value $q_\text{readout}$ is ignored?
An allowed simplification would be to take the state of the combined system $\rho_{S P}$ after the interaction step as a known quantity.
In other words, we're not so much interested in computing the evolution of $S \otimes P$ under the interaction $H_I$.
However, I think that whether or not $H_I$ commutes with $H_S$ winds up being important.
Notes


*

*While the example probability distribution (i.e. the Gaussian) is continuous, the spectrum of $\hat{Q}$, and/or the distribution $P(q_\text{readout}|q)$ may be discrete.
I suppose it's even possible to have one continuous and the other one discrete!


Resources


*

*A Straightforward Introduction to Continuous Quantum Measurement by Jacobs and Steck.

 A: Assuming the system and probe are initially uncorrelated, the initial density matrix is
$$ \rho(0) = \rho_S(0) \otimes \rho_P(0). $$
After interaction for a time $t$, the system and probe are entangled
$$ \rho(t) = e^{-i H_{\text{full}}t}\rho(0)e^{iH_{\text{full}}t} \equiv \rho_{SP}.$$
Then the observable $\hat Q$ is measured, which can be written
$$ \hat Q = \sum_q q\hat\Pi_q,\qquad\text{with}\quad\hat\Pi_q=\sum_j|{q,j}\rangle\langle{q,j}|,$$
i.e., $\hat Q$ has a discrete spectrum (continuous involves replacing the sum with an integral), where the eigenvalue $q$ has an eigenspace spanned by $|q,j\rangle$ (it is degenerate if there is more than on $j$, which is the typical situation).
In a projective measurement of $\hat Q$, you obtain the eigenvalue $q$ with a probability $P(q)=\text{Tr}[\rho(t)\hat Q]$.
The state of your system is then
$$\rho_q=\frac{\hat\Pi_q\rho(t)\hat\Pi_q}{P(q)}. $$
If you have measured $q_r$, but your measurement apparatus does not transfer the information correctly (what Norbert Schuch called "classical scrambling" in his comment), then your state is
$$ \rho_{q_r} = \int dq \, P(q|q_r)\,\rho_q. $$
(or a sum if your probability distribution is discrete.)
As Noiralef commented, $P(q|q_r)$ has to be calculated from $P(q_r|q)$ given above using Bayes' theorem. Your initial guess could be a uniform distribution for $P(q)$, though this does not necessarily imply a uniform distribution for $P(q_r)$.
If you ignore your measurement record entirely,
the state of your system is 
$$\rho_{\text{ignore}} = \int dq\, P(q)\rho_q = \int dq\,\hat\Pi_q\rho(t)\hat\Pi_q.$$
This is the unconditional state.
In this case (and in the former) all coherences between different $q$ subsectors are gone (if you write the density matrix in a basis where different blocks correspond to different $q$, then all off-diagonal blocks are zero).
Finally, you could consider the case in which the measurement is weak. This is the case in which we really should know more about the measurement. Assuming everything is continuous, you can write down a family of Kraus operators
$$\hat{\Upsilon}_{q_r}=P(q_{\text{r}}|\hat Q), $$
where you just replaced the number $q$ with the operator $\hat Q$ in your expression above. In this case the state of your system post-measurement is
$$ \rho_{q_r} = \frac{\hat\Upsilon_{q_r}\rho(t)\hat\Upsilon_{q_r}^\dagger}{\text{Tr}[\rho(t)\hat\Upsilon_{q_r}^\dagger\Upsilon_{q_r}]}. $$
When you now look at the unconditional state by integrating over $q_r$, you'll find it's not block diagonal!
Edit: Thanks to Noiralef for pointing out typos and Bayes' theorem.
