What are constraints on a "purity" operator in quantum mechanics? Consider the normalized state, written in some orthonormal basis as:
$$\psi = A |0\rangle + B |1\rangle$$
Let's define a "purity operator" for a basis as any operator whose expectation value gives 1 for a pure state in this basis, and 0 for the most mixed state in this basis. Inbetween states should give between 0 and 1, although the specific value doesn't matter.
One possible example (please note my question is on the general case though, this specific example is just to aid discussion), is 
$$\langle \mathcal{O} \rangle = 1 - 4 \frac{|A||B|}{|A|+|B|}$$
What mathematically prevents such measurements in quantum mechanics?
 A: From the example you've given, it's clear that you're using the wrong terms to describe what you want. Purity and mixedness apply to density operators and not to state vectors -- if your system is described by a state vector $|\psi\rangle$, it is already pure.
What you seem to want to know is whether there is an operator that, in a particular basis, has an expectation value that lets you know to what degree the state is in a superposition of basis states. I think the uncertainty operator might work for you here. If $S$ is the operator whose eigenstates are $|0\rangle$ and $|1\rangle$, then the operator $(S-\langle S\rangle)^2$ has zero expectation value for the basis states and a non-zero value for all superpositions. You could scale this in some way to give you a value between 0 and 1.
A: The question (v1) abuses language slightly no matter how one understands it. Here I sketch three interpretations.
1) Interpretation in terms of a density matrix $\rho$:
Assume the operator ${\cal O}$ is diagonalizable. Since the eigenstates are pure states, the eigenvalues must be $1$. In other words, ${\cal O}$ is the identity operator.
Then $\langle {\cal O} \rangle = \mathrm{tr}{\cal O}\rho=\mathrm{tr}\rho=1$ for any mixed or pure density matrix $\rho$.
2) Interpretation in terms of an operator ${\cal O}$ that satisfies
$$\langle 0 |{\cal O}| 0\rangle =1,$$
$$\langle 1 |{\cal O}| 1\rangle =1,$$
$$\langle \psi |{\cal O}| \psi\rangle =0 \qquad \mathrm{for~~all} \qquad |A|=|B|=\frac{1}{\sqrt{2}}.$$
Here 
$$|\psi\rangle = A| 0\rangle+B| 1\rangle, \qquad 1=||\psi||^2 = \langle \psi|\psi\rangle =|A|^2+|B|^2 .$$
$|A|=|B|=\frac{1}{\sqrt{2}}$ is not what is traditionally meant by being "the most mixed state". Note that there are not just one of these states, but infinitely many pairs of complex numbers $(A,B)$ that satisfy $|A|=|B|=\frac{1}{\sqrt{2}}$. Even after removing an overall phase, there is a relative phase left.
Assume ${\cal O}$ is a Hermitian operator. Then ${\cal O}$ has a matrix of the form
$${\cal O} = \left[\begin{array}{cc} 1 & c \\ c^* & 1 \end{array}\right]. $$
So
$$0=\langle \psi |{\cal O}| \psi\rangle = |A|^2+|B|^2+A^*cB+B^*c^*A = 1+2\mathrm{Re}(A^*cB).$$
But this is impossible for general complex phases of $A$ and $B$ with $|A|=|B|=\frac{1}{\sqrt{2}}$. 
3) Interpretation in terms of an operator ${\cal O}$ that satisfies
$$\langle 0 |{\cal O}| 0\rangle =1,$$
$$\langle 1 |{\cal O}| 1\rangle =1,$$
$$\langle \psi |{\cal O}| \psi\rangle =0 \qquad \mathrm{for~at~least~one} \qquad |A|=|B|=\frac{1}{\sqrt{2}}.$$
The characteristic polynomial $p_{\cal O}(\lambda)$ for the matrix ${\cal O}$ reads
$$p_{\cal O}(\lambda)=\det({\cal O}-\lambda {\bf 1})= (\lambda-1)^2-|c|^2= (\lambda-1-|c|) (\lambda-1+|c|).$$
So the two eigenvalues of ${\cal O}$ are $\lambda=1\pm|c|$. Since the expectation values for ${\cal O}$ should be between $0$ and $1$, we must demand that $c=0$. In other words, ${\cal O}$ is the identity operator, which leads to the following inconsistency 
$$0=\langle \psi |{\cal O}| \psi\rangle =\langle \psi |\psi\rangle =1.$$
