I've read that if a state is a product state, the reduced density matrices are pure and if the state is entangled, the reduced density matrices are both mixed.
What would it mean if you had a system of two particles, and when you calculated the reduced density matrices, one was pure and one was mixed?
The statement you have read is regarding states that are pure. Then, for any two-partite pure state $$|\Psi\rangle=\sum_i \psi_i |a_i\rangle\otimes |b_i\rangle,$$ the reduced density matrices $$\rho_A=\sum_i |\psi_i|^2|a_i\rangle\langle a_i|$$ and $$\rho_B=\sum_i |\psi_i|^2|b_i\rangle\langle b_i|$$ are both either pure with one coefficient $|\psi_i|=1$ and the rest vanishing, or both mixed with all coefficients $|\psi_i|<1$. It is not possible to have one reduced density matrix be pure and the other mixed.
Now, if the two-partite state is not pure to begin with, the story is different. Then it is indeed possible to have one reduced density matrix be pure and the other mixed, but the cost is that we cannot actually use the mixedness of the reduced density matrix to determine whether the initial state was entangled. For example, the initial state $$\rho=\rho_A\otimes |b\rangle\langle b|$$ is separable and can have a mixed state $\rho_A$ as the reduced density matrix for the first party and a pure state $|b\rangle$ for the second party's reduced state. In fact, as pointed out by user Tobias Fünke, this is the only possible way for a state to have a pure reduced density matrix for the second party, and thus if either party has a pure reduced density matrix then the overall state is not entangled.
