Is it true that $\mathrm{tr}(\rho^n)$ is monotonically decreasing as a function of $n$? In the case of mixed states density matrix made out of orthogonal vectors this is clear since we have $$\sum_i p_i^n,$$ but what if we use non-orthogonal vectors or a continuous mixture?
 A: On an infinite dimensional separable Hilbert space, the result still holds. It is a standard result in functional analysis that if $A$ is trace-class and $B$ is bounded, then $AB$ is trace class, with $\operatorname{tr}(|AB|) \le \operatorname{tr}(|A|) \|B \|$, where $\|B \|$ is the operator norm. A density operator has unit trace and is positive semi-definite, so its operator norm is $\le 1$. Applying this with $A = \rho^{n}$, $B = \rho$, we see that $\operatorname{tr} \rho^{n+1} \le \operatorname{tr} \rho^{n}$, with equality possible only when $\rho$ has operator norm $1$ (in which case the spectrum of $\rho$ is just $(1,0,0,0, \ldots)$).
If you want to prove it is still monotonic when $n$ is taken as a continuous variable, you can just set $B = \rho^\epsilon$ instead, where $\epsilon > 0$. Then $B$ is still bounded (but not necessarily trace-class) with operator norm $\le 1$ (in fact its operator norm is $\|B\|^\epsilon$), and the rest of the argument still works.
A: If you're working in finite dimensions, then neither of your concerns,

what if we use non orthogonal vectors or a continuous mixture?

is an actual problem: whatever sum you used to get $\rho$, it is still a positive-semidefinite hermitian operator in a finite-dimensional Hilbert space, and as such it admits an eigenvalue decomposition
$$
\rho = \sum_{i=1}^N p_i |\psi_i\rangle \langle \psi_i|
$$
with orthonormal $|\psi_i\rangle$, which can then feed back into the initial orthogonal case.
The only real change is if you have a continuous mixture over finite dimensions, say, something of the form
$$
\rho = \int p(x)|x\rangle\langle x|\mathrm dx,
$$
in which case the monotonicity follows from the triangle inequality on $L_1(\mathbb R)$, as
\begin{align}
\mathrm{Tr}(\rho^n)
& = 
\int p(x)^n \,\mathrm dx
\leq
\int p(x)^{n-1} \,\mathrm dx
\int p(x) \,\mathrm dx
=
\int p(x)^{n-1} \,\mathrm dx
,
\end{align}
and then via induction.
