I'm trying to calculate uncertainty in momentum, and I know that
$$\langle\hat P^2\rangle=\int^{\infty}_{-\infty}\hat P^2|\Psi(x)|^2\,\text dx$$
But I'm confused by what that symbol means. Does it mean I perform the operator on $|\Psi(x)|^2$ twice, or does it mean:
$$\langle\hat P^2\rangle=\int^{\infty}_{-\infty}\langle\Psi(x)|\hat P^2|\Psi(x)\rangle \,\text dx$$ where I just perform the operator on $\Psi$ twice? Or are they equivalent?
You seem a bit confused about how to use Dirac notation, so I'll derive the result from scratch. For any operator $O$, the definition of the expectation value is $$\langle O \rangle = \langle \psi | O | \psi \rangle.$$ In order to write this as an integral, just note that $$1 = \int dx \, | x \rangle \langle x |.$$ By putting in two "factors of $1$", we get $$\langle O \rangle = \int dx \, dx ' \, \langle \psi | x \rangle \langle x | O | x' \rangle \langle x' | \psi \rangle.$$ The definition of the wavefunction is $$\psi(x) = \langle x | \psi \rangle.$$ In particular, it doesn't make sense to write $|\psi(x) \rangle$ in Dirac notation, since $\psi(x)$ is just a number, not a ket. Anyway, using the definition of the wavefunction, we have $$\langle O \rangle = \int dx \, dx ' \, \psi^*(x) \langle x | O | x' \rangle \psi(x').$$ This is how you calculate the expectation value of any operator in the position basis. In your case, $$\langle x | p^2 | x ' \rangle = \left( \frac{\hbar}{i} \right)^2 \delta''(x - x')$$ essentially by the definition of $p$, so $$\langle p^2 \rangle = - \hbar^2 \int dx \, dx' \, \psi^*(x) \delta''(x - x') \psi(x').$$ Now integrate by parts with respect to $x'$ twice, to get $$\langle p^2 \rangle = - \hbar^2 \int dx \, dx' \, \psi^*(x) \delta(x - x') \psi''(x') = - \hbar^2 \int dx \, \psi^*(x) \psi''(x).$$ That's your answer.
tl;dr: Neither of your expressions are right. Instead, differentiate $\psi(x)$ twice.
By definition \begin{align} \langle O\rangle := \int dx \psi^*(x) \hat O\psi(x) \end{align} so in your case \begin{align} \langle P^2\rangle = \int dx\psi^*(x) \left(-i\hbar \frac{d}{dx}\right)\left(-i\hbar \frac{d}{dx}\right)\psi^*(x)=-\hbar^2 \int dx\,\psi^*(x)\psi^{\prime\prime}(x) \end{align}
Your second equation is right (I don't understand why knzhou doesn't like it). It's the quantum mechanical expression for $\left<\hat P^2 \right>$. But your first equation is wrong - because this is quantum mechanics.
If $\hat P$ were just a function, say $f(x)$, you could trivially rewrite $\Psi^*(x) f^2(x) \Psi(x)$ as $f^2(x) \Psi^*(x) \Psi(x)=f^2(x) |\Psi(x)^2|$ and get your first equation, which is the standard statistical expression for $\left<\hat P^2 \right>$ and you've probably got that from a textbook on statistics or statistical mechanics.
And in that context it's fine, but Quantum Mechanics breaks it. Because $\hat P$, being momentum, involves not just multiplication but differentiation and you can't do that trivial rewrite. This is one of those points where previous knowledge and formulae get disturbed and extended by Quantum Mechanics.
