Now, I've seen people try to "prove" it but I think the proof is wrong.
Well, that depends on what you mean by "prove". You can prove things based on some set of axioms. In general, the relation
$$
T(x)\phi(y)T^\dagger(x)=\phi(y-x) \quad\text{where}\quad T(x)=\exp[-iPx]\tag{1}
$$
is one of the axioms themselves$^1$, so you cannot really prove it. You can take$^2$
$$
-i\partial_\mu\phi=[P_\mu,\phi] \tag{2}
$$
as an axiom instead, from which you can prove $(1)$. But you have to postulate something. The equivalence $(1)\Leftrightarrow(2)$ is easy to prove, so take whichever axiom you like the most.
If I understand it correctly, $P$ acts on states but not labels of the operators, and every point in $\varphi$ acts on the whole vacuum.
I'm not sure what you mean by this. $P$ is an operator, so it indeed acts on states. But you can compose ("multiply") operators to get different operators.
Is the whole field solely determined by one point of the field?
More or less. You have to specify the field at one point (say, the origin $x=0$) and the equations of motion for the field. In QFT we take $(2)$ as the eom (known as Heisenberg equations of motion).
Any field that satisfies $(2)$ can be written as $\phi(x)=T^\dagger(x)\phi(0)T(x)$, where $T$ is the translation operator. Therefore, given $\phi(0)$ we know "all the dynamics", that is, $\phi(x)$.
The same thing happens in classical mechanics: you impose some differential equations, after which any trajectory is determined when you specify the initial conditions. In this sense, any path $\{q(t),p(t)\}$ is solely determined by $\{q(0),p(0)\}$, but this doesn't mean we don't have degrees of freedom!
$^1$: See https://ncatlab.org/nlab/show/Wightman+axioms, Axiom 5 (with $\Lambda=1$).
$^2$: See http://portal.kph.uni-mainz.de/T//members/wittig/talks_lecture/ral.pdf page 11, eq. (2.27)