$\phi(x)|0\rangle$ is not the state of a particle (I stress that $\phi(x)|0\rangle$ is a one-particle state) with position $x$ (when the temporal component of $x$ is zero in particular).
The situation is different form the momentum representation. Indeed, $a_p^\dagger|0\rangle$ is a momentum-defined one-particle state.
The position representation of the particles of QFT is a quite delicate issue. It is still unclear and actually there are a number of no-go theorems about its existence either in terms of projection valued measures or POVMs.
An apparent statndard definition of the position representation is the famous Newton-Wigner one, however it is plagued by a number of issues concerning locality.
A modern treatise on the issues about the position representation and the various no-go theorems, in relativistic QM can be found here and more recently here.
Probably the most powerfull no-go result is the following one.
I should premit some facts. The position representation is defined by a set of operators $P_{E}$ labeled
by sets $E \subset \mathbb{R}^3$ in the 3D rest space of a Minkowski reference space. These operators may be orthogonal projectors as in the case of a spectral measure (a PVM):
$$\vec{X} = \int_{\mathbb{R}^3} \vec{\lambda} dP(\vec{\lambda})$$
is the triple of position operators.
For instance the Newton-Wigner position operator has this form.
A weaker formulation is the one where the $P_E$ simply define a POVM (Positive-operator valued measure) and one is dealing with the modern formulation of observables.
In both cases $\langle \psi| P_E\psi \rangle$ is the probability to find the particle in $E$ at time $t=0$.
In the case of POVMS $P_E$ is simply a positive operator bounded by $I$ instead of an orthogonal projector as in the spectral decomposition where one can also take advantage of some quantum logic formulation.
One of the no-go theorems proved in 2 has the following form.
THEOREM There is no POVM (or PVM) $P_E$, labeled by sets in the 3D space of a Minkowski mreference frame such that.
1) It is covariant under the action of spacetime translations: $$U^{-1}_{t,a} P_E U_{t,a} = P^{(t)}_{E-a}$$ where $P^{(t)}$ denotes the analogous operator defined at time $t$ (the Heisemberg evolution of the initial one);
2) the generator $H$ of the time translations $U(t,0)= e^{itH}$ is positive;
3) the operators $P_E$ satisfy locality if $(t,E)$ and $(t',E')$ are spacelike separated then $$[P^{(t)}_E, P^{(t')}_{E'}]=0$$
The last requirement can be refined or weakened and it corresponds to the requirement that we cannot transmit superluminal information with the outcomes of these detectors (a version of the no-signaling requirment)
For instance the Newton-Wigner operator (obtained by intgrating its PVM) violates (3) and thus cannot be considered a physical observable.
The theorem above (I stated in a rough way actually, for a precise statement see the reference I posted) is a refinement of a number of previous results due to Hegerfeldt, Borchers, Malement, Busch in particular.