I believe central to this question is: what really are the Poincaré generators $P_\mu$ and $J_{\mu\nu}$? The relevance is that if we wish to even think of generalizing the statement that the vacuum is invariant under $P_\mu$ and $J_{\mu\nu}$ to arbitrary diffeomorphisms, we would need to understand what is the analogue of $P_\mu$ and $J_{\mu\nu}$ for diffeomorphsisms.
The key here is to understand that $P_\mu$ and $J_{\mu\nu}$ are the charges associated to a particular symmetry. In the Hamiltonian formalism of Classical Mechanics, the charge $Q$ associated to a symmetry transformation $\delta_Q$ is such that $$\{Q,f\}=\delta_Q f\tag{1}.$$
This statement gets promoted, upon quantization, to the statement that there exists an operator $\hat{Q}$ in the Hilbert space of the theory, that generates the symmetry action on operators through the commutator.
A more abstract way of rewriting (1) is to recall that a phase space in Classical Mechanics is a symplectic manifold $(\Gamma,\Omega)$ where $\Omega$ is a two-form in $\Gamma$ which is closed, $d\Omega=0$, and non-degenerate, $\Omega(X,Y)=0$ for all $Y$ implies $X=0$. Such two-form we call a sympletic form. It is related to the Poisson bracket as follows: introduce coordinates $\zeta^I$ on $\Gamma$, then if $\Omega_{IJ}$ are the components of $\Omega$ in the coordinate basis, and if $\Omega^{IJ}$ is the inverse matrix, $$\{f,g\}=\Omega^{IJ}\partial_I f\partial_J g\tag{2}.$$
A symmetry transformation is now a diffeomorphism in phase space. The relation between symmetry and charge (1) can be written then as $$\Omega(X_Q,Y)=-dQ(Y)\tag{3},$$
where $X_Q$ is vector field on $\Gamma$ generating the symmetry transformation. In particular $\delta_Qf = X_Q(f)$ for some observable $f\in C^\infty(\Gamma)$.
Now observing that spacetime diffeomorphisms, a.k.a. general coordinate transformations, are local transformations, we can precisely formulate the question as given a local symmetry of a theory in some arbitrary spacetime $(M,g)$, what is its associated charge $Q$ described in the above procedure?
It turns out this question gets precisely answered by the so-called Covariant Phase Space formalism that is studied by people in the field of asymptotic symmetries for a long time. This answer of mine gives an overview with references and also this one.
The moral of the story in these posts is that starting from a Lagrangian density (that could as well be the Einstein-Hilbert one) it is possible to construct an object called pre-symplectic density $\pmb{\omega}[\phi,\delta_1\phi,\delta_2\phi]$ such that the symplectic form reads $$\Omega[\phi,\delta_1\phi,\delta_2\phi]=\int_\Sigma \pmb{\omega}[\phi,\delta_1\phi,\delta_2\phi],\tag{4}$$
where $\Sigma$ is a Cauchy surface, $\phi$ are the fields in the theory and $\delta_i\phi$ are two variations that play the role of the phase space vectors. Then Noether's second theorem comes into play and it allows you to show that if $\delta_1\phi = \delta_\epsilon\phi$ is a symmetry transformation with local parameter $\epsilon(x)$, and $\delta_2\phi=\delta\phi$ is an arbitrary variation, then $\pmb{\omega}[\phi,\delta_\epsilon\phi,\delta\phi]$ is a closed form
$$\pmb{\omega}[\phi,\delta_\epsilon\phi,\delta\phi] = d\mathbf{k}_\epsilon[\phi,\delta\phi]\tag{5}$$
so that the symplectic form reads $$\Omega[\phi,\delta_\epsilon\phi,\delta\phi] = \int_{\partial\Sigma}\mathbf{k}_\epsilon[\phi,\delta\phi].\tag{6}$$
Since $\Sigma$ is a Cauchy surface, $\partial\Sigma$ lies at infinity. In fact, for asymptotically flat spacetimes, $\partial\Sigma$ is close to spatial infinity $i^0$. Now the RHS of (6) can be zero or not. If it is zero, as it happens when one demands that fields vanish at infinity in asymptotically flat spacetimes, then the associated charge $Q_\epsilon =0$ is trivial. In fact, in that case $\delta_\epsilon$ are degenerate vectors of $\Omega$ and it is not a true symplectic form: one must fix the gauge, because these transformations are not true symmetries, but mere redundancies. This is what is happening most of the time and is why demanding that these generators annihilate the vacuum do not make much sense: they are identically zero already because we don't have a true symmetry! These diffeomorphisms are called small diffeomorphisms or trivial ones.
But I must comment that it is possible to get a non-zero result for some diffeomorphisms, and in fact, one that allows you to eventually use (3) to define a non-trivial charge. These diffeorphisms are called large diffeomorphisms or asymptotic symmetries more generally. In asymptotically flat spacetimes they give rise to the extended BMS algebra for example.
In that case we have non-trivial charges $Q_f$ associated to super translations and $Q_Y$ associated to superstations, a subset of which recovers the Poincaré charges $P_\mu$ and $J_{\mu\nu}$. It turns out, however, that one can study what those charges do to the vacuum, and what one finds out is that apart from the Poincaré charges the other ones do not annihilate the vacuum. Instead, they shift the vacuum: these extra symmetries are spontaneously broken. For more details see the references in the threads I linked as well as Andy Strominger's lectures.
I would thus summarize as follows: we can't postulate a state that is annihilated by the generators of diffeomorphisms for two reasons: (1) firstly because small diffeomorphisms are not true symmetries and they don't have non-trivial charges, (2) secondly because for large diffeomorphisms in asymptotically flat spacetimes we can check that the state annihilated by the Poincaré charges simply cannot be annihilated by the remaining extended BMS charges. Point (2) then provides a counter-example in which that proposal just does not work.
