First of all: how does one define one operator in a Hilbert space? This is just a mathematics question and the answer is simple: we have a Hilbert space at hand $\mathcal{H}$, then we define a function $A : D(A)\subset \mathcal{H}\to \mathcal{H}$ that is linear, $D(A)$ being its domain.
Then we must define the function. In other words, we must say how $A$ acts on $D(A)$. This means that we must say what $A|\psi\rangle$ is for each $|\psi\rangle \in D(A)$. Usually we do so establishing a rule in terms of a general $|\psi\rangle$, perhaps using a basis, or sometimes we can do so in an indirect manner.
As for definining a function, this happens in all mathematics: to define a function, we need a set, and then we define the function on some subset of this set. So it is not possible to define a function, unless we know beforehand: (i) the set $\mathcal{H}$, (ii) the domain $D(f)\subset \mathcal{H}$ and (iii) the range $\mathcal{H}'$.
In the case of Hilbert spaces, $\mathcal{H}$ is a known Hilbert space to the problem, $D(f)$ is the domain of the operator and $\mathcal{H}'=\mathcal{H}$. Let us call $\mathcal{L}(\mathcal{H})$ the set of all operators on the Hilbert space $\mathcal{H}$.
This is just mathematics. Now, if we want to define a field of operators on spacetime, what do we need? Well, following this logic (which is the standard math, not anything fancy) we need a function $\phi : M\to \mathcal{L}(\mathcal{H})$. But, wait a minute, to build this function we need to say how it acts. In other words, for each event $x\in M$ we must say what $\phi(x)$ is.
Fine, so how do we say what is $\phi(x)$? It is an operator in $\mathcal{H}$. Hence to define $\phi(x)$ we need to say how it acts on $\mathcal{H}$, in its domain $D(\phi(x))$. In other words, we need to specify for each $x\in M$ what is the action $\phi(x)|\psi\rangle$ for each $|\psi\rangle \in \mathcal{H}$, otherwise we haven't defined $\phi$ at all.
One can argue that quantum fields must be viewed as operator-valued distributions, although I'm still unsure if this is the standard approach, but anyway, the problem remains and it is the same thing. To define the quantum field $\phi(f)$ we must say how it acts $\phi(f)|\psi\rangle$ for each test function $f$ and each $|\psi\rangle\in \mathcal{H}$.
And of course, we need $\mathcal{H}$. Although this is less important since all Hilbert spaces of same dimension are isomorphic. On the other hand, truly defining $\phi$ is vital.
Now comes the question: what do physicists do in QFT? They pick one scalar field $\phi(x)$ and say: "ok, now we just make $\phi(x)$ become operators obeying $[\phi(x),\pi(y)]=i\delta(x-y)$ and we are done". Even more is done! One writes $\phi(x)$ in terms of other operators $a(p)$, which are also not known and relate to the commutation relation. Then you imagine: "fine, the next step is naturally to define these operators" and nothing is done, all operators are left undefined with the commutation relations proposed. How can one have commutation relations and not have the operators?
In quantum mechanics I can even accept that. The state space $\mathcal{E}$ is there by postulate, the observables are there by postulate, and we assume all of them (we can even invoke Stone-von Neumann theorem if we want to be more rigorous). In QFT we have a field, so we need the functional dependency $\phi(x)$, and neither $x\mapsto \phi(x)$ nor $|\psi\rangle \mapsto \phi(x)|\psi\rangle$ are ever defined.
In that sense I'm really confused. What does actually mean for physicists in the context of QFT to define a field? How can one work with operator-valued fields (distributions) when one just says that commutation relations are obeyed without ever defining the fields and operators? What is actually going on here?