Whenever physicists or physics students start an experiment or thought-experiment, the first thing we do is try to isolate the system from all outside influence so that we can precisely vary initial conditions, let the system go, and watch what happens. Metaphorically, experimentation (and solving textbook problems) is a process of building a tiny universe in a box and looking at it through a peep-hole while we start it running, let the entire history of spacetime in the universe elapse, and then destroy and rebuild the universe if we want to test it again.
Even in relativity, it's convenient to take this approach. We might need a bigger box to fit blueshifting galaxies and space ships zipping past each other at 0.99c, but building a metaphorical box around an isolated section of the universe and putting the observer outside of it, peering in through a peep-hole, works great. We position observers in the box, let them take their measurements, and have them send their measurements to us through the peep-hole.
The approach completely pervades our thinking about how to think about the sciences, to the point that we naturally reach for it whenever we try to answer any question. But it is a fallacy. The observer has to be inside the universe with the experiment, affecting and being affected by the experiment. By definition, a universe is everything that can ever affect and be affected by everything else, so if our observer is really outside of the universe, they can make no observations - or even falsifiable guesses - about what's going on inside.
Therefore the success of the universe-in-a-box method comes not from putting the observer outside, but from using a consistent observer: similar mass, velocity, position, charge, etc... and keeping them far enough away from the subject of experimentation so that their contribution to the initial conditions of the experiment is small.
For cosmology questions, the universe-in-a-box method completely breaks down. We want to ask a question about everything, and the first thing we reach for, because it's worked so well for us so far, is to put the universe in a box and put the observer "outside" - or if we're wise to the fallacy of our own thinking, to put a humanlike observer in a specific place at the edge of the box.
Neither approach works. The naieve approach, observer outside the universe, doesn't work, because the box is the whole universe and there's no "outside" to put the observer in. The nuanced approach doesn't work because wherever we put the observer, they'll be in the exact center of their observable universe, and besides, we don't want to ask a question about what the cosmos is like from the perspective of a human on the edge of the universe, we want to know what the cosmos is like, period. So, when we ask cosmology questions, we have to frame the question to include a variable observer.
Our whole system of doing mathematical physics is to chart a sequential procession of present configurations in a way that can be expressed in a system of equations or charted on a graph. But we can't make any definitive statements about the present configuration of the cosmos, because without a single definitive observer, we don't have access to a single definitive present.
This doesn't stop cosmologists from making mathematical statements about the nature of the cosmos... but it does make their job conceptually and mathematically difficult.