Quantum decoherence distinguishes the whole big system into "system" and environment, and shows how system, when density matrix is traced over environment, comes to be decoupled from environment.
But this requires distinguishing environment from system, and I do not get how clear separation is possible. Doesn't the fact the whole big system is quantum should bring caution to separating systems arbitrarily, especially considering special relativity effects?
The fact that the whole system is a quantum system is not relevant to this issue and neither does special relativity prohibits this separation as it is not physical.
The separation is made on the basis of what is the system that is the subject of an experiment. The environment is then everything else, including you.
Decoherence then comes from the practical impossibility of totally decoupling the system from this environment. As such the environment interacts with the system potentially causing it to collapse in an eigenstate of this interaction. We say this interaction causes collapse (or however you prefer to interpret measurement) when the information about the state is widely repeated throughout the environment, thus being classical as measuring the environment is now equivalent to having measured the state in the same basis. This is einselection.
Quantum decoherence distinguishes the whole big system into "system" and environment, and shows how system, when density matrix is traced over environment, comes to be decoupled from environment.
This summary is wrong. The system is coupled to the environment. As a result of that coupling interference is suppressed and you can trace over the environment to get a mixed state instead of a pure state.
But this requires distinguishing environment from system, and I do not get how clear separation is possible. Doesn't the fact the whole big system is quantum should bring caution to separating systems arbitrarily, especially considering special relativity effects?
Separation between systems is not arbitrary. You can interact with the environment without interacting with the system and vice versa. So they are separate systems.
You may be thinking that quantum mechanics is non-local and you can change the state of system A by interacting with system B. In reality, quantum mechanics is entirely local:
https://arxiv.org/abs/quant-ph/9906007.
You might think that Bell's theorem implies that quantum mechanics is non-local, but if so you are wrong. Bell's theorem implies that if systems are described by stochastic variables, then to match the predictions of quantum mechanics they would have to interact non-locally. But in quantum mechanics, systems are described by Heisenberg picture observables, not stochastic variables and the observables change locally.
