What exactly does the observer's coordinate system mean in special relativity? This seems like a very basic question but I'm having a hard time understanding it, more precisely hard time visualizing it. Let say there is an observer, and we wish to somewhat formalize his observation by using coordinate system. 
We adopt four dimensional Euclidean coordinate system where three of them measure space-coordinates in xyz direction and the other measures time $t$, let us denote this as $(t,x,y,z)$ aka event coordinate. We have no trouble measuring the coordinate of the observer himself, we can just set his location as the origin $(0,0,0,0)$. But I think a problem occurs when we wish to denote the location of some object. An object he observes at time $t$ happens to be at the point $(x,y,z)$, so its event coordinate is $(t,x,y,z)$. If I were to visualize this, I'd draw 3D Cartesian space and mark points at $(0,0,0)$ and $(x,y,z)$.
But this is actually wrong, because you don't know that the object is still there at $(x,y,z)$. What the observer is actually seeing is the past of the object $\sqrt{x^2+y^2+z^2}/c$ seconds ago (here $c$ is the speed of light), so you can't really mark the point at $(x,y,z)$ and call it the location of the object.
So how one can correctly visualize the location of observed object?
 A: In general, a coordinate system is not associated with any observer at all. Any observer can use any coordinate system, and the coordinate system doesn't have to be "rigid"; it can twist and turn and change in all kinds of ways. Whatever coordinate system we use is chosen based on convenience, not dictated by whatever observers are involved.
As highlighted in the OP, the relationship between a coordinate system and what the observer actually sees is only indirect at best. No matter what coordinate system we use, determining what a given observer sees requires doing some "ray-tracing" — that is, determining which lightlike geodesics connect the object-of-interest's worldline to the observer's worldline, and paying attention to the proper times along each of those two worldlines. And that's if the object is pointlike; if it's an extended object, then we need to consider each little piece of the object separately. Some special situations may admit special shortcuts, but in general, this is what we need to do. These calculations might be easier in some coordinate systems than others, but the important point is that it's the calculations — not the coordinate system — that determines what the observer sees.
When people talk about "the" coordinate system associated with a particular inertial observer in special relativity, they usually mean a coordinate system $t,x,y,z$ that has both of these two properties:


*

*The given observer's worldline is described by $(x,y,z) = (0,0,0)$.

*The proper time increment $d\tau$ is given by 
$$
d\tau^2=dt^2-\frac{dx^2+dy^2+dz^2}{c^2}
\tag{1}
$$
along any timelike worldline — including arbitrary non-inertial worldlines.
By the way, if the observer is non-inertial, then no coordinate system has both of these properties.
In the coordinate system highlighted above, the given observer's proper time is simply $\tau=t$. However, there are lots of of different coordinate systems in which the given inertial observer is at $(x,y,z)=(0,0,0)$ and has proper time $\tau=t$. An observer is described by a single worldline, whereas a coordinate system can cover all of spacetime. What the coordinate system does along the given observer's worldline doesn't say much about what it does elsewhere. That's why when people talk about "the" coordinate system associated with a given inertial observer, the more restrictive condition (1) is usually implied.
The post
How do frames of reference work in general relativity, and are they described by coordinate systems?
offers more thoughts about this subject from the perspective of general relativity.
A: When an observer physically located at $(0,0,0)$ "observes" an event at $(t,x,y,z)$, the observer actually makes the measurement at $t^\prime=t+r/c$, where $r=\sqrt{x^2+y^2+z^2}$, and extrapolates that the object must have been at $(x,y,z)$ at time $t$ in their reference frame.
A: The special relativity usage differs significantly from the ordinary English meaning of "observer".
It is rather a whole frame of reference from which objects or events are being measured.
Reference frames are inherently nonlocal constructs, covering all of space and time or a nontrivial part of it; thus it does not make sense to speak of an observer (in the special relativistic sense) having a location.
Speaking of an observer in special relativity is not specifically hypothesizing an individual person who is experiencing events, but rather it is a particular mathematical context which objects and events are to be evaluated from. 
To some extent the term "observer" should be replaced by an observer team (or family of observers). Each of these observer knows his spatial coordinate, has a clock*  and makes observations in immediate vicinity, where delays are negligible, cooperating with the rest of the team to set up synchronized clocks across the entire region of observation, and all team members sending their various results back to a data collector for synthesis.
*all these clock have been previously synchronized across the whole reference frame by Einstein's synchronization technique, which assumes that one - way speed of light is isotropic and equal to the value с.
