I think a way to visualize this is if you imagine that the ball suddently popped into existence at time $t$ and position $x$ for observer $A$. Then observer $A'$ would see the ball pop into existence at time $t'$ and position $x'$, with $(t', x')$ related to $(t, x)$ by a Lorentz transformation. In this case the event is unambiguous (both observers should agree that something popped into existence at some point). Physically this is a bit unrealistic, but you can imagine other events such as the ball's color changing, or the ball colliding with a wall. Such events are unambiguous.

Alternately, you can suppose that observer $A$ has set up a network of synchronized clocks and detectors that are all at rest in his own frame, and similarly observer $A'$ has his own system of clocks and detectors at rest in his own frame. The trajectory of the ball is observed by $A$ and $A'$ via the collection of all detection events and clock readings. Thus what the observers really observe is a coordinate representation of the ball's worldline in their respective frames, i.e. they measure the path $x^\mu(\tau)$, but only up to unit-length reparametrizations of proper time $\tau$ (i.e. paths $x^\mu(\tau)$ and $x^\mu(\pm\tau + \tau_0)$ are really the "same" worldline and physically indistinguishable). The ambiguity in choosing $\tau$ is related to the ambiguity you asked about whether $A$ and $A'$ are looking at the same event.