I think a way to visualize this is if you imagine that the ball suddently popped into existence at time $t$ and position $x$ for observer $A$. Then observer $A'$ would see the ball pop into existence at time $t'$ and position $x'$, with $(t', x')$ related to $(t, x)$ by a Lorentz transformation. In this case the event is unambiguous (both observers should agree that something popped into existence at some point). Physically this is a bit unrealistic, but you can imagine other events such as the ball's color changing, or the ball colliding with a wall. Such events are unambiguous.
Alternately, you can suppose that observer $A$ has set up a network of synchronized clocks and detectors that are all at rest in his own frame, and similarly observer $A'$ has his own system of clocks and detectors at rest in his own frame. The trajectory of the ball is observed by $A$ and $A'$ via the collection of all detection events and clock readings (i.e. the set of all tuples $(t, x)$ collected by the detectors over the duration of the experiment). Thus what the observers really observe is a coordinate representation of the ball's worldline in their respective frames, i.e. they measure the path $x^\mu(\tau)$, but only up to unit-length reparametrizations of proper time $\tau$ (i.e. paths $x^\mu(\tau)$ and $x^\mu(\tau + \tau_0)$ are really the "same" worldline and physically indistinguishable). The ambiguity in choosing the parametrization of $\tau$ is related to the ambiguity you asked about whether $A$ and $A'$ are looking at the same event.
If the particle is moving in a straight line with uniform velocity for ever and ever after with no external interactions, then there is no way to agree on a specific origin of proper time unambiguously. But in practice, there will be a natural way to choose the origin of proper time unambiguously, such as defining $\tau = 0$ to represent the point where the particle collided with a wall, or changed its color, or reached a maximal/minimal speed. Then both observers can use $\tau = 0$ to represent this event, and $x^\mu(\tau)$, ${x^\mu}'(\tau)$ will represent the same events for all other $\tau$ and thus be related by Lorentz transformations.
(Actually, they also need to agree on the direction of time, due to $x^\mu(\tau)$ and $x^\mu(-\tau)$ representing the same worldline too. So there needs to be another unambiguous event that allows defining the direction of time, such as a second collision, or some assumption of entropy increase, or assumption that the ball eventually slows down due to friction forces, etc).