I think a way to visualize this is if you imagine that the ball suddenly popped into existence at time $t$ and position $x$ for observer $A$. Then observer $A'$ would see the ball pop into existence at time $t'$ and position $x'$, with $(t', x')$ related to $(t, x)$ by a Lorentz transformation. In this case the event is unambiguous (both observers should agree that something popped into existence at some point). Physically this is a bit unrealistic, but you can imagine other events such as the ball's color changing, or the ball colliding with a wall. Such events are unambiguous.
Alternately, you can suppose that observer $A$ has set up a network of synchronized clocks and detectors that are all at rest in his own frame, and similarly observer $A'$ has his own system of clocks and detectors at rest in his own frame. The trajectory of the ball is observed by $A$ and $A'$ via the collection of all detection events and clock readings (i.e. the set of all tuples $x^\mu = (t, x)$ collected by the detectors over the duration of the experiment). Thus what the observers really observe is a coordinate representation of the ball's worldline in their respective frames, i.e. they measure the path $x^\mu(\tau)$, but only up to unit-length reparametrizations of proper time $\tau$ (i.e. paths $x^\mu(\tau)$ and $x^\mu(\tau + \tau_0)$ are really the "same" worldline and physically indistinguishable). The parameter $\tau$ is simply a label for an arbitrary element of the set of tuples $(t, x)$ collected by the detectors. The ambiguity in choosing the parametrization of $\tau$ is related to the ambiguity you asked about whether $A$ and $A'$ are looking at the same event.
If the particle is moving in a straight line with uniform velocity for ever and ever after with no external interactions, then there is no way to agree on a specific origin of proper time unambiguously. But in practice, there will be a natural way to choose the origin of proper time unambiguously, such as defining $\tau = 0$ to represent the point where the particle collided with a wall, or changed its color, or reached a maximal/minimal speed. Then both observers can use $\tau = 0$ to represent this event, and $x^\mu(\tau)$, ${x^\mu}'(\tau)$ will represent the same events for all other $\tau$ and thus be related by Lorentz transformations.
(Actually, they also need to agree on the direction of time, due to $x^\mu(\tau)$ and $x^\mu(-\tau)$ representing the same worldline too. So there needs to be another unambiguous event that allows defining the direction of time, such as a second collision, or some assumption of entropy increase, or assumption that the ball eventually slows down due to friction forces, etc).
Another option is for the observers to agree on another event independent of the ball, such as a light switch turning on to signal the start of the experiment, and turning off to signal the end of the experiment. Then they can use $\tau = 0$ to label the ball's position when the light switch turned on (assuming the detectors record the state of the light switch as well as the ball), and require that the light turns off for some $\tau \gt 0$ to set the direction of time. This does not require using a distinctive feature in the particle's trajectory.