fig1: A marked ruler as it appears in its rest frame $S'$ (top) and as it appears to observers with respect to whom it is moving with uniform constant velocity (frame $S$, bottom). The appearances have been aligned so that the origins coincide. Clearly, the ruler appears contracted towards the origin ($\gamma=1.2$).
Elaborating further, when we talk of the LC of a moving rod, what we really mean is the length between the simultaneous measurements of the positions of its ends (by stationary observers). The fact that this length is smaller (wrt. its value in its rest frame) doesn't imply that one end is contracting towards the other as the other end remains fixed. As you point out, this is ambiguous. Instead distances of both ends measured from the origin, contract towards the origin, by the same factor, and so does their difference. Indeed, every point of the rod, nay, the entire space$^{[1]}$ contracts towards the origin. Since the contraction is uniform, there is no fracture.
Is there anything special about the origin of the moving frame? Absolutely not. The fixed point is determined by the initial alignment of the origins of the two frames that are moving relative to each other, and the initial sync of their clocks. Indeed, with an appropriate choice, the origin can be arbitrarily shifted. This is why associating the quality of 'towardness' to scaling isn't really useful.
This skippable section puts the above explanation on a firmer footing. let $S'$ be a reference frame ('rod frame') moving with constant uniform velocity $\beta\hat x$$^{[2]}$ wrtwrt. the frame $S$ ('lab/observer frame'). Further, by choice, let their origins coincide and axes align when their clocks are synchronized. Let some arbitrary event $E$ be described by the coordinates $(t',x')$ in $S'$ and $(t,x)$ in $S^{[3]}$. These are connected by a Lorentz transformation:
$$ \begin{align} x'&=\gamma(x-\beta t)\\ t'&=\gamma(t-\beta x) \end{align}\tag{1} $$
where where $\gamma=(1-\beta^2)^{-1/2}$.
Consider the two events as described by observers in $S$ (for some $L>0$).
$$ \begin{align} E_0&:(t,\beta t)\\ E_1&:(t,\beta t\pm L)\tag{in $S;\>2$} \end{align} $$ As
As you may have guessed, $E_0$ is the spacetime coordinate of the origin of the moving frame, and $E_1$ an arbitrary point at distance $L$ (wrt. observers in $S$) fromfrom it. Note the same time coordinate: the events must be simultaneous for observers in $S$ to constitute a length measurement.
Observers in frameframe $S'$, on the other hand, describe the same events with coordinates $$ \begin{align} E_0&:(t/\gamma,0)\\ E_1&:(t/\gamma\mp\beta\gamma L,\pm \gamma L)\tag{in $S';\>3$} \end{align}$$
$$ \begin{align} E_0&:(t/\gamma,0)\\ E_1&:(t/\gamma\mp\beta\gamma L,\pm \gamma L)\tag{in $S';\>3$} \end{align}$$ Ignore the time coordinate (it's complicated because of time dilation). The space coordinate of $E_0$ is always $0$, which it must be since to observers in $S'$ it corresponds to their origin. Note the correct contraction: observers in $S'$ conclude that observers of $S$ are measuring a contracted $L$ between the two events instead of the 'correct' $\pm \gamma L$. Notice how the $\pm$ denotes the contraction towards the origin.
But $E_1$ could have referred to any point. So the length of all points measured from the point $\beta t$ at time $t$, appearsappears contracted by $\gamma$ to the observers inin $S$, in the opinion of observersobservers in $S'$. Indeed, as seen in eqns. $(1)$ for the Lorentz transformation, the contraction happens on the length $x-\beta t$ i.e. length measured (by observers in $S$) from the origin of the moving ref. frame.