This is where the textbooks, in a way, lie to you. The operator $\hat{x}$ (and its counterpart, $\hat{p}$) is not a "good" quantum operator for a number of reasons, including the fact that these operators do not have normalizable eigenvectors, as you have seen. In particular,
$$|x\rangle$$
is not a sensible eigenvector as it is not normalizable. Rather, it just an "ideal point" of the space, a Hilbert-space analogue of $\infty$ in the real numbers (there's just a lot more of them), that makes it easy for us to locate a state in terms of a positional wave function by writing
$$|\psi\rangle = \int_{-\infty}^{\infty} [\psi_x(x)\ dx]\ |x\rangle$$
. Indeed, it would be better to write this as
$$|\psi\rangle = \int_{-\infty}^{\infty} [\psi_x(x)\ dx]\ ``|x\rangle"$$
with scaredy quotes!
And yes, this means that a particle cannot ever be localized perfectly to a single point in space. There is no such thing as a particle in state $|x\rangle$! In a way, this makes Galilean quantum mechanics not so different from its relativistic counterpart, relativistic quantum field theory or RQFT - the only difference is that there is no upper bound on the strength of localization permitted. (This should make physical sense when you realize GQM is just the $c \rightarrow \infty$ limit of RQFT; the details may change, but the paradigm cannot.) That is, GQM permits unbounded localization, not perfect localization.
But in neither case does this mean position is unmeasurable at all. Instead, we have to remember that all position measurements will only extract a finite amount of bits from the system. This, in turn, implies we are talking about localization to (certain, "good") subsets of space, $\mathbb{R}^3$ (in RQFT, it gets more complicated than this, because in principle, localization on a set still implies a sharp border, and we cannot do that either). Namely, while we cannot ever have either states where $x = x_0$ nor measurements asking if $x = x_0$, we can have states where $x \in S$ and measurements asking if $x \in S$, so long as $\mu(S) > 0$, where $\mu$ is the Lebesgue measure.
A way to formalize this that avoids overly-complicated mathematical constructs of the type required to use impossible things like $|x\rangle$ is to ditch the usual observables formalism in favor of an answer operator or projection operator formalism. In this case, if we are representing our vectors using the textbook "nonsense", i.e.
$$|\psi\rangle = \int_{-\infty}^{\infty} [\psi_x(x)\ dx]\ ``|x\rangle"$$
then we have a family of answer operators for position, of the form $\hat{A}(x \in S)$, which each mean "I heard that $x \in S$". The action of this thing on an honest quantum vector $|\psi\rangle$ then looks like
$$[\hat{A}(x \in S)]|\psi\rangle = \int_{-\infty}^{\infty} [\mathbf{1}_S(x)\ \psi_x(x)\ dx]\ ``|x\rangle"$$
In effect, we just change the positional wave function by multiplying by the indicator function $\mathbf{1}_S$ of the set $S$. You can see this is a projection because two applications leaves $|\psi\rangle$ unchanged, viz.
$$[\hat{A}(x \in S)]^2 = \hat{A}(x \in S)$$
And the meaning is that we are describing measurements by their "collapses" after they return a given result, here that the position $x$ was found to be within the set $S$, even if the measurement doesn't give it a number. These are, in effect, one-bit measurements or yes-no measurements. If one wants to take the subjective thesis regarding quantum states, where that $|\psi\rangle$ is the knowledge held by an agent regarding possible questions that can be asked about a system, the answer operator describes updating our agent's knowledge with a single bit of new information.