Why do $\hat{X}$ and $\hat{P}$ have to correspond to position and momentum?

As far as I understand, in QM we treat observables as operators, and the eigenvalues of these operators are the possible values we can measure of the observables. It is usually simpler to work in the eigenbasis of an operator if we are talking about its corresponding observable. I have a doubt regarding the interpretation of two operators in QM.

The two texts that I've turned to both say something along the following lines:

First Part

If we have a function $f(x)$, we can think of it as being the projection of the vector $|f\rangle$ on the $|x\rangle$ basis. For this basis, we must have $\langle x|x'\rangle=\delta(x-x')$, so that the basis kets are orthogonal, and the completeness relation is satisfied.

Next, we define an operator $\hat{K}=-i\hat{D}$ where $\hat{D}$ is the differential operator. This operator has an eigenbasis $|k\rangle$. Interestingly enough, there's an awesome relation between the function $f(k)$, which is $|f\rangle$ expanded in the $|k\rangle$ basis, and $f(x)$, the function we'd talked about before. They're each others' Fourier transforms: $$f(k)=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^\infty e^{-ikx}f(x) dx$$ Up to here, I can follow comfortably. All that was needed was to solve the eigenvalue problem for $\hat{K}$ in the $|x\rangle$ basis, which gives the relation between the two bases, and then churn through a few calculations.

Second Part

My problem is with the interpretation of the operators we just spoke of. Just because we gave them the names $\hat{X}$ and $\hat{K}$, do we have to interpret them as position $x$ and wave number $k$? I understand that to each observable corresponds a Hermitian operator, but what if, for example, we had started with two operators named $\hat{A}$ and $\hat{B}$ instead of $\hat{X}$ and $\hat{K}$, and found the following: $$f(a)=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^\infty e^{-iab}f(b) db$$

Where we're working with the bases $|a\rangle$ and $|b\rangle$. Seeing these letters, maybe I decide to associate with $\hat{A}$ the acceleration of a particle, and with $\hat{B}$ the particle's position, or even worse, the magnetic field that the particle is in. I know that this is a crazy example, but I want to illustrate my point- What is it that tells us that the position and momentum (or wave number) in QM must be associated to those operators that I mentioned in the first part? Thanks in advance.

PS. If anyone's curious, I'm reading Shankar's and Zettili's texts.

The answer may be deeper than you expect.

The operator $K$ that the OP defined --- here it will be called $-i\nabla_x$ --- is the generator of space translations. And the operator $X$ --- here called $x$ --- is the generator of momentum translations. If you exponentiate them, to form the corresponding one-parameter unitary groups of translation, you get $V(\xi)=e^{i\xi\cdot x}$, and $T(q)=e^{q\cdot\nabla_x}$. Their action on a wavefunction $\psi(x)$ (position representation) or $\hat{\psi}(k)$ (momentum representation) is quite clear: $$(T(q)\psi)(x)=\psi(x+q)\; ,\\ (\hat{V}(\xi)\hat{\psi})(k)=\hat{\psi}(k+\xi)\; .^1$$ These two operators satisfy the following commutation relation, due to Weyl (essentially by definition): $$V(\xi)T(q)=e^{-i\xi\cdot q}T(q)V(\xi)\; .$$

Now there is a very deep theorem of Stone and von Neumann that tells you that every (irreducible) representation as operators on a Hilbert space of such a family of objects $\{V(\xi),\xi\in\mathbb{R}^d\}\cup\{T(q),q\in\mathbb{R}^d\}$ that satisfy the Weyl commutation relation is unitarily equivalent to the representation described above in $L^2(\mathbb{R}^d)$.

So there is a unique way, up to unitary transformations, of representing such operators; and that way is by translation operators on $L^2(\mathbb{R}^d)$. Now from the Weyl commutation relation you can obtain the commutation relation for the generators, and that is $$[x,-i\nabla]=i\; .$$

So the operators $x$ and $-i\nabla$ have all the observed physical properties of the position and momentum operators: they satisfy the canonical commutation relations (quantization of the classical Poisson brackets), and they are the generators of space and momentum translations when acting on $L^2(\mathbb{R}^d)$. In addition, they are special operators that can only be represented in Hilbert spaces in a way unitary to the $L^2$ one. In my opinion, that is enough evidence to undoubtedly use them to describe position and momentum in quantum mechanics.

$1$. In the usual representation, obviously $(V(\xi)\psi)(x)=e^{i\xi\cdot x}\psi(x)$.

I would say the answer is rather simple. If you do various experiments to measure the position (or momentum) of a particle, then QM only gives you consistent and accurate predictions when the mathematical symbols $x$ and $p$ are associated with the measured position or momentum.

However, position and momentum are not the only canonically conjugate operators in quantum mechanics. Other examples include the quadratures of the electric field, or the phase and number operators of a Bose-Einstein condensate. In each case, the QM models give good predictions when the operators associated to these observables satisfy similar properties to $x$ and $p$.