I would say that one part of the question is still unanswered. That would be, how the operator $\hat{p}$ acts on a state that is not trivial linear combination of states of definite position.

I would say that one part of the question is still unanswered. That would be, how the operator $\hat{p}$ acts on a state that is not a trivial linear combination of the position eigenstates.

Let's say we are trying to calculate $$\hat{\frac{\partial}{\partial x}}|n\rangle=\hat{\frac{\partial}{\partial x}} \int dx' \langle x'|n\rangle |x'\rangle, $$ where $|n\rangle$ is some state that can be represented as a linear combination of states $|x\rangle$. To grasp the intuition we can check the case of $$\hat{x}|n\rangle=\hat{x} \int dx' \langle x'|n\rangle |x'\rangle, $$ where the answer is very simple, as the $|x\rangle$ is eigenstate of the $\hat{x}$ operator with eigenvalue $x$ $$\hat{x}|n\rangle=\int dx' \langle x'|n\rangle \hat{x}|x'\rangle = \int dx' \langle x'|n\rangle x'|x'\rangle. $$ Applying the same reasoning to the initial case, we get $$\hat{\frac{\partial}{\partial x}} \int dx' \langle x'|n\rangle |x'\rangle=\int dx' \langle x'|n\rangle \hat{\frac{\partial}{\partial x}}|x'\rangle=\int dx' \langle x'|n\rangle \lim_{h\rightarrow 0}\frac{1}{h}\Big(|x'+h\rangle - |x'\rangle\Big), $$ where I simply used the definition of the derivative on a vector. Then we simply separate the integral on two parts, and redefine the integration variables so that we can extract the state $$=\lim_{h\rightarrow 0}\frac{1}{h}\Big(\int dx' \langle x'|n\rangle|x'+h\rangle - \int dx' \langle x'|n\rangle|x'\rangle\Big), $$ $$=\lim_{h\rightarrow 0}\frac{1}{h}\Big(\int dx'' \langle x''-h|n\rangle|x''\rangle - \int dx' \langle x'|n\rangle|x'\rangle\Big), $$ $$=\lim_{h\rightarrow 0}\frac{1}{h}\int dx' \Big(\langle x'-h|n\rangle - \langle x'|n\rangle\Big)|x'\rangle, $$ but this is the derivative of the coefficients $$=-\int dx' \Big(\frac{\partial\langle x|n\rangle}{\partial x}\Big)\Bigg|_{x = x'}|x'\rangle, $$ or in a more familiar form with the wave function defined as $\langle x|n\rangle = \psi_{n}(x)$ $$\hat{\frac{\partial}{\partial x}}|n\rangle=\hat{\frac{\partial}{\partial x}} \int dx' \psi_{n}(x') |x'\rangle = -\int dx' \psi'_{n}(x') |x'\rangle. $$ Therefore acting with the spatial derivative on a state gives you the derivative of a wave function, or in other words, the derivative of the coefficient that gives you the mapping from a state you are differentiating to the position basis.