Assuming you arrange your problem to maintain self adjointness, $L^2$, etc, you could work in the x-representation, in which case the inverse of *X* would be represented by *1/x*.

For the inverse of *P<sub>x</sub>*, acting on a function *f(x)*,  you'd consider the antiderivative 
$$
P_x^{-1} f(x)= {i\over \hbar} \int ^x_{-\infty} \!\! dy ~f(y) ~,
$$
provided  *f* vanishes at −∞. In that case, 
$\partial_x   \int ^x_{-\infty} \!\! dy=  \int ^x_{-\infty} \!\! dy ~\partial_x =1$.

Heaviside's [operational calculus](https://en.wikipedia.org/wiki/Operational_calculus) uses such tricks as a scaffolding, but you'd have to be quite careful with boundary conditions, and shrug off mathematical counterexample purveyors... It's a dark art. Oliver Heaviside kept them at bay with peremptory dismissals, usually warranted: 
>I think I have given sufficient information to enable any competent person to follow up the matter in more detail if it is thought to be desirable. It is obvious that the methods of the professedly rigorous mathematicians are sadly lacking in demonstrativeness as well as in comprehensiveness.  

You may find toy examples to amuse yourself this way, 
explore $[X^{-1},P]=-i\hbar X^{-2}$, and $[X,P^{-1}]=-i\hbar P^{-2}$, etc... In matrix mechanics the analog would be harder, but maybe you can be creative.

NB, if you wanted the hidebound formal definition of the above, without the standard  notational somersault that confuses some, 
$$
X^{-1}=  \int^\infty_{-\infty} \!\! dx~~|x\rangle { 1\over x}   \langle x|~~,  \\
P= \int^\infty_{-\infty} \!\! dx~~|x\rangle {\hbar\over i} \partial_x \langle x|~~,  \\
P^{-1}= \int^\infty_{-\infty} \!\! dx~~|x\rangle { i\over \hbar} \int ^x_{-\infty} \!\! dy ~\langle y| ~~, 
$$
essentially a Heaviside step function.