This question has caught my attention because I was unaware of the fact that the position-momentum canonical commutation relations could be derived out of the only assumption for $\langle x | p\rangle$ to be a plane wave $\textrm{e}^{ipx}$. It seems that the trick follows from the properties of the exponential function and the Dirac delta to be proportional to their derivatives times the exact accidental factors appearing in the game.
I was wondering whether the same could apply to any other pair of operators: namely let $A, B$ be two non-commuting operators with $\{|a\rangle\}, \{|b\rangle\}$ being the corresponding set of eigenstates. Along the same lines as in the answer one has: $$ \langle a |[A,B]| a' \rangle = (a-a')\int\textrm{d}b'\,b'\,\langle a | b'\rangle\langle b' | a'\rangle $$ or any other sum replacing the integral according to what the eigenvalues look like. In the above general case nothing can be said (can it?); in the other very special case of the angular momentum, with the eigenfunctions being the spherical harmonics, do we still have any accidental simplification of the wave functions in order to gain back the $\textrm{SU}(2)$ commutation relations (maybe due to some special recursive properties of those functions)?
Generalising, the question can be put as: given $A, B$ two self-adjoint operators, does the scalar product between elements of the corresponding complete sets of eigenstates fully determine the commutator (i. e. the action) of the two operators on the entire Hilbert space?