The idea goes back to Heisenberg. He believed that physics could only describe quantities that could be measured experimentally, and sought to develop a mathematical theory that would reflect this and correctly predict the relative intensities of spectral lines.
In classical physics, the radiated intensities depend (in the first approximation) on electric dipoles, which are dependent on the position of the electrons. To account for the fact that the position of the electrons cannot be measured during a transition, he introduced a number $x_{nm}$ to characterize the position in the transition from $n\to m$. He also introduced $v_{nm}$ for the velocities of the electrons during the transitions and a related acceleration $a_{nm}$.
Heisenberg was eventually able to reproduce the energy levels $E_n$ (actually their differences $E_n-E_m$) using these quantities but only if the quantities satisfied the "unusual" combination properties
$$
\sum_{m} x_{nm} v_{mk} = A_{nk} \ne A_{kn} = \sum_m v_{km} x_{mn}\, .
$$
In particular, using his "tables" of $x_{nm}$ and $v_{nm}$ he was able to work out what we now write as $[x,p]$.
The story goes that Pascual Jordan met Heisenberg in a train at the time Heisenberg was working on this. Jordan, who had mathematical training, recognized the combination rule as matrix multiplication. Jordan, together with Max Born and Paul Dirac, realized that the use of non-commutative quantities was essential to Heisenberg's description.
Dirac in particular postulated that the multiplication rules had to follow from dynamical considerations; inspired by the correspondence principle, he was able to relate, up to an overall factor, the classical Poisson bracket to a quantum bracket to find the now-famous
$$
[q_i,q_j]=0\, ,\qquad [p_i,p_j]=0\, ,\qquad [q_j,p_k]=i\hbar \delta_{jk}\, .
$$
There are several accounts of this discovery. The most historical is by Max Jammer, who was able to interview firsthand some of actors in the story. There is also an interesting and more recent text by Roland Omnes but it doesn't focus so much on the history. I'm sure there are others.
Edit: After reading @hyportnex account, I found Jammer online and checked. hyportnex account is accurate when it comes to Born recognizing the matrix form of Heisenberg's expression. As to the story of the train: it is Born that met Jordan in a train. Quoting from Jammer, page 109:
Now it happened that Born, while traveling by train to Hanover, told a colleague of his from Gottingen about the fast progress of his work but also mentioned the peculiar difficulties involved in the calculations with matrices. It was fortunate and almost an act of providence that Jordan, who shared the same compartment in the train, overheard thispiece of conversation. At the station in Hanover Jordan then introduced himself to Born, told him of his experience in handling matrices, and expressed his readiness to assist Born in his work.