If they do a Fourier transform, how can they know the formula to find coefficients?
Cochleae, the little spiralling tubes in our inner ears, separate the frequencies of incoming mechanical wave.
Human cochlea is equipped with thousands of hair cells attached to organ of Corti that runs along the length of the tube. Hair cells vibrate with the surrounding liquid. Each of the hair cells have different resonant frequencies. That way the ear get the 'coefficients' (more precisely, spectral power distribution)
(Imagine having thousands of radios tuned to different frequencies to get a really fast Fourier Transform of an incoming radio wave!)
Obviously ears don't "know" a formula. But, the process of trial and error and keeping all the little incremental improvements we call evolution can result in some pretty sophisticated (although often kludgy) solutions.
In the case of our ears, there is a mechanical structure that sortof separates frequencies spacially, or at least produces different frequency filtering of the input signal as a function of distance. Different nerves sense vibrations at different points along this filter structure, and the rest is signal processing to produce the perception of a frequency spread.
More details belong elsewhere, possibly on the biology site.
Link that might answer your question in part: http://www.biophysics.uwa.edu.au/e_book/9d.html
Definition of the phon scale and what loudness means. This lecture note is very comprehensive, the only problem is in german:
General biophysical explanation on the ear and the various topics concerning hearing like chromatics etc.
This one is in english:
The same topics are covered but in english.