How to distinguish female and male voices via Fourier analysis? What makes one, without looking, be able to identify the gender of the talker as male or female?
I mean if we Fourier analysed the voice of males and females, how the 2 spectrums are different which account for that distinction in sounds?
 A: My impression would be that the lower frequencies are more apparent in the male spectrum than the female spectrum.
If you want to build a nice test, my approach would be to determine some average male and average female spectrum. Then you can see which of your average or most common spectrum correlates best the test person. 
However, you should take are about the noise in the measured spectrums
A: This has been extensively studied in linguistics and acoustics. Humans and other primates predict speaker gender through a combination of fundamental frequency $F_0$ ("pitch") and Vocal-Tract-Length estimates ($VTL$) which are a proxy for body size.
Sometimes "formant dispersion" is used for $VTL$. It is usually defined as $$\frac{\sum_{i=1}^n(F_{i+1}-F_i)}{n-1}$$where $F_i$ is the $i$th formant frequency and $n$ is the number of formants measured. However this measure is problematic and does not capture information about midrange formants or about formant positioning. See Masculine voices signal men's threat potential in forager and industrial societies
An alternative $VTL$ measure is 'formant position', defined as:$$\frac{\sum_{i=1}^nF'_i}{n}$$where $F'_i$ is the $i$th formant standardized across the population measured.
However, the usual finding is that a combination of pitch and estimates of vocal tract length give us information about speaker gender and sexual maturity.  Looking at male vs female spectra, on average you'd see male voices lower-pitched and and more closely-spaced formants. 
Acoustic correlates of talker sex and individual talker identity are present in a short vowel segment produced in running speech
Vocal Tract Length Perception and the Evolution of Language
Vocal tract length and formant frequency dispersion correlate
with body size in rhesus macaques, but see Formant frequencies and body size of speaker: a weak relationship in adult humans
