What is the shape of the MTF curve in coherent imaging? For incoherent imaging, the shape of the diffraction-limited MTF curve would look roughly like a triangle, with normalized contrast starting at 1 for zero spatial frequency and decreasing to 0 at the cutoff frequency (2*NA/lambda). The characteristic triangle shape is formed by the auto-correlation of the pupil function, which is often assumed to be a uniformly illuminated circular aperture.
For coherent imaging, I have seen it written that the contrast is 1 for all spatial frequencies below half the cutoff frequency (NA/lambda), and 0 for all spatial frequencies above. In this case, there is no auto-correlation, simply the aperture function.  Instead of MTF it is written as ATF for amplitude transfer function because the complex amplitudes at the image plane will need to be added not the intensities.
But I'm having a little trouble wrapping my head around that one. The illustrations I have seen all seem to assume the pupil mask is located at the fourier plane of the lens, and though it is not stated, I get the feeling that the lens is assumed to be infinite in size.
In a real, physical system, wouldn't the lens aperture vignette some portion of the light? And the smaller the spatial frequency, the greater the angle, and the more light would be lost, thereby reducing contrast in a fairly linear way just like the classic triangle-shaped MTF curve we are all used to?
Definitions:


*

*ATF - amplitude transfer function 

*lambda - wavelength of light (in units of millimeters) 

*MTF - modulation transfer function 

*NA - numerical aperture (equal to sine of the half-angle)

 A: An optical system's MTF assumes/requires the light to be spatially incoherent.  Obviously there doesn't exist a source that is completely spatially incoherent or coherent, but most microscopes use a light bulb of sorts.  A light bulb or any extended source which generates light as a black body radiator, emission due to heating, will be very spatially incoherent (see Speckle Statistics reference below). In this case it is appropriate to quantify an optical system's MTF and intensities will add.  However, if the source is spatially coherent, for instance in the case of a laser, an optical system's MTF doesn't apply since as you said the complex amplitudes add, not the intensities and in this case, it is more appropriate to quantify the system's ATF.  In fact the diffraction limited MTF can be calculated using the ATF, but it is meaningless if the light is coherent.  Chapter 6.1.3-6.5 in the link below, Fourier Optics by Goodman, describes what is called a Mutual Intensity which is a measure of the coherence of the light.  When the mutual intensity is approximated as spatially incoherent, you can mathematically define a system's MTF and find that intensities add linearly.  However, in the more general spatially coherent case, the MTF cannot be determined and it would be more appropriate to calculate the ATF.
That being said I have yet to find many paper where the system ATF is measured and used as a metric for optical performance.  Often in the literature, an MTF is defined and measured even for coherent imaging systems.  How can this be if the MTF doesn't apply?  Well most geometric optic software used to design optical systems use MTF as a metric for the performance of a lens system, so when designing optics, the incoherent behavior is optimized in simulation to reduce aberrations and field issues.  Aberrations always degrade image quality no matter what light is used (see Goodman 6.4), so this is an appropriate means for designing and optimizing an optical system; however, when it comes time to measure these properties in the actual system, there needs to be a way to compare and document actual system performance to the expected simulation; however, the simulation only has an MTF, so researchers only document MTF even when the light is coherent.  As far as how they draw these comparisons, read through the literature for coherent optical system designs and pay close attention to the authors' knowledge of these terms.
As far as vignetting is concerned, all of what I describe requires a diffraction limited system, which means that the stop surface or pupil of the system is the limiting aperture – i.e., all other surfaces are over-sized such that there are no field issues and no vignetting.  Fourier Optics by Goodman assumes diffraction limited performance as well for comparison of these concepts, the difference between MTF and ATF has nothing to do with field issues like vignetting or aberrations.  For example, say an optical wide-field diffraction limited imaging system has an appropriate stop surface in the Fourier plane as you described.  If the illumination is incoherent, the spatial frequencies will taper linearly in accordance with the MTF.  If you instead use a coherent light source with the exact same optical system, you will see fringes and you will be able to calculate the system's ATF as you described.  The spatial frequencies will have contrast of 1 below a cutoff value and 0 outside this cutoff value.
References

*

*Fourier Optics by Goodman


*Statistical Properties of Laser Speckle Patterns by Goodman
