# Could a computer unblur the image from an out of focus microscope?

Basically I'm wondering what is the nature of an out of focus image. Is it randomized information? Could the blur be undone by some algorithm?

• I think this should probably be on cross-validated. But I think the answer is 'no, but if you know the characteristics of the blurring process (which is the physics bit) you can do some clever tricks to make the image seem less bad'. – tfb Aug 2 '17 at 7:50
• I'm less interested in the algorithm part and more interested in the physics bit, I only used the computery wording I did because I couldn't think of a more physicsy way to word things. Optics has always confounded and confused me. – user273872 Aug 2 '17 at 8:12
• When the Hubble telescope was first launched and they found that the mirror was flawed, they did exactly this for a year or two until corrective optics could be installed. – Paul Sinclair Aug 2 '17 at 16:36
• Consider that an image blurred due to simple poor focus is like an electronic signal sent through a "low pass" filter. It is possible to reverse this (to a degree) by running the signal through the corresponding "high pass" filter. This will, of course, always be imperfect, and it has the nasty effect of amplifying any noise in the signal. – Hot Licks Aug 3 '17 at 2:17
• The fact that a question would be on-topic on another site doesn't make it off-topic here. Not even if it would be better on some other site. The question can be put into physics terms by asking about light contributions at places other than the focal plane. There is also a nice connection to experimental analysis of data near the bounds of limited acceptance when you have re-distributing effects that are conservative only over all phase space (something that came up explicitly in my dissertation work, for instance). – dmckee Aug 3 '17 at 22:09

The blurring is not randomised, it is predictable. See Can someone please explain what happens on microscopic scale when an image becomes unfocused on a screen from a projector lens? for a basic explanation.

Each point of the in-focus image is spread out into a diffraction pattern of rings called a point spread function (PSF), and these ring patterns overlap to form the out-of-focus image. The blurred image is the convolution of the object and the PSF.

Convolution is a mathematical transformation which can in some circumstances be reversed (deconvolution) - for example when the image has been made using coherent light (from a laser) and the PSF is known. When photos are taken using ordinary incoherent light, and the PSF is unknown, the blurring cannot be reversed completely, but a significant improvement can be made, eg using the blind deconvolution algorithm. Examples of objects and resulting images can be used to approximately re-construct the PSF, or a Gaussian function can be used.

Blurring due to motion (of the camera or object) can also be corrected. For both cases the techniques and problems are discussed in Restoration of De-Focussed and Blurred Images, and examples given of what can be achieved.

Software is available online to fix blurred images.

• Is the blurring from incorrect focus actually a planar PSF? I expected it to be more complicated. – R.. Aug 2 '17 at 18:37
• @R to my understanding, yes it is. Actually it is a set of parallel PSFs to explain the entire 3D field. You can unblur a specific plane assuming you have a table of the predictable ray paths for that plane. – Octopus Aug 2 '17 at 20:18
• Great link. I didn't realize you can use a Weiner filter to unblur an image. I use it to smooth out position value from motion sensors. I guess the two applications are similar: to figure out the "real" value from a noisy input. – slebetman Aug 2 '17 at 22:59
• It would make the unblurring process much easier if camera image sensors could capture phase information, but I guess that's not so easy with incoherent light. :D – PM 2Ring Aug 3 '17 at 13:59
• This leads to another question: given a well-focussed microscope image, could one apply some sort of image enhancement to extract more detail? – jamesqf Aug 3 '17 at 17:50

Yes, it's called deconvolution. Here are some examples of deconvolved images from microscopes:

I found these by Googling for "example of deconvolved image from a microscope".

It's possible by shooting lasers through the lens and generating a map of ray data that can be used in the deconvolution algorithm. I have also seen this done with an in focus image and deconvolved to bring a completely different plane into focus.

• Finally, a fellow cephalopod! Greetings! – DanielSank Aug 2 '17 at 20:22

Yes, it can be (partially) undone, because the process is not random and only part of the information is lost.

Physics

You comment that you are interested in the physics aspect of the question, so let's first clear that up: an image is focused when every point of the object corresponds to a point of the image - all the rays emanating from a given point of the object, passing, e.g., through the lens, converge to a single point (of the sensor, be it your eyes, a CCD, or a film). An image is out of focus when that doesn't happen: when these rays are spread and each point of the image receives light from different points of the object. If you know exactly the blur took place, you can try to reverse it.

That's better explained in this answer to the question Why image is blurry or focused with convex lenses?.

Algorithms

Algorithms for blur removal must work with hypotheses on how exactly the blur took place, i.e., which function better describe it, or the blurring process model. Restoration algorithms can be found in many image processing apps, but that's still an active area of research. You can read about it here and here, among many other sources. Notice, however, that restoration is only partial e artifacts seem to be always present in the restored images.

Purposeful obscuration

If pixelation is applied, though, the image is going through coarse graining, so information is irreversibly lost, and the same is true for other forms of blurring, such as averaging and addition of noise to the image: then in general an algorithm cannot recover information from the picture, because the information is not there anymore.

But, an algorithm can guess. You can train an artificial neural network for recognizing whatever you expect or want to find in the picture, and allow it to fill in the details, as explained here and here.

• Just to note, that some of those guesses can be hillariously wrong (search for google deepmind images for intentional changes like this) or surprisingly accurate. Or both. It can be hard to tell sometimes. – Baldrickk Aug 2 '17 at 15:37
• One of the more interesting ways to "guess" is to use compressive (aka compressed) sensing. When the assumptions are met, which is to say the original input is sparse in the relevant domain, the results are often unbelievable. The first hit I got googling "compressive sensing deblur" illustrates this in a non-blind case. There are also blind deconvolution approaches such as this with a familiar image. – Derek Elkins Aug 3 '17 at 5:17
• This answer contains some particularly important points. The physical nature of the blurring process can often make information (essentially) impossible to recover. Noise and pixellation are two examples, so would be loss of the illumination edges for the images, making deconvolution impossible. – uhoh Aug 4 '17 at 8:44

The goal with a camera lens system (whether a microscope or not) is to deliver light from one point on the object to one point on the sensor. This however cannot be perfectly achieved for several reasons.

1. Light will diffract off the apeture, the smaller the apeture the more the diffraction.
2. The system will only be perfectly in focus for an infinitely thin plane. The larger the apeture the faster the focus will drop off with distance from that plane.
3. The lenses themselves will be imperfect.

All of these effects act to attenuate high (spacial) frequency components in the image. We call this blurring.

To reverse blurring we must first characterise the blurring. Characterising the blurring requires making some assumptions about what is in the image and/or the characteristics of the optical elements causing the blurring.

Once we have a mathematical function discribing the blurring process we can calculate an inverse of that function and apply it to the image.

However there are limits to how much we can mitigate the impact of blurring. Firstly because our mathematical model of the blurring is only an approximation and secondly because the inversion process inevitablly amplifies noise in the image.

I've work on exactly this problem and the answer is extremely controversial!

At the heart of the controversy is a disconnect between tomography (3D imaging) and the surface maps (2.5D, "we all live in a hologram") approaches to image formation. The "we all live in a hologram" approach has recently seen a lot of commercial/experimental success much to the chagrin of the tomography old timers.

Different optical systems have drastically different abilities to re-focus the image. Further, each system operates on different physical assumptions. Lets talk about a few:

Light-field systems such as the Lytro can produce selective focus for macroscopic objects, on the other hand similar affects can be achieved with digital filters.

At low NA (>0.3), coherent (laser) illumination matches well with the linear systems approach to wave propagation in a half-space given by the Fresnel propagator. You can apply the Fresnel propagator and focus and de-focus the system, although you need some kind of measure of the complex field. Popular approaches include Hartmann-Shack style wave front sensors (similar to Lytro), traditional interferometry, and transport of intensity (TIE). The images in these systems look pretty terrible because junk somewhere in the beam path (perhaps a few cm away) contributes to image formation.

Now stuff starts getting wacky when coherence effects are introduced. On the other hand the image looks better under incoherent illumination. The first problem is that even for systems that are linear with the object's scattering potential (not many!), coherence introduces a spatial band limit. That is to say objects outside of the spatio-temporal coherence (you can't actually split these) radius fade away. So, now your system is only sensitive to a narrow range of object sizes, and you start getting optical sectioning. This is typically called the "coherence gating" and was famously used by OCT. In practice, you can't propagate the field beyond whats allowed by the coherence. In most broadband systems this means you can't, digitally, move the field up-and-down by more than a few microns! (and it gets worse at high NAs)

On the other hand you can simply move the sample (or adjust the focus). When you do this a new degree of freedom is introduced, which is the origin of tomographic imaging (diffraction tomography). For people who use these kind of systems their world is made of 3D slices. They would tell you that one cannot simply refocus the light indefinitely.

Another problem is that many systems are not actually linear with the object, for example, the infamous halo in phase contrast microscopy. In those cases you can't meaningfully digitally focus/refocus the sample because your system typically doesn't measure the field due to the sample.

On the theoretical side there is a lot of debate as to if fields are random, but the positions taken by various research groups are closely tied to their experimental setups.

• Could you add some references? – stafusa Aug 4 '17 at 8:53
• I think this answer introduces too many technicalities, and is not appropriate here. It should be rewritten to be accessible to a general audience. – AlQuemist Aug 9 '17 at 7:43

In this article the limits of the details that can be recovered using deconvolution are derived. It's explained that noise leads to limits on how effective deconvolution can be to recover details. In the ideal case there will only be Poisson noise due to the finite number of detected photons. The smallest recovarable details scale as $N^{-\dfrac{1}{8}}$. So, getting ten times more details requires 100 million times more exposure time. Clearly, a wide aperture without a lens and attempting to focus using deconvolution, is not going to work in practice.

• See academic.oup.com/mnrasl/article-lookup/doi/10.1093/mnrasl/… Machine Learning techniques have improved on the deconvolution limit – gokul_uf Aug 4 '17 at 22:01
• In the vein of gokul_uf's comment, these kinds of "limits" are often based on assumptions that any original (uncorrupted) image is possible and equally likely. On the one hand, this avoids a "seeing what we expect to see" bias, but on the other hand it is completely unreasonable. If I slipped an astronomical image into your family picnic photo album or vice versa, there's absolutely no way you'd be even briefly uncertain as to whether it belonged. – Derek Elkins Aug 6 '17 at 1:38

## protected by Qmechanic♦Aug 2 '17 at 12:55

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).