# Does Bayer demosaicing from RGB-filter sensors work *because* the color filters are imprecise?

I have an ongoing friendly dispute with another member over on Photo Stack Exchange about the fundamentals of how RAW works, and I'm hoping you can settle it.

My understanding is that Bayer demosaicing works basically entirely on the assumption that one can deduce likely color information from monochromatic pixels. That is, for a pixel with a blue filter, you can assume that the correct green value is close to the average of neighboring green-filtered pixels, and the red value close to the average of neighboring red-filtered pixels — even if those filters were to be theoretically perfect.

My site-colleague argues that, as shown in this graph:

the filters have a lot of overlap, and describes the demosiacing algorithms as working because they recover that overlap.

Who is correct here? Would Bayer demosaicing work if the filters had no overlap?

(Bonus questions: would this be an improvement, or actually a downside? Presumably you'd be letting in less light overall; would you get more accurate color for that price? I know that simple averaging "works" for demosiacing; are there more complicated algorithms which do take overlap into effect explicitly?)

• I have no idea why this was downvoted. Is this not topical (with a site like biology more relevant because of the human-vision component)? If it is on topic but there is some other problem, please help me improve it. – mattdm Jun 4 '17 at 15:31
• Given that the early figures in understanding colour vision were Newton and then Maxwell (who produced the 1st colour photograph) I think it is entirely justified in being a physics question! I'd also add that 1. I tried to answer below and 2. I'm a physicist – Euan Smith Jul 3 '17 at 15:47

The demosaicing algorithm which you describe (there are many more complex algorithms, but sticking with the simplest for the moment) is a linear operation - the resultant RGB values for each pixel are a linear combination of the raw pixel values.

To subsiquently correct the demosaiced pixels to reflect a colour primary standard (sRGB or rec709) is also, most commonly, a linear operation.

To start off with RGB filters which are purer, that is which better reflect a colour standard, would be equivalent to being able to perform the colour correction step first. As both of these steps are linear then it should make no difference in which order you perform them therefore there would be no difference in the output image.

So IF the demosaicing is linear and IF you will be colour correcting to the same standard in the output THEN there will be no difference.

In practice (we work with RAW data and use custom non-linear demosaicing and substantially linear colour correction) this also seems to more generally hold true for less linear steps. You do get a smoother result due to pixel correlation in the RAW image but the colour correction step then magnifies those differences so that additional correlation then disappears.

BTW the Bayer demosaicing algorithms generally rely on the fact that hue variations are 'slower' in an image than lightness variations (and indeed less visible due to our own image processing) so there will be a longer-range correlation between the relative R, G and B values. So in other words they are not relying on spectral correlations between R, G and B, rather they rely on the R, G and B signals to correlate in their reflections from a particuar object or surface.

• So, as I understand it, your "BTW" at the end is the answer to my question, right? – mattdm Jun 4 '17 at 15:32
• The short answer is that I think that demosaicking would work just fine even if there was no overlap - it relies on the fact that spectral reflection off real objects has broad features, not that the RGB filters have broad features. The long answer was an attempt to justify that by explaining that the choice of RGB filter is pretty arbitrary. – Euan Smith Jul 3 '17 at 15:39