Technical explanation of "Hitachi lensless camera"

Question

Hitachi is bragging about a thin lensless camera. I can't find any technical information on how it works---just press releases etc.† Can anyone work backwards from the published descriptions to reconstruct how this works?

The "concentric-circle pattern" looks an awful lot like a fresnel zone plate (a.k.a. diffractive lens), but it's not that simple. They also indicate heavy computer processing (the raw image sensor data is not recognizable AFAICT), and the images suggest that there is little if any chromatic aberration.

Sample online descriptions:

http://image-sensors-world.blogspot.com/2016/11/hitachi-lensless-camera-adjusts-focus.html http://image-sensors-world.blogspot.com/2016/11/more-on-hitach-lensless-camera.html

Sample diagrams:

†(there may be papers or patents, but I haven't found them.)

Answer 1

A fringed pattern can effectively acts as an analog computer that computes a Fourier transform or an analogous related quantity. A second mask (computed) probably implements a filtering of the Fourier transform after which a second Fourier transform is performed. This whole process would result in the emulation of an effective dispersion relation.

I would guess the idea is then to emulate a lense by using this effective dispersion relation to recreate the effect a lense would have.

Answer 2

It seems to me that they use a combination of a Fresnel zone plate and Moire fringes to reconstruct the image. One would need to work through the math, which I don't have time for at the moment, but the idea seems to be something like this. First you multiply the light falling on the sensor array with the Fresnel zone plate. Now, this cannot just be the black and white rings that is shown in the image. It would have to be a phase function, I think. The processing of the sensor data is then done by multiplying it with a shifted version of the Fresnel zone plate. The result would contain, it seems, some linear fringe pattern emerging as a Moire pattern. To extract this linear Moire fringe pattern, one would need to compute the absolute value and then do a low-pass filtering. Then one can Fourier transform the linear fringe to obtain its amplitude and phase (the low-pass filtering can be done after the Fourier transform). This process would probably be done for all possible shifts to obtain all possible linear Moire fringe frequencies and orientation, thus, building up the image. Does this make sense?

Answer 3

It probably started as something like the following kinda-simple idea, although I'm sure there are refinements and details.

We have an absorptive filter with a concentric ring pattern, sitting a certain distance in front of an image sensor.

I'm guessing that the ring pattern is: $$T(x,y) = (1+\sin \phi)/2 \qquad , \qquad \phi = a(x^2 + y^2)$$ where $T$ is transmissivity of the filter at the point $(x,y)$, and $a$ is some constant with units of inverse length squared, whose value does not matter at this level of analysis.

(This is the same as a (parabolic) fresnel zone plate with focal length $f=\pi/(\lambda a)$, but that's just a coincidence! Indeed, I'll assume geometric optics!! Specifically: I'll assume that a light source, after passing through the filter, simply creates a shadow. Of course there is diffraction, but I think it causes distortion, rather than being the intended principle of operation. In practice, this probably requires putting the image sensor much closer to the filter than $f$.)

Now that we're assuming geometric optics, a point light source creates a shadow of the ring pattern on the image sensor. If the light source is at infinity, the shadow is shifted around based on the location of the light source. If the light source is closer than infinity, the shadow is both shifted and enlarged compared to the original ring-pattern filter.

Let's do the infinitely-far-away light source case. Then our image sensor will measure a sinusoidal ring pattern with intensity $$I(x,y) = (1+\sin \phi')/2 \qquad , \qquad \phi' = a((x+\Delta x)^2 + (y + \Delta y)^2)$$ where $\Delta x$ and $\Delta y$ are the direction cosines of the source, times the distance from the filter to the image sensor.

We know $T$ digitally, and we just measured $I$, so therefore we can digitally superimpose them. We'll get Moiré fringes with phase $$\phi'-\phi = (2a)(x,y)\cdot(\Delta x,\Delta y) + C$$ i.e. the Moiré fringes form a nice sinusoid whose 2D wavevector is proportional to the direction cosine of the incoming light. ($C$ is some constant.)

So take the image, multiply it (point-by-point) by $T$, and Fourier transform it, and you'll get an image focused at infinity. Or if you use a magnified version of $T$, you'll get an image focused at the object plane in which a light source would enlarge the shadow pattern by the same amount.

Or something like that, plus surely lots of refinements and details.

Answer 4

From a press release at Hitachi's web site (which you may already have seen, still not too technical):

(1) Image processing technology using Moiré fringes

A film patterned with concentric circles (whose interval narrow toward the edge of the film) is positioned in front of an image sensor, and the image of a shadow formed by a light beam irradiated onto the film is captured by the image sensor. During the image processing, a similar concentric - circle pattern is superimposed on the shadow and Moiré fringes with spacing dependent on the incidence angle of a light beam are formed. By utilizing the Moiré fringes, it is possible to capture images by simple and commonly used image processing called “Fourier transform” *4 (Figure).

(2) Focus adjustment technology of captured images

The focal position can be changed by changing the size of the concentric-circle pattern superimposed on the shadow formed on the image sensor by a light beam irradiated onto the film. By superposing the concentric-circle pattern by image processing after image capturing , the focal position can be adjusted freely.

Answer 5

Interesting, never seen this before.

My best shot would be to look at the point spread function from an object point located at some place in front of the lens.

The 0th order of the image is just the shadow of the mask.

The 1st order interference focuses the object point to some image point behind the mask.

Now I try to imagine what both fields (0th and 1st orders) look like just slightly behind the mask (but still far away from the focus plane). The 0th order is an image of the mask, shifted off-center according to the angle of the original incident light beam. It is slightly magnified (the source was placed at a finite distance from the mask). The 1st order field is also shifted, and slightly reduced (it will converge in the focal point).

If we are neglecting the magnification/reduction of both parts, we basically have two images of the mask, shifted by a tiny, but different amount. As in the figure presented in the question, they will produce a Moiré pattern. By analyzing the wave vector of the pattern, we can reconstruct the original source point. This is conveniently done by a Fourier transform.

Of course some more filtering will be required, but I think that might be the gist of it.