Fourier Ptychography Microscopy Resolution Improvement

Employing Refocusing

Arban Uka

, Gent Imeraj

, Bjorna Qesaraku and Besmir Shehu

Department of Computer Engineering, Epoka University, Tirane, Albania

Keywords: Microscopy, Fourier Ptychography, Refocusing, Microfluidic Chambers.

Abstract: Fourier ptychography is a computational imaging technique that is used to overcome the physical limitations

in determining the spatial resolution of an optical system by combining a large number of low resolution

images. The low resolution images are acquired using programmed illumination from an array of light sources,

thus enabling the scanning of the k-space, which is the reciprocal space of the real domain. The use of this

approach would improve the resolution when biological samples from patients are analyzed in

multiparametric chambers. When an off-center light source is used at an oblique incidence angle, the optical

path length changes thus defocusing the expected image. The common depth of focus of microscopes is a few

micrometers and when a chamber of thickness from 0.5-1.0 cm is used, an adjustment of the focusing is

needed. Here in this work, we report Fourier ptychography using LED illumination and an improved image

quality is acquired when refocusing is implemented.

1 INTRODUCTION

Acquisition of high resolution images is essential in

medical imaging and this is facilitated by an optimal

combination of high-end optical systems and

computational imaging. Costly machineries often

have bench-top design that are sturdy and need the

associating components to adapt to their design. The

development of the experimental instruments has

enabled the measurement of several vital parameters

of patients in the same setup. In this case, in order to

use a small amount of diagnostic specimen (saliva,

blood etc) microfluidic platforms are used

(Chmayssem et al., 2021). To monitor the health state

of cells under different stress conditions microscopy

is employed. To increase the resolution of microscope

intuitive modifications have been applied in the

illumination such as replacing the light source by a

LED array (Zheng et al., 2011) and then later this

LED array enabled illumination is used to scan the

reciprocal space (k-space) of the light propagation.

This technique has led to the development of Fourier

ptychography that enables the increase of the

resolution of the acquired images in both phase and

amplitude (Zheng et al., 2013). This technique can

https://orcid.org/0000-0003-0037-0207

https://orcid.org/0000-0002-7877-3906

overcome the physical limitations of the microscope

as it can increase the depth of focus, thus enabling a

broader range of focused sample. Tian et al. (2014)

implemented multiplexed illumination to reduce the

number of images to be acquired thus optimizing the

runtime of the experiment and the time needed to run

the reconstruction algorithms when acquiring the

final image. This latter contribution would prove

valuable in case when one is using microfluidic

platforms and monitoring is conducted by

microscopes. If a biological sample is undergoing

some change because of some induced stress, then

one would have to acquire all the images and then

analyse them in a short time. The work reported so far

in the literature has been the proof of principle and is

applied on bare samples that are attached on

microscope glass slides. Even when applied on

simple glass slides in a multiplexed mode, some of

the images could diverge out of focus as the depth of

focus is limited. This happens as the optical path

length in air surrounding the sample (on the side of

the incoming light illumination and on the side of the

outgoing modified light past the sample) is different

when different LEDs are turned on. In this case the

optical path length is comparable to the geometric

Uka, A., Imeraj, G., Qesaraku, B. and Shehu, B.

Fourier Ptychography Microscopy Resolution Improvement Employing Refocusing.

DOI: 10.5220/0010915500003123

In Proceedings of the 15th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2022) - Volume 1: BIODEVICES, pages 191-195

ISBN: 978-989-758-552-4; ISSN: 2184-4305

191

path length. When microfluidic chambers are used,

their larger than one index of refraction (comparing

to air) leads to an increase of the difference between

the optical path length and the geometric path length.

Considering that a single LED source results in a

normal incidence to the sample, all the other

secondary LED illumination sources have an oblique

incident angle and this brings to the attention the need

to refocus the sample. At the same time one may

consider that the microfluidic chambers do not have a

perfect width across the volume as few micrometers

of difference in thickness may be inevitable. This

constitutes another source of the undesired

defocusing. At the same time, complex fluids that are

used in the microfluidic channels may exhibit varying

index of refraction as a result of mixing with different

densities of biological samples and this will then

change the optical path length. All these arguments

reminds one on the need of careful refocusing before

each image acquisition. Here in this work we compare

the performance of Fourier ptychography

reconstruction as the focus control is implemented

each time we acquire an image. We observe that a

refocusing improves the quality of the reconstructed

image.

2 THEORY AND

EXPERIMENTAL DESIGN

The main limitation in medical imaging using optical

microscopy is the spatial resolution. Under the visible

light spectrum, it is not possible to observe images of

objects smaller than half of the wavelength of the

incident light source (0.4 to 0.7 μm). Furthermore, in

live cell examination, unsuccessful results may be

obtained because often contrast is low. Staining with

selective dyes, which are used to increase the

contrast, may modify the sample and introduce other

structural features that are not present in the

specimen. With all the associating challenges, there is

a continuous increase in the number of patients being

examined using microscopic techniques rather than

other diagnosis methods, as they are consistently

accurate and of low cost with the latter being essential

in resource-limited settings. An increase in the

demand for digital microscopes in the current

coronavirus pandemic has been observed. Some of

the greatest innovators that operate in the market of

digital portable microscopes include Carl Zeiss AG,

Olympus Corporation, Keyence Corporation, Nikon

Corporation, Leica Microsystems etc. These

benchtop commercial microscopes can be used to

implement Fourier ptychography.

2.1 Theory and Mathematical

Apparatus

Fourier ptychography is an excellent example of how

one can apply algorithms in both the real (x-space)

and the reciprocal domains (k-space) to improve the

spatial resolution (micrometer scale). The

information in the reciprocal domain (with the unit of

𝑚



) is indirectly collected as the k vector - denoting

the direction of the light illumination to the camera

sensor – can be selected for each image by turning on

LED one by one. It uses objective lens with small

numerical aperture and is able to increase the field of

view. This method requires no manual scanning of the

specimen plane, since a LED array is used as an

illumination source. So instead of using a narrow

beam of light to illuminate the sample, different angle

illuminations are provided by programming the LED

array to turn on individual LEDs without having to

move any part of the physical system.

2.2 Experimental Design

The major construct is the combination of the

programmable LED array, sample, light gathering

system, and the CMOS camera. As mentioned earlier

LED array is placed sufficiently below the sample so

that the illumination is considered to be spatially

coherent. In order to achieve this, the sample must be

lifted at a fixed distance of few centimeters above the

fixed LED matrix. This is done by means of metric

adapter plates fixed on the metric breadboard using a

90° metric adapter that glues them together. 32x32

LED array will be used with a spacing of 4 mm

between neighboring LEDs. This spacing is

convenient to cover the sample area and to clearly

define the changes and shifts in each data

measurement. The structure of the experimental setup

is shown in Figure 1. A magnifying lens (model

number 378-805-3) with a relatively small numerical

aperture is used with. It is a 50x magnifying lens with

infinity correction.

This means that the image is directly passed to the

camera sensor without being diffracted and modified

inside the path from objective to the sensor of the

camera. In order to perform the bright field refocusing

that will correct for geometrical shifts and phase

aberrations that degrade the resolution of the final

image, the focus will be adjusted manually as

BIODEVICES 2022 - 15th International Conference on Biomedical Electronics and Devices

192

Figure 1: Experimental setup used in this work

proposed by using a step-controlled motor with high

precision. The camera together with the lens is

mounted to the Zaber motor (Zaber Technologies Inc.

with a resolution of 0.0476 m) that is responsible for

controlling the up and down movements of the

sample each time the illumination angle hence the

change of focus occurs.

Figure 2: Microfluidic chamber material with width of 5

mm. a) Front view of the chamber enclosing USAF 1951 b)

side view of the microfluidic chamber.

Its own software controls this motor while the LEDs

are turned on sequentially by using a raspberry pi and

a python script that identifies how the LEDs are

turned on and the time slots for each of them being

turned on. To acquire the images a CMOS sensor

camera is used. It imposes a lot of advantages

compared to CCD cameras where the most important

one was the faster data acquisition rate. These

cameras have lower cost compared to other cameras

that may be used in modern digital microscopes. The

USAF 1951 that facilitates a quantitative analysis is

enclosed inside a double-sided microfluidic chamber

as shown in Figure 2. The use of the chamber

increased the working distance by 1 mm.

3 RESULTS

The output of the algorithm is a reconstructed image

with higher resolution, one image per each dataset

composed of 25 images. The details become more

significant at the end of the preset number of

iterations. The image with and without focus for each

dataset are compared with each other by means of

algorithms that calculate the contrast and the

sharpness of each output. Beginning with the first and

second dataset ‘5x5 LEDs with USAF 1951 with and

without refocusing’, the synthesized numerical

aperture of Fourier Ptychography is 0.9363 and the

reconstructed image is displayed below in Figure 3

together with the central raw image with maximum

supposed intensity.

Figure 3: a) Original image captured with central LED, b)

Reconstructed image without applying focus adjustment c)

Reconstructed image with focus adjustment.

Computational power is used to determine the

image quality of both reconstructed images with the

refocus method and without. The measured

parameters are Brightness, Contrast, Sharpness, PIQE

(Perception based Image Quality Evaluator) and

NIQE (Natural Image Quality Evaluator). The

respective values are displayed in Table 1:

Table 1: Image quality measurement parameters.

Brightness Sharpness PIQE NIQE

Original Image 127.90 3.39 6.18 3.71

Rec. Img NR 102.56 8.04 26.83 6.19

Rec. Img WR 86.60 7.45 26.15 5.95

The other datasets produced similar results, but

since they were turned into grayscale during the

reconstruction process, it was more difficult to

observe the differences caused by effect of refocusing

the image. Another important aspect of FPM is the

effect of thickness, or the consideration of the third

dimension of the sample. Since Fourier algorithm

takes into account the slightest detail of the image, the

angular illumination will cause some aberrations to

the obtained data. The effect of such thickness was

discussed in Chapter 4, and the experiment results are

also analyzed. Theoretically the glass specimen

Fourier Ptychography Microscopy Resolution Improvement Employing Refocusing

193

imposed an error to the data acquisition process of

nearly 1 percent, which can or cannot be negligible.

This depends in the application of FPM which is

about slightest details in minute length scale. In case

of micro fluidic chamber of thickness nearly 4 mm

per each side, the error was very significant when

compared to the same sample without the chambers.

The thickness effect completely distorted the signal

of the image, which was reconstructed with a not very

pleasant resolution. So, this must also be taken into

consideration when using different types of samples

with varying thickness.

Figure 4: Reconstructed image a) without refocus, b) with

refocus.

4 CONCLUSIONS

In this work we proposed a different approach to the

increment of reconstructed image resolution, by

adjusting the focus manually using a geometrical

approach to solve the problem. The relevant concepts

to the theoretical approach of the proposed solution

were also explained. Theoretically the work proves to

be successful, relying also in the computational

power to resolve the adjustment of image resolution.

When estimating the performance of an experiment

that is conducted for the first time, there may be a lot

of space for improvement. However, assuming the

aberrations arising from moderate conditions in

which the experiment was realized, the results proved

the point even though not much significant

improvement was made in the resulting images. In the

reconstruction process, much less raw images are

used, preventing the computer from drowning in

unnecessary data of low to none important

information of the phase and amplitude (including

darkfield images as well). Reducing the number of

images to 25 was not based on any logical fact rather

than experience in image reconstruction. The

illuminating source was enough to produce

electromagnetic waves with oblique incidence,

however in practice illumination is not coherent,

which leads to images having reduced spatial

coherence. The NA was increased from the synthetic

NA produced in the algorithm of FP and the image

resolution was increased up to 4 times. This could be

further improved by increasing the number of

captured images which comes with expense in the

computational run time and large number of iterations

required.

The effect of refocusing became much clear

during the data analysis process, where the refocused

raw data managed to produce an image of a higher

quality and resolution. The manual adjustment was

made step by step in order to carefully observe the

change in the shape and 3D effect of each captured

image. Even though it took a few minutes of

adjustment for 25 images, the pattern of motion can

be translated into an algorithm and run by the Zaber

Console Software in order to perform the active

refocusing in an automated way. Another important

aspect that was discussed in the fourth chapter was

the thickness of the sample. According to the

theoretical analysis, it imposed no notable error for

the glass sample. However, this was not the case for

the practical aspect, where the resolution was

decreased significantly.

The purpose of this study was to introduce a

different path towards the automatization of the

microscope with the help of Artificial Intelligence.

The corrections that are made physically before the

data acquisition reduce the computational time by

performing just as well, not even better. Active

focusing is in its early stages and still requires the

expertise of biologists and scientists that can train the

AI in the appropriate way.

ACKNOWLEDGEMENTS

This project has received funding from the European

Union’s Horizon 2020 research and innovation

program under grant agreement No 760921

(PANBioRA).

REFERENCES

Chmayssem, A., Verplanck, N., Tanase, C. E., Costa, G.,

Monsalve-Grijalba, K., Amigues, S., ... & Mailley, P.

(2021). Development of a multiparametric (bio) sensing

platform for continuous monitoring of stress

metabolites. Talanta, 229, 122275.

Zheng, G., Kolner, C., & Yang, C. (2011). Microscopy

refocusing and dark-field imaging by using a simple

LED array. Optics letters, 36(20), 3987-3989.

BIODEVICES 2022 - 15th International Conference on Biomedical Electronics and Devices

194

Zheng, G., Horstmeyer, R., & Yang, C. (2013). Wide-field,

high-resolution Fourier ptychographic microscopy.

Nature photonics, 7(9), 739-745.

Tian, L., Li, X., Ramchandran, K., & Waller, L. (2014).

Multiplexed coded illumination for Fourier Ptycho-

graphy with an LED array microscope. Biomedical

optics express, 5(7), 2376-2389.

Kellman, M., Bostan, E., Chen, M., & Waller, L. (2019,

May). Data-driven design for fourier ptychographic

microscopy. In 2019 IEEE International Conference on

Computational Photography (ICCP) (pp. 1-8). IEEE.

Konda, P. C., Taylor, J. M., & Harvey, A. R. (2015,

September). High-resolution microscopy with low-

resolution objectives: correcting phase aberrations in

Fourier ptychography. In Optical Systems Design 2015:

Computational Optics (Vol. 9630, p. 96300X).

International Society for Optics and Photonics.

Lee, D., Ryu, S., Kim, U., Jung, D., & Joo, C. (2015).

Color-coded LED microscopy for multi-contrast and

quantitative phase-gradient imaging. Biomedical optics

express, 6(12), 4912-4922.

Zuo, C., Sun, J., & Chen, Q. (2016). Adaptive step-size

strategy for noise-robust Fourier ptychographic

microscopy. Optics express, 24(18), 20724-20744.

Zheng, G., Horstmeyer, R., & Yang, C. (2013). Wide-field,

high-resolution Fourier ptychographic microscopy.

Nature photonics, 7(9), 739-745.

Tian, L., & Waller, L. (2015). 3D intensity and phase

imaging from light field measurements in an LED array

microscope. optica, 2(2), 104-111.

Huang, W., Pan, S., Zhou, Q., Liao, M., Zhang, C., Tang,

Q… & Peng, X. (2020, October). Positional

misalignment correction for Fourier ptychographic

microscopy based on intensity distribution. In

Advanced Optical Imaging Technologies III (Vol.

11549, p. 115490D). International Society for Optics

and Photonics.

Bian, L., Suo, J., Dai, Q., & Chen, F. (2017). Fourier

ptychography for high space-bandwidth product

microscopy. Advanced Optical Technologies, 6(6),

449-457.

Claveau, R., Manescu, P., Elmi, M., Pawar, V., Shaw, M.,

& Fernandez-Reyes, D. (2020). Digital refocusing and

extended depth of field reconstruction in Fourier

ptychographic microscopy. Biomedical optics express,

11(1), 215-226.

Dong, S., Horstmeyer, R., Shiradkar, R., Guo, K., Ou, X.,

Bian, Z., ... & Zheng, G. (2014). Aperture-scanning

Fourier ptychography for 3D refocusing and super-

resolution macroscopic imaging. Optics express,

22(11), 13586-13599.

Konda, P. C., Loetgering, L., Zhou, K. C., Xu, S., Harvey,

A. R., & Horstmeyer, R. (2020). Fourier ptychography:

current applications and future promises. Optics

express, 28(7), 9603-9630.

Zheng, G., Kolner, C., & Yang, C. (2011). Microscopy

refocusing and dark-field imaging by using a simple

LED array. Optics letters, 36(20), 3987-3989.

Dong, S., Liao, J., Guo, K., Bian, L., Suo, J., & Zheng, G.

(2015). Resolution doubling with a reduced number of

image acquisitions. Biomedical optics express, 6(8),

2946-2952.

Dai, X., Konda, P. C., Xu, S., & Horstmeyer, R. (2021,

March). Polarization and phase imaging using an LED

array microscope. In Polarized Light and Optical

Angular Momentum for Biomedical Diagnostics (Vol.

11646, p. 116460U). International Society for Optics

and Photonics.

Kellman, M., Chen, M., Phillips, Z. F., Lustig, M., &

Waller, L. (2018). Motion-resolved quantitative phase

imaging. Biomedical optics express, 9(11), 5456-5466.

Bian, L., Suo, J., Situ, G., Zheng, G., Chen, F., & Dai, Q.

(2014). Content adaptive illumination for Fourier

ptychography. Optics letters, 39(23), 6648-6651.

Williams, A., Chung, J., Yang, C., & Cote, R. J. (2017).

Fourier ptychographic microscopy for rapid, high-

resolution imaging of circulating tumor cells enriched

by microfiltration. In Circulating Tumor Cells (pp. 107-

117). Humana Press, New York, NY.

Fourier Ptychography Microscopy Resolution Improvement Employing Refocusing

195