The Method for Generating High Resolution Images in Low Light

Conditions

Junfeng Shi

, Qinghua Zhu, Youlan Wu, Zhifang Cai and Dongqin Hu

The School of Electronic and Information Engineering, Beijing Polytechnic, Beijing, China

Keywords: Infrared Image, Image Fusion, Electro-Optic Image.

Abstract: This method through fusion of edges detected in an electro-optic image with a corresponding transformed

infrared image and the original infrared image. Alternatively, the method generates high resolution images in

low light conditions when an electro-optic image is not available by edge detection of an infrared image,

transformation of the infrared image and fusion of the transformed infrared image with the edges detected in

the infrared image. The method is particularly useful for night vision applications.

1 INTRODUCTION

Electro-optic ("EO") imaging techniques are capable

of achieving high-resolution images when the object

to be imaged is illuminated by adequate light of an

appropriate wavelength, as during the day. Electro-

optic imaging devices are less useful at night or in

other low light conditions. Infrared ("IR") imaging

technology is dependent upon heat emitted by the ob-

ject to be imaged and is not dependent on incident

light reflected by the object. Infrared imaging tech-

niques hence can be used in conditions of complete

darkness. Infrared imaging is inherently of lower res-

olution that electro-optic imaging and suffers from

noise in the signal. Infrared imaging is also poor at

discriminating between object having a similar tem-

perature. Night vision systems are known. For exam-

ple, it is shown that a night vision system in which

edge detection is applied to an infrared image and the

detected edges fused with an image-intensified visi-

ble light image (U.S. Patent 7,864,432 to Ottney is-

sued 2011). See literature (U.S. Patent 7,864,432 to

Ottney issued 2007, U.S. Patent 7,864,432 to Ottney

issued 2003) for the same disclosure and teaches a

head worn night vision system include image intensi-

fiers operating in the near infrared. In the literature,

see "Cognitive Image Fusion and Assessment," Alex-

ander Teot, chapter of Image Fusion, edited by

Osamu Ukimura (Ruzic, Pizurica 2012). See also

"Experimental Tests of Image Fusion for Night Vi-

sion," Y. Chen and R. Blum, available (huang,

Netravali 2012, Huang, Man, Lawrence 2012). Cop-

ies of the Teot and Chen articles are attached to this

provisional application and incorporated by reference

herein. Using current imaging techniques, it is not

possible to distinguish the edges of objects in low

light conditions, such as at night, where the objects

have a similar temperature and a similar background,

or tiny parts of any objects.

2 THE RESEARCH METHOD

2.1 First Innovative Method

The first method involves the following steps: (a)

capturing an electro-optic image of objects; (b) pro-

cessing the electro-optic image by applying edge de-

tection technology to detect the edges within the elec-

tro-optic image; (c) collecting an infrared image of

objects corresponding to the electro-optic image; (d)

transforming the infrared image by applying a Wiener

filter or an inverse filter based on the point spread

function of the thermal imaging method; (e) register-

ing the infrared and electro-optic images to match

features of one image with the other; and (f) fusing

the transformed infrared image and the detected edges

of the electro-optic image to create a composite im-

age. The composite image features better resolution

among objects than the original infrared image.

572

Shi, J., Zhu, Q., Wu, Y., Cai, Z. and Hu, D.

The Method for Generating High Resolution Images in Low Light Conditions.

DOI: 10.5220/0011752200003607

In Proceedings of the 1st International Conference on Public Management, Digital Economy and Internet Technology (ICPDI 2022), pages 572-577

ISBN: 978-989-758-620-0

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

The step of capturing the electro-optic image may

occur at a time different from the time that the infra-

red image is captured. For example, the electro-optic

image may be captured during daylight hours so that

the object to be imaged is adequately illuminated to

create a high resolution, low noise image having read-

ily identifiable edges. The corresponding infrared im-

age may be captured during periods of low light when

the electro-optic detector is not effective.

The step of transforming the IR image may occur

in any of several ways, either alone or in combination.

For example, the IR image may be transformed by ap-

plying an inverse filter using a Fourier transform

based on the theoretical point spread function ("PSF")

of the infrared detector. The IR image transformation

may be based on the measured, rather than theoreti-

cal, point spread function for the infrared imaging

system. The inverse filtering process utilizing either

the theoretical or measured point spread function of

the IR imaging system reduces the noise in the image

and makes the transformed infrared image look

“sharper” than the original one. The IR image may be

filtered by applying the Wiener filter as an alternative

to transform the IR image if noise is not neglectable.

The inverse filter is a special case of the more general

Wiener filter.

The step of edge detection of the EO image in-

volves applying an edge detection algorithm to the

EO image. The resulting edge-detected image com-

prises the detected edges. The step of registering the

transformed IR image and the edge-detected EO im-

age may be as simple or as complex as the data re-

quire and involves the identification and matching of

corresponding features on the IR and EO images. The

step of blending the edge-detected EO image and the

transformed IR image involves overlaying the de-

tected edges on the corresponding locations of the

transformed IR image. The blending step may include

blending of the original, un-transformed IR image

with the transformed IR image and the detected

edges.

2.2 B. Second Innovative Method

The method of the method can generate images of im-

proved resolution when only an IR image and no cor-

responding EO image is available. In this second

method, the steps include (a) capturing an IR image,

(b) transforming the IR image by applying a Wiener

filter or an inverse filter using a Fourier transform

based on either a theoretical point spread function or

a measured point spread function of the infrared im-

age, (c) applying an edge detection algorithm to de-

tect the edges in the IR image, and (d) blending the

edge-detected IR image, the original IR image and the

transformed IR image to form a fused IR image.

3 METHOD DESCRIPTION

Figs. 1 and 2 are schematic diagrams illustrating the

first method of the method. Fig. 1 shows the flow of

information in the first method while Fig. 2 shows the

method of the first method. As shown by Figs. 1 and

2, an EO sensor captures an EO image. An IR sensor

captures an IO image, either at the same or at a differ-

ent time from the capture of the EO image. The IR

and EO images are registered to match features of the

IR and EO images for use in blending the processed

images. The EO image is analyzed using an edge de-

tection algorithm to detect differences in hue, color or

intensity that may indicate an edge of an object. The

result of the edge detection is an edge-detected EO

image comprising the detected edges. The other in-

formation in the original image generally is omitted

in the edge-detected EO image. As indicated by Fig.

7, the edgedetected EO image may be further pro-

cessed by applying a small-size low pass filter to the

edgedetected image. The IR image is transformed us-

ing either a Wiener filter or an inverse filter based on

the point spread function of the IR sensor. The inverse

filter is a particular application of the Wiener filter

and comprises transforming the IR image using the

point spread function of the IR sensor. The selection

of either the Wiener filter or the inverse filter may de-

pend upon the noise level in the original IR image. If

the original IR image has a high noise level, then the

Wiener filter will be adopted to reduce that noise

level. If the original IR image has little or no noise,

then the inverse filter is the filter of choice. The

method of the method may be configured to select the

appropriate filter based on the noise level of the orig-

inal IR image. The point spread function of the IR

sensor applied in either the Wiener filter or the in-

verse filter may be either a theoretical point spread

function or a point spread function determined by

measurement. As an optional step, the transformed IR

image and the edge-detected image may be registered

to match the detected edges in the edge-detected im-

age to the edges shown by the transformed IR image.

The edge-detected EO image, the transformed IR

image, and the original, un-transformed IR image

The Method for Generating High Resolution Images in Low Light Conditions

573

Figure 1: Flow of information(self-created).

Figure 2: Flow chart(self-created).

then are blended to define a fused image. The result-

ing fused image demonstrates a better resolution

among objects shown by the images than either the IR

image or the EO image alone. For further explanation,

see the attachments.

Figs. 3-5 illustrate the second method of the

method. The second method applies when an IR im-

age, but not an EO image, is to be processed. Fig. 3

illustrates the information flow of the second method.

Fig. 4 illustrates the method of the method. From

Figs. 3 and 4, an IR sensor captures an IR image. The

IR image is processed along two different paths and

the results of the processing are blended to form a

fused image. In the first method, an edge detection al-

gorithm is applied to the IR image to determine an

edge-detected image. The edge-detected image com-

prises the detected edges and generally does not in-

clude the other information in the original IR image.

In the second path, the original IR image is trans-

formed by applying either a Wiener filter based or an

ICPDI 2022 - International Conference on Public Management, Digital Economy and Internet Technology

574

Figure 3: Flow chart(self-created).

Figure 4: Flow chart(self-created).

inverse filter comprising a Fourier transform based on

the point spread function of the IR sensor. The step of

transforming the IR image is conducted as indicated

above for the first method. The transformed IR image

is then blended with the edge-detected IR image and

the original IR image to define a fused output image.

The fused output image demonstrates a better resolu-

tion among objects shown by the images than the IR

image alone. Fig. 5 also relates to the second method

and illustrates that the filtering of the IR image shown

by Figs. 3 and 4 may be based on either a theoretical

point spread function of the IR sensor or may be based

on an actual measured point spread function of the IR

sensor.

Fig. 6 illustrates the method for the first method

of the method. The method comprises a microproces-

sor operably connected to the IR sensor, the EO sen-

sor, a computer memory and a computer display. The

method receives data from the IR sensor defining the

IR image and data from the EO sensor defining the

EO image. The microprocessor is configured to re-

ceive and manipulate the data by programming in the

computer memory. The microprocessor is configured

to perform each of the steps described above or as

The Method for Generating High Resolution Images in Low Light Conditions

575

Figure 5: Flow chart(self-created).

shown in the attachments, resulting in the fused im-

age. The fused image is displayed to the user on a dis-

play. Also from Fig. 6, the method of the method for

the second method of the method has characteristics

similar those shown, except that the microprocessor

does not receive or does not process EO image data

from the EO sensor. The microprocessor is otherwise

configured to perform each of the steps set forth

Figure 6: Flow chart(self-created).

Figure 7: Flow chart(self-created).

above or in the attachments for the second method,

resulting in a fused image. The display is configured

to display the fused image to the user. Fig. 7 illus-

trates an additional filtering step that may be applied

to an edge-detected image. The additional step is

passing the edge-detected image through a small-size

low pass filter for the purpose of smoothing the de-

tected edges. The small-size low pass filter, for exam-

ple a 2 by 2 average filter as known in the art, allows

the detected edges to seamlessly match the objects

shown by the original IR image and transformed IR

image. The low pass filter must be small in size to

prevent loss of the edge information in the edge-de-

tected image. While Fig. 7 illustrates the additional

filtering step as applied in the first method of the

method as shown by Figs. 1 and 2, the technique also

may be applied to the edge-detected IR image in the

second method, shown by Figs. 3

4 CONCLUSION

This paper proposes a method of image fusion, with

combination of infrared image and the original infra-

red image. Through the above analysis, we can obtain

some useful results: the proposed method generates

high resolution images in low light conditions, partic-

ularly, edge detection of an infrared image is not

available for electro-optic image.

In the future study, our works are mainly focus on

multimedia video processing that employ image fu-

sion method proposed in this paper.

ICPDI 2022 - International Conference on Public Management, Digital Economy and Internet Technology

576

REFERENCES

U.S. Patent 7,864,432 to Ottney issued January 4, 2011.

U.S. Patent 7,307,793 to Ottney issued December 11, 2007

U.S. Patent 6,560,029 to Dobbie issued May 6, 2003

"Fusing electro-optic and infrared signals for high resolu-

tion night images," SPIE Image Processing: Algorithms

and Systems X; and Parallel Processing for Imaging

Applications II, Tijana Ruzic; Aleksandra Pizurica:

February 5, 2012.

"Improved fusing infrared and electro-optic signals for

high-resolution night images," Proceedings of SPIE,

Volume 8355 (1), Xiaopeng huang; Ravi Netravali,

May 11, 2012.

Multi-Sensor Fusion of Infrared and Electro-Optic Signals

for High Resolution Night Images, Xiaopeng Huang;

Hong Man;Victor Lawrence, 2012, 12, 10326-10338.

The Method for Generating High Resolution Images in Low Light Conditions

577