A PROCEDURE FOR AUTOMATED REGISTRATION OF FINE

ART IMAGES IN VISIBLE AND X-RAY SPECTRAL BANDS

Dmitry Murashov

Dorodnicyn Computing Centre of RAS, Vavilov st. 40, 119333, Moscow, Russian Federation

Keywords: Multimodal image registration, X-ray image, Fine art paintings, Point set matching.

Abstract: This paper presents a two-step procedure for automated registration of photographs and roentgenograms of

fine art paintings. Grayscale local maxima in blurred images are used as the control points. The coherent

point drift (CPD) point sets matching algorithm is combined with iterative procedure for excluding false

correspondences. General projective transformation model is used for image registration. The precise step of

the procedure reduces registration error obtained at the coarse step.

1 INTRODUCTION

In this paper, a problem concerned to analysis of

multispectral images of fine art paintings is

considered. Multispectral images are widely used in

the research aimed on restoration and attribution of

paintings. One of the aspects of such a research is

the analysis of information hidden under the visible

paint layer (Kirsh, 2000). The way to analyze the

paintings is to combine images of different

modalities in order to localize an object in IR, UV,

or X-ray image and its corresponding position in the

color image.

For the efficient acquisition of information

hidden under the visible paint layer, it is necessary

to automate operations of image registration,

comparison, and analysis of registered images. For

this purpose, the computer technologies are widely

used (Stork, 2009, Kammerer, 2004, Maitre, 2001,

Heitz, 1990, Martinez, 2002).

The majority of the developed systems provides the

automated operations for registration and analysis

of IR, UV, and visible images. The properties of X-

ray images obstruct the automated registration. The

main goal of this work is to automate registration of

images taken in visible and X-ray spectral bands

(see Figure 1).

The images under research are the JPEG images

of size 2800x4200 and of 8 or 24 bpp depth. The

size and the viewing fields are different and

conditioned by the parameters of X-ray unit and the

restorer’s regions of interest. Visible and X-Ray

images differ in size, viewpoint, viewing field, and

content. In Figure 2, the same fragment in color

photograph and X-ray image is shown.

(a) (b)

Figure 1: Images of the painting obtained in optical (a) and

X-ray (b) spectral bands.

The considered problem is identical to the

conventional problem of image registration, but the

listed above properties of X-ray pictures spoil the

solution.

The problem can be formulated as follows. Let

→(, ):uxy R R

be a model image obtained in X-

ray spectral band and

→(','):vx y R R be a

data image obtained in optical spectral band. It is

necessary to find a transform

→:TR R

minimizing the mean squared error and mapping the

data

(',')vx y into the model image

(')XFX,

(1)

162

Murashov D..

A PROCEDURE FOR AUTOMATED REGISTRATION OF FINE ART IMAGES IN VISIBLE AND X-RAY SPECTRAL BANDS.

DOI: 10.5220/0003374801620167

In Proceedings of the International Conference on Computer Vision Theory and Applications (VISAPP-2011), pages 162-167

ISBN: 978-989-8425-47-8

 2011 SCITEPRESS (Science and Technology Publications, Lda.)

where

=∈(, )

Xxy R and

=∈' ( ', ')

Xxy R

are the vectors of image coordinates. In the next

section conventional approaches to the problem are

considered.

2 RELATED WORKS

A problem of multimodal image registration in fine

arts is analogous to the problems of multimodal

medical or aerial image registration (Maintz, 1998).

Solution of the problem includes the following four

steps (Zitova, 2003): (a) feature detection; (b)

feature matching; (c) transformation model

estimating; (d) image resampling and

transformation.

(a) (b)

Figure 2: The same image object in color photograph (a)

and X-ray (b) spectral bands.

Implementation of each registration step meets

its specific problems.

The features extracted at the first step are

classified with respect to the types of objects

detected in the images: region features, line features,

and point features. The features should be associated

with distinctive objects and should be invariant with

respect to selected transformation model. If the high-

contrast details cannot be find in the image one can

use features calculated from information

characteristics of images. The features are

represented by control points (CP), which are used

for calculating parameters of image transformation.

In (Kammerer, 2004) the control points are selected

manually and adjusted by normalized cross-

correlation-based algorithm. In automated

techniques, Harris corner detector (Harris, 1998) is

widely used ((Schmid, 1997, Delponte, 2006), and

many others). Additional local feature descriptions

such as differential geometry invariants (Schmid,

1997), moment invariants, scale invariant features

(SIFT (Lowe, 1999)) can be used (Delponte, 2006).

In (Cappellini, 2005) an algorithm for registration of

UV and visible images implements the

maximization of the mutual information technique

based on maximizing measure of statistical

dependency of two images. The maximization

process implemented as a heuristic iterative search

procedure in the space of four parameters.

A variety of feature matching techniques is

known. In (Schmid, 1997) an algorithm based on

differential invariant description and CPs spatial

relations defined as the angles between directions to

neighbour points is presented.

Another technique is named the Iteration closest

point registration (ICP) (Chen, 1992). The algorithm

iteratively assigns correspondences utilizing the

nearest neighbour criterion to minimize the sum of

squared distances between the points of the two sets.

The ICP requires a good initial estimate to converge

to a global minimum. The improvements of the ICP

technique are described in (Rusinkiewicz, 2001) and

(Sharp, 2002).

Another group of feature matching techniques

uses the spectral properties of the proximity matrix.

The elements of the matrix represent the degree of

attraction between image features via a Gaussian-

weighted distance metric. In (Scott, 1991) an SVD-

based feature matching technique is proposed. The

technique can work with point sets of different size

but is sensitive to rotation and scaling. In (Shapiro,

1992) an eigenvector approach to the problem of

feature matching is presented. Correspondences are

established by comparing the ordered eigenvectors

of the proximity matrices of different images. This

method shows the best results with point sets of the

same size. A variety of the spectral-based techniques

improving (Scott, 1991) and (Shapiro1992) were

developed (Pilu, 1997, Zhao, 2004). In (Myronenko,

2010), a probabilistic technique called the Coherent

Point Drift (CPD) algorithm is presented. The

alignment of two point sets is considered as a

probability density estimation problem. The first

point set, represented as the Gaussian mixture model

(GMM) centroids, is fitted to the data (the second

point set) by maximizing the likelihood. The GMM

centroids are moved coherently as a group, which

preserves the topological structure of the point sets.

The coherence constraint is imposed by

regularization of the displacement field. The

technique can be used in cases of rigid and non-rigid

point set transformations.

The most widely used transformation models

describing the geometric deformations specific to

image acquisition process are the affine (Kammerer,

2004, Cappellini, 2005) and perspective projection

models (Hartley, 2004). In some cases, the other

types of models are studied by the authors.

The analysis of publications has shown that:

(a) the problem of automated registration of X-ray

images of fine art paintings is purely represented in

A PROCEDURE FOR AUTOMATED REGISTRATION OF FINE ART IMAGES IN VISIBLE AND X-RAY

SPECTRAL BANDS

163

literature; (b) the feature sets commonly used in

image registration techniques are ineffective in the

current task due to the properties of X-ray images;

technique is attractive in the current research; (d) the

model of perspective projection is adequate to the

problem under consideration.

Here, we propose a two step procedure, oriented

on the specificity of the problem. The main

operations of the developed procedure are as

follows: (a) image preprocessing (color reduction,

correcting X-ray image acquisition deformation,

filtering, downsampling, etc.); b) localization of

control points; (c) establishing correspondence of

the control point sets; (d) calculating transformation

matrix and image registration.

To increase the precision of registration, the

operations (b) - (d) are running twice at the steps of

rough and precise registration.

3 THE PROPOSED SOLUTION

Localization of control points for registration X-ray

and visible images is complicated by difference in

image content. Characteristic points found in one of

the images may not be found in another image of the

pair. Also, it is not easy to find geometrical

primitives and local features suitable for image

registration in the images of fine art paintings. In

roentgenograms of paintings, the objects painted

using the white lead are strongly discernible and

looking bright. The bright regions in X ray images

usually correspond to the bright regions in

photographs (see Figure 1). This property will be

used for selecting the control points. In this work,

the local grayscale maxima associated with bright

regions of painting will be used as the control points.

The local intensity extrema are invariant to

translation, rotation, scaling, and global intensity

variance. In order to exclude the maxima

corresponding to small image details or conditioned

by noise, the images should be smoothed. The

degree of smoothing should be selected taking into

account the value of registration error. We propose a

two step registration process. At the coarse step, the

strongly smoothed images are used for localizing the

control points. In this case, only a few reciprocal

points associated with large bright image details will

be found. At the precise registration step, the slightly

smoothed images are used for the control points

detection. This yields that the number of reciprocal

points will increase, and the registration error will

decrease.

The choice of the best technique for finding

correspondences between the control point sets of

the images is based on the results of the comparative

study of several matching techniques. SVD-,

eigenvector-based, structural and Coherent Point

Drift algorithms were tested using artificial and real

data sets. The CPD algorithm demonstrated the best

true/false correspondence ratio for scale factor

changes up to 30 percents and rotation angles up to

20 degrees (Murashov, 2010

At the first step of the procedure the Coherent

Point Drift algorithm (Myronenko, 2010) seems to

be the most appropriate for finding correspondences

between the control points of the two images. At the

second step the control point correspondences are

found directly from the analysis of coordinate-based

proximity matrix. For eliminating the false

correspondences, the special iterative procedure is

proposed. The task of obtaining the optimal

transform between two images is solved using

conventional technique (Hartley, 2004). The next

sections are devoted to the main operations of the

proposed procedure.

4 IMAGE PREPROCESSING

For simplification, all of the image processing

operations deal with grayscale images. Hence, the

first operation is the color reduction. The second

operation is aimed on correcting distortions in X-ray

images conditioned by the construction of used X-

ray unit. For this purpose the following

transformation is used:

+Δ'( )rrLL L,

where

r and 'r are the lengths of the position

vectors of the same point in the corrected and

original images,

L is the distance from the X-ray

emitter to the painting,

L is the painting thickness.

The canvas texture of X-ray image (see Figures

1, 2) obstructs control points detection. To suppress

the periodical intensity oscillations, the image

filtering is applied:

1−

ΦΦ ⋅(() )

vvI,

where

v and

v are the initial and the filtered

images,

and

−

are the operations of forward

and backward Fourier transform,

I is the filter

mask,

()

⋅

denotes the operation of elementwise

multiplication. The filter mask is obtained from the

VISAPP 2011 - International Conference on Computer Vision Theory and Applications

164

inverted Fourier spectrum image after thresholding

and removing the low pass component.

In order to reduce the computational expenses at

the main stages of the procedure, image processing

algorithms operates with images of size equal to 1/4

or 1/8 of the initial size. At the final stages, the full-

size images are utilized. As soon as the control

points are associated with the bright image areas, it

is reasonable to segment the regions of interest in

order to exclude the waste of points. For this

purpose the algorithm of adaptive thresholding is

used (Niblack, 1986).

5 IMAGE REGISTRATION

AT THE COARSE STEP

Both at the coarse and the precise steps, the local

intensity maxima associated with the bright details

existing in X-ray and visible images are used as the

control points candidates. For reducing the noise

remained after filtering and suppressing the

influence of small objects and details, the images are

blurred by convolution with Gaussian kernel:

22 2

πσ

−+

()

(,, )

Gxy e ,

where

,xy are the spacial coordinates,

is the

parameter. Chosen

value should provide only a

few local maxima in one region of interest.

For detection local intensity maxima, an

algorithm proposed in (Kuijper, 2002) is used. Local

intensity maxima detected in the images presented in

Figure 1 at

= 6 are shown in Figure 3.

The next operation to be done is to find

correspondences between two point sets detected in

X-ray and visible images.

For this purpose the Coherent Point Drift method

(Myronenko, 2010) is applied. The result of the

algorithm is the correspondence probability matrix.

Maximal element in row i and column j shows the

correspondence of point i in the first image to point j

in the second one. The CPD method is efficient, but

it can establish false correspondences working with

real images. In order to decrease the number of false

associations additional local feature descriptions are

usually used (Pilu, 1997, Zhao, 2004).

In case of significant differences in image

modalities (see Figure 2) the local feature

descriptions cannot improve the result. For

eliminating false correspondences the following

iterative procedure is proposed. The correspondence

is considered as false if the registration error for this

control point couple gives maximal contribution to

an error functional.

(a) (b)

Figure 3: Local intensity extrema in visual and in X-ray

images.

Let the quality of control point registration at

iteration i is defined as

−

∑

iij

222

=− +−()()

Ti Ti

ij ij ij ij ij

dxx yy

where i is an iteration number, j is a point couple

number, p is an initial number of control point

couples,

x and

are the coordinates of j point

couple in the data image transformed by a

transformation T calculated at step i. The control

point couple number k is excluded if the following

condition is held:

== ≤≤−max( ),

ik ij ij

Id d kpi

The process is terminated when

max

dd,

where

max

d is the absolute error bound, or if

maxi

II,

max

I is depending on the mean squared

error bound. The registration quality is evaluated by

the absolute error value, mean squared error, and

visually.

As it was mentioned above, the suitable

transformation for solving the considered problem is

the general projective transformation. The problem

for calculating transformation matrix is formulated

as follows (Hartley, 2004). Let

→(, ):uxy R R

be a model image obtained in X-ray spectral band

and

→(','):vx y R R be a data image obtained

A PROCEDURE FOR AUTOMATED REGISTRATION OF FINE ART IMAGES IN VISIBLE AND X-RAY

SPECTRAL BANDS

165

in optical spectral band. It is necessary to find a

transformation matrix



'XHX

(2)

minimizing the mean squared error and mapping

(',')vx y into (, )uxy . In (2) 1=



(,,)

Xxy and



' ( ', ', )

Xxy are the homogeneous coordinates

of the model and the data images,

H is a

homogeneous 3x3 matrix. For calculating

H , it is

necessary to solve 2n algebraic equations for n

associated control points. We use Levenberg-

Marquardt algorithm providing good convergence

(Madsen, 2004). Computational complexity of the

procedure is comprised of the complexity of the

CPD algorithm (linear) and Levenberg-Marquardt

algorithm applied n times, where n is the number of

iterations depending on the number of control point

pairs. For image registration task, computational

cost of Levenberg-Marquardt technique is

comparable to that of the gradient descent method.

6 IMAGE REGISTRATION

AT THE PRECISE STEP

At the precise registration step in order to provide

suitable precision, the control points are detected in

slightly blurred images (

≤

). At this step for

control points association we analyze directly the

proximity matrix of two point sets because

corresponding maxima in the images registered at

the previous step are closely spaced. For association

of found intensity maxima points the following

operations are needed: (a) coordinates of the newly

detected data image maxima are transformed using

the matrix

H obtained at the coarse step; (b) the

proximity matrix of two maxima sets is calculated;

proximity matrix are found. Indices of the found

matrix element define the correspondence of an

element i of the first point set to an element j of the

second set; (d) a new transformation matrix

H is

calculated, and the data image of the original size is

transformed.

The result of registration of images shown in

Figure 1 is shown in Figure 4.

7 EXPERIMENTAL RESULTS

The developed procedure was tested on six real

image pairs. At the first step, from 8 to 30

corresponding pairs of control points were detected

for different images. At the second step, up to the

several hundreds of points are detected in

photograph and roentgenogram and up to 100

control points couples selected by the point set

matching algorithm.

Figure 4: Registered images from Figure 1.

The registration precision needed for fine art

paintings restoration can be achieved by the proper

localization of control points. For this purpose it is

necessary to detect control points in different regions

of the images. However, this cannot be done in some

cases due to the properties of the roentgenograms

depending on the amount of the white lead used by

the painter. The registration quality is evaluated by

the mean squared error

e computed at the control

points. The suitable precision of registration of

images of paintings at the precise step

≤

obtained when the error value

4e ≤

pixels. This

result is in accordance with the results presented in

(Kammerer, 2004) for infrared and visible images of

paintings.

8 CONCLUSIONS

The two-step procedure of the automated

registration of multimodal images of fine art

paintings is developed. Local intensity extrema

detected in blurred images are used as the control

points. The control points are associated with bright

regions in the painting visible in X-ray and optical

spectral bands. At the coarse step the Coherent Point

Drift algorithm is applied for establishing

VISAPP 2011 - International Conference on Computer Vision Theory and Applications

166

correspondences between the characteristic point

sets. The control point coordinates are used as the

features for constructing proximity matrix. The

algorithm is combined with iterative procedure for

excluding false correspondences. The general

projective transformation model is used for image

registration. The registration precision is in

accordance with the existing method. The future

research will be aimed on improving the control

point set matching technique and application of

alternative transformation models.

ACKNOWLEDGEMENTS

This work was supported by the RFBR grant No 09-

07-00368.

REFERENCES

Cappellini, V., et. al., 2005. An automatic registration

algorithm for cultural heritage images. In ICIP’2005,

2005 International Conference on Image Processing.

IEEE, 2, 566–569.

Carcassoni, M., Hancock, E. R., 2003. Correspondence

Matching with Modal Clusters. IEEE Transactions on

Pattern Analysis and Machine Intelligence, 25 (12),

1609-1615.

Chen Y. and Medioni, G., 1992. Object Modelling by

Registration of Multiple Range Images. International

Journal of Computer Vision and Image

Understanding, 10(3), 145-155.

Delponte, E., Isgrò, F., Odone, F., Verri, A., 2006. SVD-

matching using SIFT features. Graphical Models,

68(5), 415-431.

Harris, C., Stephens, M., 1998. A combined corner and

edge detector. In Alvey Vision Conf. 147-151.

Hartley, R., Zisserman, A., 2004. Multiple View Geometry

in Computer Vision. Cambridge University Press.

Heitz, F., Maitre, H. de Couessin, C., 1990. Event

Detection in Multisource Imaging: Application to Fine

Arts Painting Analysis. IEEE transactions on

acoustics, speech, and signal processin, 38(1), 695-

704.

Kammerer, P., Hanbury, A., Zolda, E., 2004. A

Visualization Tool for Comparing Paintings and Their

Underdrawings. In EVA’2004, Conference on

Electronic Imaging & the Visual Arts, 148–153.

Kirsh, A., and Levenson, R. S., 2000. Seeing through

paintings: Physical examination in art historical

studies. Yale U. Press, New Haven, CT.

Kuijper, F., 2002. The Deep Structure of Gaussian Scale

Space Images. Ph.D. Thesis, Utrecht University. ISBN

90-393-3061-1.

Lowe, D. G., 1999. Object recognition from local scale-

invariant features. In ICCV’99, 7th International

Conference on Computer Vision. IEEE, 1150–1157.

Madsen, K., Nielsen, H. B., O. Tingleff, 2004. Methods

for Non-Linear Least Squares Problems. Technical

Report. University of Denmark.

Maintz, J. B. A., Viegever, M. A., 1998. An Overview of

Medical Image Registration Methods. URN:

NBN:NL:UI:10-1874-18921, Utrecht University.

Maitre, H., Schmitt F., Lahanier, C., 2001. 15 years of

image processing and the fine arts. In: ICIP'2001,

2001 International Conference on Image Processing.

IEEE, 1, 557-561.

Martinez, K., Cupitt, J., Saunders, D., Pillay, R., 2002.

Ten Years of Art Imaging Research. In Proceedings of

the IEEE, 90(1), 28-41.

Murashov, D., Kamyshanov, E., 2010. A Comparative

Study of Point Set Registration Algorithms. In. PRIA-

10-2010, 10th Int. Conference on Pattern Recognition

and Image Analysis: New Information Technologies,

2, 323-326.

Myronenko, A., 2010. Song, X. Point Set Registration:

Coherent Point Drift. IEEE Transactions on Pattern

Analysis and Machine Intelligence. Retrieved from

IEEE Computer Society Digital Library: http://doi.ieee

computersociety.org/10.1109/TPAMI.2010.46

Niblack, W., 1986. An Introduction to Digital Image

Processing. Prentice Hall, Englewood Cliffs, NJ.

Pilu M., 1997. A direct method for stereo correspondence

based on singular value decomposition. CVPR’97,

1997 Conference on Computer Vision and Pattern

Recognition. IEEE, 261-266.

Rusinkiewicz, S. Levoy, M., 2001. Efficient Variants of

the ICP Algorithm, In 3DIM'01, 3rd International

Conference on 3-D Digital Imaging and Modeling,

145-152.

Schmid, C., Mohr, R., 1997. Local Greyvalue Invariants

for Image Retrieval. PAMI, 19(5), 872-877.

Scott, G. and Longuet-Higgins, H. C., 1991. An Algorithm

for Associating the Features of Two Images.

Proceedings of the Royal Society London, B244, 21-

26.

Shapiro, L. S., Brady, J. M., 1992. Feature-based

correspondence: an eigenvector approach. Image and

Vision Computing, 10, (5), 283 – 288.

Sharp, G., Lee, S., and Wehe, D., 2002. ICP registration

using invariant features. IEEE Transactions on Pattern

Anal., 24(1), 90–102.

Stork, D. G., 2009. Computer Vision and Computer

Graphics Analysis of Paintings and Drawings: An

Introduction to the Literature. LNCS, Springer-Verlag,

5702, 9–24.

Zhao, F., 2004. Image matching based on singular value

decomposition. In PCM’2004, 5th Pacific Rim

Conference on Multimedia, LNCS, Springer-Verlag

3333, 19–126.

Zitova, B., Flusser, J., 2003. Image registration methods: a

survey. Image and Vision Computing, 21(11), 977–

1000.

A PROCEDURE FOR AUTOMATED REGISTRATION OF FINE ART IMAGES IN VISIBLE AND X-RAY

SPECTRAL BANDS

167