POISSON LOCAL COLOR CORRECTION FOR IMAGE STITCHING
Mohammad Amin Sadeghi, Seyyed Mohammad Mohsen Hejrati and Niloofar Gheissari
Computer Vision Group, School of Mathematics
Institute for studies in theoretical Physics and Mathematics, Tehran, Iran
Keywords:
Image stitching, Poisson equation, Panorama, Color correction, Registration, Image mosaicing.
Abstract:
A new method for seamless image stitching is presented. The proposed algorithm is a hybrid method which
uses optimal seam methods and smoothes the intensity transition between two images by color correction. A
dynamic programming algorithm that finds an optimal seam along which gradient disparities are minimized
is used. A modification of Poisson image editing is utilized to correct color differences between two images.
Different boundary conditions for the Poisson equation were investigated and tested, and mixed boundary
conditions generated the most accurate results. To evaluate and compare the proposed method with competing
ones, a large image database consisting of more than two hundred image pairs was created. The test image
pairs are taken at different lighting conditions, scene geometries and camera positions. On this database the
proposed approach tested favorably as compared to standard methods and has shown to be very effective in
producing visually acceptable images.
1 INTRODUCTION
Since two registered images usually have local and
global intensity differences, in order to create visu-
ally acceptable images it is essential to complement
the registration process with seamless image stitch-
ing. As noted in (Hasler and Susstrunk, 2000), sev-
eral factors reflect the intensities recorded by cameras.
They include: exposure variances, white balancing,
gamma correction, vignetting and digitizer parame-
ters. Seamless stitching has important applications in
many high level tasks such as building panoramic im-
ages, virtual reality, super resolution and texture syn-
thesis. Recently there have been many breakthrough
attempts in seamless image stitching such as (Levin
et al., 2003), (Uyttendaele et al., 2001), (Jia and Tang,
2005a), (Jia and Tang, 2005b).
There are two main approaches for the elimination
of seams and artifacts in the overlapping images. The
first approach is to find an optimal seam along which
some energy function is minimized (Jia and Tang,
2005a), (Agarwala et al., 2004). Jia and Tang (Jia and
Tang, 2005a) have described different possible energy
functions (weighting functions) that might be use to
find the optimal seam. These energy functions may
be based on intensity or gradient differences of the
two images. Dynamic programming or graph cut is
used to minimize the energy function. One advantage
of optimal seam methods is that they are applicable
to dynamic scenes. This is because the optimal seam
does not pass through areas of high gradient disparity.
However, optimal seam algorithms are less suitable
when the illuminations of the input images are not the
same or their overlapping area consists of thin strips.
The second approach is based on enhancing the
intensity values of one or both images so that the re-
sulting stitched image looks natural. This task can be
performed either globally or locally.
Szeliski and Shum (Shum and Szeliski, 2001) pro-
pose a method for building panoramic images capa-
ble of correcting local and global intensity misalign-
ments. Their feathering algorithm results in more nat-
ural blending effects. Still in many cases it produces
unnatural seams under different illumination condi-
tions. Belt and Adelson (Burt and Adelson, 1983)
use multiresolution splines for color correction. Uyt-
tendaele et al. (Uyttendaele et al., 2001) proposed a
method capable of stitching multiple images taken at
various exposures and illuminance conditions. They
allow for large scene changes or misregistrations be-
tween two images. Levin et al. (Levin et al., 2003)
introduced Gradient Domain Image Stitching (GIST)
method. GIST utilizes on an optimization algorithm
to minimize a cost function over image derivatives.
275
Amin Sadeghi M., Mohammad Mohsen Hejrati S. and Gheissari N. (2008).
POISSON LOCAL COLOR CORRECTION FOR IMAGE STITCHING.
In Proceedings of the Third International Conference on Computer Vision Theory and Applications, pages 275-282
DOI: 10.5220/0001077102750282
Copyright
c
SciTePress
This cost function is a dissimilarity measure between
the derivatives of the resulting stitched image and the
derivatives of the input images. Jia and Tang (Jia and
Tang, 2005b) proposed a tensor voting method for
global and local intensity alignment. They use a sym-
metric tensor representation to model the joint image
space (source image and target image intensities) as a
token. They extract curves in the image and let tokens
on a curve communicate mutually to reinforce each
other. A dense tensor field postulates smooth connec-
tions among tokens and ensures natural color transi-
tions. This method can successfully handle occlusion
and motion blur. Their local intensity alignment ap-
proaches is devised to correct vignetting effects.
We have observed that a hybrid method that can
take advantage of the strengths of both main ap-
proaches, generates more reliable results. Therefore
the proposed hybrid method first finds an optimal
seam to stitch two images. Then it uses the Pois-
son equation with mixed boundary conditions in the
color domain. As a result, the local color differences
between two images are corrected. Poisson equation
has been already used for image editing (P´erez et al.,
2003) or correcting structure misalignments produced
by stitching (Jia and Tang, 2005a). An advantage of
the proposed method is its low computational time
that makes it particularly desirable for applications to
high resolution images. The proposed method relies
on a model based registration algorithm to align two
images (Gheissari et al., 2008). In fact, the registra-
tion algorithm computes the transformation parame-
ters after recovering the transformation model.
This paper is organized as follows: Section 2,
gives a brief summery of the registration method
used. Methods for finding optimal cutting curve are
described in section 3. In section 4, the proposed
method, Poisson Local Color Correction (PLCC) is
explained. Different boundary conditions for the so-
lution of the Poisson equation are described in the
following section (4). Some challenging images for
local color correction and the results of applying the
proposed algorithm on them are shown in the section
5.
2 MODEL-BASED IMAGE
REGISTRATION
In this work, a registration method introduced in
(Gheissari et al., 2008) is utilized. In summary this
registration method first finds a set of SIFT inter-
est points (Lowe, 1999) and the correspondences be-
tween them. The outliers are eliminated to gener-
ate an outlier-free set of correspondences. This is
achieved by robustly fitting a translation model to the
correspondences. Next all models in the model library
are fitted to the above set of inliers.
(a) (b)
(c) (d)
(e)
Figure 1: Different transformation models for registration:
(a) Translation, (b) Euclidian (c) Similarity, (d) Affine and
(e) Projective transformation.
CAIC is applied to select the correct transforma-
tion model from a library of candidate models (Boz-
dogan, 1987). Figure 1 shows the set of candidate
models. The final set of inliers is determined by ap-
plying the chosen model to the data set. The algorithm
estimates the parameters of the chosen model (based
on the set of inliers) to generate the transformation
matrix.
Before finding the optimal seam and using the
Poisson equation to correct color difference, we ap-
ply the transformation matrix to the source image.
This results in two registered images with an over-
lapping area which can be a triangle, a rectangle, or
a higher order polygon depending on the transforma-
tion model.
3 OPTIMAL SEAM METHODS
Optimal seam algorithms are used widely in appli-
cations such as pattern synthesis and image stitching
(Kwatra et al., 2003), (Jia and Tang, 2005a), (Uyt-
tendaele et al., 2001). These methods find a cutting
curve in the overlapping area and stitch the images
VISAPP 2008 - International Conference on Computer Vision Theory and Applications
276
along the cutting curve as their common boundary.
The main approach in optimal seam algorithms is to
specify a weight for each pixel belonging to the over-
lapping area. Then a curve that minimizes this weight
function is selected as the optimal cutting seam.
(a)
(b)
(e)
Figure 2: (a) Two registered images and their overlapping
area, (b) optimal cutting curve, (c) final stitched images.
Assume we have registered images I
T
and I
S
. The
overlapping area is I
ov
T
in image I
T
and I
ov
S
in image I
S
(as shown in figure 2). We would like to cut and stitch
I
T
and I
S
along an optimal curve C.
Let C be a cutting curve, then the cost of havingC
as cutting curve is defined as:
W(C) =
pC
W(p), (1)
where W(p) is the value of the weight function asso-
ciated with pixel p. The optimal curve(seam) is the
curve which minimizes the cost W(C).
Different candidates for weight function are
W
g
(p) = ||I
S
(p) I
T
(p)||
n
n
, (2)
and
W
i
(p) = ||I
S
(p) I
T
(p)||
n
n
, (3)
where ||.||
n
n
is the
n
–norm. A linear combination of
W
g
(p) and W
i
(p) could also be considered as weight
function (Jia and Tang, 2005a). We have tested differ-
ent energy functions and observed that the function
W(p) = ||I
S
(p) I
T
(p)||
1
1
(4)
(a) (b)
(c) (d) (e)
( f) (g) (h)
Figure 3: (a) And (b) are the input images. The images are
borrowed from figure 6 of (Jia and Tang, 2005a). The im-
ages, used in our experiments, are of our size 337 × 334.
(c) cutting curve using intensity based weight function
W
i
(p) = ||I
S
(p) I
T
(p)||
1
1
. (d) Stitching using intensity
based weight function without color correction (without
PLCC) (e) Stitching using intensity based weight function
with color correction (PLCC) (f) cutting curve using gra-
dient based weight function W
g
(p) = ||I
S
(p) I
T
(p)||
1
1
.
(g) Stitching using gradient based weight function without
color correction (without PLCC) (h) Final result after using
gradient based weight function and applying PLCC method.
is more reliable than the other candidates specially
when the illuminations of I
T
and I
S
are different. Af-
ter finding the optimal seam PLCC is applied to cor-
rect color and illumination differences between two
input images. Setting n = 1 (using
1
–norm) leads
to better results because small intensity differences
are ignored in higher norms. We use a 3-dimensional
RGB color vector for each pixel as I(p), and observed
that using HSV color space has the same performance
as the RGB color space.
The result of using intensity based weight func-
tion and gradient based weight function is shown in
Figures 3 and 5.
POISSON LOCAL COLOR CORRECTION FOR IMAGE STITCHING
277
There are various algorithms with different com-
plexities for finding the optimal cutting curve. Min-
Cut(Max-flow) algorithm (Kwatra et al., 2003) works
accurately. However due to its high complexity, it im-
poses significant computational burden on images of
high resolution. We used a combination of dynamic
programming and BFS (Breadth First Search) algo-
rithm in our experiments to find optimal cutting curve.
Its complexity is O(n) where n is the number of pixels
in the overlapping area.
Figure 4: In some cases the optimal seam is not a simple
curve.
In some cases the optimal seam gets more com-
plex as shown in Figure 4. Therefore an optimal seam
is not necessarily one simple curve.
4 COLOR CORRECTION BASED
ON POISSON EQUATION
Poisson equation has been used for image editing
and image alignment applications, for example (P´erez
et al., 2003), (Jia and Tang, 2005a) and (Agarwala
et al., 2004). Here we use Poisson equation to cor-
rect color differences between two already registered
images.
Lets be the region (in I
S
) that lies on one side of
the optimal seam (the side belonging to the source im-
age). The boundary of is denoted by ∂Ω. We sep-
arate ∂Ω into two parts: the common boundary with
I
T
(referred to as ∂Ω
C
), and the other boundaries (de-
fined as ∂Ω
B
). Our notations are illustrated in Figure
6.
To eliminate artifacts and color differences over
the seam ∂Ω
C
, we define a correction function Ψ on
as
ˆ
I
S
= I
S
+ Ψ (5)
where
ˆ
I
S
is the corrected image. Then the Poisson
equation is solved on and the resulting correction
function Ψ, is added to I
S
to compensate for color dif-
ferences between two images. For colored images,
we apply the same method on each color channel of
the input image. According to our experiments the
performance of the method is the same in RGB and
(a) (b) (c)
(d) (e) ( f)
(g) (h) (i)
Figure 5: Illustration of applying PLCC on two images.
(a) and (b) input images of size 800 × 600. (c) the result
of putting two registered images together without applying
any corrections. (d) a cutting curve found by using inten-
sity based weight function. (e) the solution of the poisson
equation solved on the corresponding boundary. (f) the re-
sult of stitching the images using an intensity based seam
line. (g) shows a cutting curve found using Gradient based
weight function. (h) the solution of the poisson equation
associated with the cutting curve in (g) and (i) the final re-
sult of stitching using intensity based seam line. As you
can see, the color is not corrected perfectly in (f) because
it’s corresponding cutting curve has passed dark areas and
the boundary conditions of the poisson equation in (e) is not
valid.
Figure 6: Notation of the region : is the region of the
visible part of I
S
. The common boundary with I
T
is defined
as ∂Ω
C
, and the other boundaries are defined as ∂Ω
B
.
HSV color spaces. For solving Poisson equation on
several boundary conditions could be assumed. The
VISAPP 2008 - International Conference on Computer Vision Theory and Applications
278
(a) (b)
(c) (d)
(e) (f )
Figure 7: Comparison between the results of using Drich-
let boundary conditions or Neumann boundary conditions
on ∂Ω
B
. (a) and (b) source and target images, both of size
800× 600. (c) the solution of Poisson equation with Drich-
let boundary conditions on ∂Ω
B
as color correction factor.
(d) the respective result of color correction. It is equivalent
to applying Poisson image editing (P´erez et al., 2003) (e)
the solution of Poisson equation with Neumann boundary
conditions on ∂Ω
B
and (f) the the result of color correction
using the correction factor (e). The correction factor (c) and
(e) are solved on a rectangular domain before applying the
transformation and added to source images after transfor-
mation.
Dirichlet boundary condition equal to the difference
between intensity of two input images along ∂Ω
C
was
imposed on ∂Ω
C
. For ∂Ω
B
the Neumann boundary
condition was used.
Imposing the proper boundary condition is critical in
our application. One might use Dirichlet boundary
condition on ∂Ω
B
(equation 6). However, the prob-
lem here is the choice of the boundary function on
∂Ω
B
. For the Dirichlet boundary condition, an ap-
proximation to the value on ∂Ω
B
is necessary. The
trivial approximation zero for the boundary value re-
sults in imperfect color correction as shown in Fig.
7.
∆Ψ = 0 over with
Ψ = I
T
I
S
, On ∂Ω
C
Ψ = Constant, On ∂Ω
B
,
(6)
where denotes the Laplace operator.
The Neumann boundary condition on ∂Ω
B
in
PLCC (equation 7) produced significantly better re-
sults. The superior performance of the Neumann
boundary condition on ∂Ω
B
is explicable in view of
the fact that it does not fix intensity values along ∂Ω
B
but constraints the gradient values along ∂Ω
B
to be
zero. Fixing the intensity on ∂Ω
B
results in Ψ to
be constant over ∂Ω
B
, while the ideal values of Ψ
over the ∂Ω are not necessarily constant. Further-
more, there is no straightforward theoretical way to
set the values of Ψ over ∂Ω
B
and an approximation is
inevitable. This problem is aggravated where the il-
lumination condition between the input images differ
significantly.
∆Ψ = 0 over with
Ψ = I
T
I
S
, On ∂Ω
C
∂Ψ
n
= 0, On ∂Ω
B
,
(7)
where ˆn is normal vector at ∂Ω. This Poisson equa-
tion is the result of the following minimization prob-
lem (equation 8).
(a)I
T
(b)I
S
(a) (b)
(c) (d)
Figure 8: Applying PLCC and optimal seam to dynamic
scene. (a) and (b) input images of size 800× 600. (c) Opti-
mal seam based on gradient disparities. (d) Stitching with-
out using optimal seam. The shadow of pedestrians walk-
ing behind is visible, while the pedestrians body are elimi-
nated from the stitching result. The color difference is very
prominent. (e) Stitching using optimal seam. The shadow
of pedestrians are no longer visible. The color differences is
still prominent (f) Final result of applying color correction
(PLCC) and optimal seam.
POISSON LOCAL COLOR CORRECTION FOR IMAGE STITCHING
279
(a) (b)
(c) (d)
(e)
( f)
Figure 9: Creating super resolution images using images
of poor quality. (a), (b), (c) and (d) are input images (of a
puzzle hanged to a wall) taken at four different viewpoints
and illumination conditions. input image (a) is taken un-
der yellow light source, (b) taken under yellow and white
light sources, (c) taken at low exposure time under white
light source and (d) taken under white light source. (e) the
stitching result without PLCC. (f) the stitching result after
PLCC.
min
Ψ
ZZ
|∇Ψ|
2
with
Ψ = I
T
I
S
, On ∂Ω
C
∂Ψ
n
= 0, On ∂Ω
B
,
(8)
where the gradient operator [
x
,
y
] is represented by
. This minimization problem results in visually nat-
ural output images because it minimizes the
2
–norm
of the gradient of intensities near the boundary.
We used the method of finite differences and the
central difference formula to solve the Poisson equa-
tion and the resulting system of linear equations.
5 EXPERIMENTAL RESULTS
A database for testing and evaluating image stitching
algorithms was created. This database contains over
two hundred pairs of images from over forty scenes.
All pairs are available in three differentstandard sizes.
The scenes vary in color, camera properties, camera
location and lighting. The database will be publicly
available at ”math.ipm.ac.ir/vision”. The transforma-
tion matrix for each pair of images was determined
using (Gheissari et al., 2008). This database contains
challenging images for registration, color correction,
and building panoramic images.
As mentioned before the optimal seam method is
applicable to dynamic scenes. This is because the op-
timal seam does not pass through areas of high gra-
dient disparity. This capability is shown in figure 8.
As can be seen from this figure, two different groups
of pedestrians are present in the input images. The
optimal seam passes through the narrow gap between
these two groups and therefore both groups are pre-
sented in the resulting stitched image. If the opti-
mal seam has not been used, then the shadow of the
pedestrian group behind would be visible without the
pedestrians themselves.
Performance of the proposed method is compared
with some standard competing methods. As shown in
figure 10, a combination of PLCC and optimal seam
outperforms feathering (Uyttendaele et al., 2001),
pyramid blending (Adelson et al., 1984), GIST (Levin
et al., 2003) and Poisson image editing (P´erez et al.,
2003).
The proposed algorithm is desirable for making
super resolution images due to its high accuracy and
low time complexity. To show this capability we
chose four images of a puzzle hanged to a wall. The
images were taken at four different view points and
significantly various illumination conditions as shown
in figure 9. The results of applying the proposed al-
gorithm shows its success .
VISAPP 2008 - International Conference on Computer Vision Theory and Applications
280
(a)I
T
(b)I
S
(c)
(d)
(e) ( f) (g) (h) (i) ( j)
Figure 10: Comparing PLCC with some competing meth-
ods: (a) and (b) are input images borrowed from figure 4
of (Levin et al., 2003). The images are of size 348 × 255
and 290× 257 respectively. (c) Stitched image after apply-
ing optimal seam , (d) The color correction result generated
by PLCC after finding optimal seam, (e) Feathering (Uyt-
tendaele et al., 2001), (f) Pyramid blending (Adelson et al.,
1984), (g) Feathering on the gradients (Uyttendaele et al.,
2001), (h) Poisson editing (P´erez et al., 2003), (i) GIST
(Levin et al., 2003) (j) The result of PLCC method.
6 CONCLUSIONS
A new method for seamless image stitching was pre-
sented. The proposed algorithm is a hybrid method,
that combines optimal seam approaches and methods
which smooth the intensity transition between two
images (by color correction). Using a dynamic pro-
gramming algorithm, an optimal seam along which
gradient disparities are minimized is determined. A
local color correction method based on Poisson equa-
tion is used to enhance color variations between im-
ages. Different boundary conditions to solve the Pois-
son equation were investigated and tested. We con-
cluded that using mixed boundary conditions gen-
erates the most accurate results. To evaluate and
compare the proposed method (with some competing
methods), a large image database consisting of over
two hundred image pairs was utilized. The test im-
age pairs are taken at different lightening conditions,
scene geometry and camera positions. Compared to
standard methods, proposed approach performed fa-
vorably on this database and has shown to be very
effective in producing visually acceptable images.
ACKNOWLEDGEMENTS
We would like to acknowledge Dr. Mohammad Reza
Mokhtarzade for his valuable advices on solving
PDEs. In addition, we would like to acknowledge
Dr. Mehrdad Shahshahani for his kind helps in this
project and in writing this paper. Mr. Mostafa Ka-
mali Tabrizi helped us in modifying the source code
for registration. We appreciate the valuable discus-
sions he had with us. Finally we thank Zohre Sharafi,
Parisa Mirshams, Paria Mehrani, Sheyda Beigpoor for
providing us with beautiful test images. This project
was supported by Institute for studies in theoretical
Physics and Mathematics (IPM).
REFERENCES
Adelson, E. H., Anderson, C. H., Bergen, J. R., Burt, P. J.,
and Ogden, J. M. (1984). Pyramid methods in image
processing. RCA Engineer, 29(6).
Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S.,
Colburn, A., Curless, B., Salesin, D., and Cohen, M.
(2004). Interactive digital photomontage. ACM Trans.
Graph., 23(3):294–302.
Bozdogan, H. (1987). Model selection and akaike’s infor-
mation criterion (aic): The general theory and its ana-
lytical extensions.
Burt, P. J. and Adelson, E. H. (1983). A multiresolution
spline with application to image mosaics. ACM Trans.
Graph., 2(4):217–236.
Gheissari, N., Tabrizi, M. K., Sharafi, Z., and Mirshams, P.
(2008). Model-based global image registration. In
Proc. of the 3rd International Conference on Com-
puter Vision Theory and Applications, page to appear.
POISSON LOCAL COLOR CORRECTION FOR IMAGE STITCHING
281
Hasler, D. and Susstrunk, S. (2000). Colour handling in
panoramic photography.
Jia, J. and Tang, C.-K. (2005a). Eliminating structure and
intensity misalignment in image stitching. In ICCV
’05: Proceedings of the Tenth IEEE International
Conference on Computer Vision, pages 1651–1658,
Washington, DC, USA. IEEE Computer Society.
Jia, J. and Tang, M.-C.-K. (2005b). Tensor voting for im-
age correction by global and local intensity alignment.
IEEE Trans. Pattern Anal. Mach. Intell., 27(1):36–50.
Kwatra, V., Schodl, A., Essa, I., Turk, G., and Bobick, A.
(2003). Graphcut textures: image and video synthe-
sis using graph cuts. In SIGGRAPH ’03: ACM SIG-
GRAPH 2003 Papers, pages 277–286, New York, NY,
USA. ACM Press.
Levin, A., Zomet, A., Peleg, S., and Weiss, Y. (2003).
Seamless image stitching in the gradient domain.
Lowe, D. G. (1999). Object recognition from local scale-
invariant features. In Proc. of the International
Conference on Computer Vision ICCV, Corfu, pages
1150–1157.
P´erez, P., Gangnet, M., and Blake, A. (2003). Poisson im-
age editing. In SIGGRAPH ’03: ACM SIGGRAPH
2003 Papers, pages 313–318, New York, NY, USA.
ACM Press.
Shum, H.-Y. and Szeliski, R. (2001). Construction of
panoramic image mosaics with global and local align-
ment. pages 227–268.
Uyttendaele, M., Eden, A., and Skeliski, R. (2001). Elim-
inating ghosting and exposure artifacts in image mo-
saics. pages II–509– II–516 vol.2.
VISAPP 2008 - International Conference on Computer Vision Theory and Applications
282