Attribute Operators for Color Images: Color Harmonization based on

Maximal Harmonic Segmentation

ergio Sousa Filho and Franklin C

esar Flores

Department of Informatics, State University of Maring

a, Maring

a, Paran

a, Brazil

Keywords:

Color Gradient, Attribute Filter, Image Segmentation, Color Harmonization.

Abstract:

Attribute openings and thinnings are morphological connected operators that remove structures from images

according to a given criterion. These operators were successfully extended from binary to grayscale images,

but such extension to color images is not straightforward. Color attribute operators have been proposed by a

combination of color gradients and thresholding decomposition. In this approach, not only structural criteria

may be applied, but also criteria based on color features and statistics. This work proposes, in a segmentation

framework, a criterion based on color histogram divergence from a harmonic model. This criterion permits a

segmentation in maximal harmonic regions. An application indicated that the harmonic segmentation permit-

ted a hue correction that would not cause false colors to appear in regions already harmonic.

1 INTRODUCTION

The Mathematical Morphology (MM) provides a

toolbox for developing image ﬁlters (Najman and Tal-

bot, 2013; Serra and Vincent, 1992). Some of these

morphological ﬁlters are grouped in a category, the

connected operators, that has the characteristic of

simplifying the image by merging ﬂat zones (regions

that have the same gray value). And as such, they

have the property of reducing the number of regions

without introducing new borders (Heijmans, 1999).

The attribute ﬁlter (Breen and Jones, 1996) be-

longs to this category, a particular connected operator

that removes components of the image according to

a criterion. Among the most usual criteria it is worth

of mentioning the area, height and volume measure-

ment (Vachier and Meyer, 1995; Vachier, 1995). They

are known to have been applied to solving problems

such as segmentation of medical images, image com-

pression and structural analysis of ore (Breen and Jo-

nes, 1996).

Recenlty, an extension of attribute operators to co-

lor images have been proposed and two criteria based

on color homogeneity have been applied for impro-

vement color segmentation (Sousa Filho and Flores,

2017). This paper proposes, in the color attribute fra-

mework, a harmonic gradient (Ou and Luo, 2006a)

and another color criterion based on the harmonic di-

vergence (Baveye et al., 2013).

This criterion has the property of identifying max-

imal harmonic regions in an image. Based on this pro-

perty, an application is devised for the improvement

of color harmonization (Cohen-or et al., 2006). In

this application, these regions are corrected separately

which in turn provides that a concentration of color in

a speciﬁc region of the image will not bias the harmo-

nic correction as a whole.

2 PRELIMINARY CONCEPTS

Let E ⊂ Z ×Z be a rectangular ﬁnite subset of points.

A binary image may be denoted by a subset X ⊆ E.

Let the power set P (E) be the set of all binary images.

Let the discrete interval of real numbers K =

[lk, uk] be a totally ordered set. Denote by Fun[E, K]

the set of all functions f : E → K. A graylevel image

is one of these functions.

Let Fun[E, C] be the set of all functions f : E → C,

where C = {c

, c

, . . . , c

}, n ≥ 1 and c

∈ R : lk

≤

≤ uk

. Fun[E, C] denotes the set of all color images.

Let A and B be two images, as described in the

last three paragraphs. An image operator (operator,

for simplicity) is a mapping ψ : A → B.

Let B

be the structuring element that deﬁnes a

connectivity (Najman and Talbot, 2013). A connected

component of E is a subset X ⊂ E such that, ∀x,y ∈ X,

there is a path P(x, y) = (p

, p

, ..., p

), p

∈ X, such

that p

= x, p

= y and ∀i ∈ [0,t −1], ∃b ∈ B

: p

+b =

i+1

366

Filho, S. and Flores, F.

Attribute Operators for Color Images: Color Harmonization based on Maximal Harmonic Segmentation.

DOI: 10.5220/0007376203660372

In Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2019), pages 366-372

ISBN: 978-989-758-354-4

2.1 Color Gradient

Literature presents several ways to compute color gra-

dients (Busin et al., 2008; Lucchese and Mitra, 2001).

They are usually designed taking into account the ana-

lysis of each color component image under a certain

color space model. For instance, color gradients may

be designed under RGB (Busin et al., 2008; Evans and

Liu, 2006), HSL (Rittner et al., 2010) or L

∗

(Ru-

zon and Tomasi, 2001).

Let f ∈ Fun[E, C] be a color image under the

∗

color space model (Busin et al., 2008). Let

D(a, b) be a measure of distance between two co-

lors a, b ∈ C. The color gradient ∇

: Fun[E, C] →

Fun[E, R

] is given by, ∀x ∈ E,

∇

(f)(x) =

∈B

D(f(x + b

), f(x + b

)). (1)

In order to apply the thresholding decomposition,

the color gradient needs to be converted to an image

∇(f) ∈ Fun[E, K], as follows:

1. Normalize ∇

(f) from the interval

[min{∇

(f)}, ··· , max{∇

(f)}] to [0, ··· , k]

(rounding it down);

2. Image ∇(f) ∈ Fun[E, K] is given by the comple-

ment of the normalized gradient. This is done be-

cause objects are represented by valleys in the co-

lor gradient, and the negation of such valleys ma-

kes them sliceable by the thresholding decompo-

sition.

2.2 Thresholding Decomposition

Let thr : Fun[E, K] × K → P (E) be the thresholding

function, given by, ∀ f ∈ Fun[E, K], ∀t ∈ K,

thr( f ,t) = {x ∈ E : f (x) ≥ t}. (2)

Let f

bin

: P (E) → Fun[E, k] be the mapping that

gives a numerical representation of a binary image

X ⊆ E, such that, ∀x ∈ E,

bin

(X)(x) =

(

1, if x ∈ X,

0, otherwise.

(3)

Let f ∈ Fun[E, K]. The thresholding decomposi-

tion is given by,

f =

∑

t=1

bin

(thr( f ,t)). (4)

In other words, a grayscale image f may be decom-

posed as a stack of binary images, each one provided

by the thresholding of f by a distinct graylevel k ∈ K.

The “addition” of all binary images in that stack re-

turns image f .

Thresholding decomposition is a way to extend

some operators - designed for binary images - to the

grayscale context (Breen and Jones, 1996). Formally,

the extension of ψ : P (E) → P (E) to Ψ : Fun[E, K] →

Fun[E, K] is given by,

Ψ( f ) =

∑

t=1

bin

(ψ(thr( f ,t))). (5)

2.3 Attribute Operators

Attribute operators are applied to remove all structu-

res that do not ﬁt into a given criterion (Breen and

Jones, 1996). Such criterion is usually tied to a mea-

surement and a comparison to a numerical parameter

p. For instance, an image structure must be removed

if its area is not greater than p - this is the criterion for

area opening.

Let X ∈ P (E) be a binary image. Let C ⊆ X

be a connected component. The trivial operator

: P (E) → P (E) evaluates if C satisﬁes a criterion

T (Breen and Jones, 1996):

(

C, if C satisﬁes criterion T,

∅, otherwise,

(6)

where Γ

(∅) = ∅.

The attribute operator is given by, for all X ∈

P (E),

(X) =

[

C⊆X

(C), (7)

where C is a connected component of X. Note that

this is the binary case - the extension of Eq. 7 to grays-

cale case is given by thresholding decomposition (see

Eq. 5).

For detailed information about attribute openings

and thinnings, see (Breen and Jones, 1996). Notice

that only structural criteria are mentioned in that pa-

per.

2.4 Color Attribute Operators

Color attribute operators (Sousa Filho and Flores,

2017) allows the choice and application of criteria ba-

sed on color information for attribute ﬁltering of color

images. Analysis of local morphological structures

from a color gradient provides regions where infor-

mation is collected from the color input image. And,

such morphological structures are ﬁltered in function

of the collected color information.

Let Γ

: P (E)×Fun[E, C] → P (E) denote the co-

lor trivial operator. It is similar to Eq. 6, but Γ

(C, f)

Attribute Operators for Color Images: Color Harmonization based on Maximal Harmonic Segmentation

367

may also take into account local color information gi-

ven by {f(x) : x ∈ C}, if a criterion T involves color

features or statistics.

Connected component C is enough for assessment

when criterion T is structural, like in graylevel ver-

sion (Breen and Jones, 1996). For criteria based on

color features or statistics, we cite average color er-

ror (Zhang et al., 2003; Borsotti et al., 1998), color

harmony (Ou and Luo, 2006b) and entropy (Duda

et al., 2001). In this paper, we apply the color har-

mony criterion (see Sec. 3).

The computation of the color attribute operator

: Fun[E, C] → Fun[E, K] is described in the algo-

rithm as follows:

1: procedure COLORATRIBUTEOPERATOR(image

f, criterion T )

2: g ← ∇(f)

3: decompose g into a set of slices G

= thr(g,i),

∀i ∈ [1..k]

4: for all G

5: F

←

C⊆G

(C, f)

6: end for

7: Γ

(f) ←

∑

i=1

bin

)

8: return Γ

(f)

9: end procedure

Note that, except for the application of the color

trivial operator Γ

(C, f), the color attribute operator is

computed like the grayscale version (Breen and Jones,

1996).

The use of the ﬁltered image Γ

(f) depends on

the application of the method. For instance, one can

use the residue of Γ

(f). Another approach is to ap-

ply the watershed operator (Najman and Talbot, 2013;

Beucher and Meyer, 1992) to the negation of Γ

(f) in

order to compute a hierarchical segmentation (Meyer,

2001) of f. This approach is adopted in this work.

2.5 Color Harmony

Generally, color harmony can be understood as a co-

lor combination that causes a pleasing effect to the ob-

server. Subjective, the deﬁnition of harmony can have

a different interpretation in according to the context in

which it is applied as in product design, in painting or

in photo enhancement.

Among scientists there is no harmonic theory lar-

gely accepted yet. Across the centuries many works

tried to explain the psychological effects caused by

the colors. Some works tried to explain through a or-

dered arrangement of colors (Ostwald, 1969; Mun-

sell, 1969; Itten, 1961) while others approach the in-

dividual relations between colors (Von Goethe, 1840;

Chevreul, 1967; Moon and Spencer, 1944).

2.5.1 Two-color Harmony

In the line of the harmony theories that explore these

color relations, it is worth mentioning a function (Ou

and Luo, 2006a) that can quantify this relation.

Let h

, C

∗

, L

sum

, 4

, 4H

∗

, 4C

∗

being re-

spectively the hue, the chroma, the sum of lightness,

the absolute difference of lightness and the difference

of hue and chroma values deﬁned in the L*a*b* color

space. Two-color harmony CH is deﬁned as:

CH = H

+ H

, (8)

where H

is the chromatic effect, H

is the lightness

effect and H

is the hue effect, deﬁned empirically in

his paper.

2.5.2 Harmonic Template Determination

The classic Kullback-Leibler divergence is an relative

entropy between two probabilities distributions and

can be interpreted as the information loss when using

a probabilities distribution Q to approximate a distri-

bution P. In the discrete case it is deﬁned as:

(PkQ) =

∑

P(i)ln

P(i)

Q(i)

. (9)

One application of this divergence is when P is a

distribution obtained from real data and Q represents

a model. The model Q that minimizes this divergence

is the one that best ﬁts distribution P.

This application can be used (Baveye et al., 2013)

to ﬁnd which harmonic template best describes a co-

lor image. P is the weighted Hue histogram of the

image computed on the HSV using 360 bins:

P(i) =

∑

x\H(x)=i

S(x) ×V (x)

∑

S(x) ×V (x)

, (10)

i ∈ [0, 359].

Q is calculated for each rotation of the eight har-

monic templates T

(m ∈ {i,V, L, J, I, T,Y, X }) deﬁned

in the chromatic circle (Matsuda, 1995) as shown in

ﬁgure 1. Each template is composed of sectors, each

sector having a center and a width.

The distribution Q given a template m and a rota-

tion al pha is given by:

(i) =







∑

k=1

(i, k) if

∑

k=1

(i, k) 6= 0

0, 01

360

∑

(

∑

k=1

(i, k)) otherwise,

(11)

where n is the number of sectors deﬁned for the tem-

plate T

and

VISAPP 2019 - 14th International Conference on Computer Vision Theory and Applications

368

Figure 1: Harmonic models T

. The gray area represents a sector of hue values that composes the harmonic model.

(i, k) =











exp







−

1 −



2× k i − α









if i ∈]α

−

, α

[

0 otherwise,

(12)

where k · k is the angular distance in the chromatic

circle and α

and w are respectively the center and

width of the sector k of the template T

2.5.3 Color Harmonization

Color harmonization (Cohen-or et al., 2006) is a pro-

cess of mapping the hue values of a image to match a

harmonic model T

. This can be done by a function

(Baveye et al., 2013) as deﬁned:

(x) = C(x)+ Sinal

quadrante

×tanh



2× k H(x) −C(x) k



(13)

where C(x) and w represents the central hue vale and

the angular width, respectively, of the sector associa-

ted with the pixel x and k · k is the angular distance in

the chromatic circle.

To ﬁnd the the template sector and the signal that

one pixel must be mapped, the chromatic circle of the

hue histogram is divided in 4 sectors (or 2 for templa-

tes with just one sector) separated by the centers of

each template sector and by two borders as illustrated

in the ﬁgure 2.

Each of the 4 sectors deﬁne a different combina-

tion of a template sector and a signal, where the sec-

tors around a template sector center are a mapping to

it.

Figure 2: Example of the chromatic circle division.

3 CRITERION: HARMONIC

DIVERGENCE

We can deﬁne a color attribute operator based on the

harmonic divergence to analyze harmony within the

regions of the image. The trivial operator of the har-

monic divergence is given by the minimum diver-

gence between the harmonic distributions and the co-

lor histogram of the elements within the region:

D(R, f ) = min D

(PkQ

) (14)

The harmonic divergence criterion T

is deﬁned

by T

= (D(R, f ) ≤ p). In other words, the harmonic

divergence D(R, f ) must be lesser or equal p.

Attribute Operators for Color Images: Color Harmonization based on Maximal Harmonic Segmentation

369

4 IMPROVEMENTS FOR COLOR

HARMONIZATION

Finding the maximal regions of an image is a very

interesting propriety and to explore it this work ap-

plied it for harmonic correction. The application

was implemented using the Mathematical Morpho-

logy Library (Beucher, 2014) and tested on images of

the Berkeley Segmentation Dataset and Benchmark

500 (Arbelaez et al., 2011).

As a measure of distance for the color gradient,

a variation of the harmony between two colors (Ou

and Luo, 2006a) was chosen. The use of this distance

had the objective of enhancing the borders between

harmonic regions. This distance was deﬁned as:

(

−CH if CH < 0

0 otherwise,

An use of this gradient can be seen in the ﬁgure 3.

It’s behavior is similar to others colors gradients, but

the border is only enhanced where the color transition

is disharmonic.

(a) Sample “23050” (b) Harmonic gradient

Figure 3: An example of the gradient using the harmonic

distance.

The segmentation is then obtained using the wa-

tershed on the negation of Γ

(f) when varying the

criterion parameter p until the resulting regions are

optimal. This can be automatized by ﬁxing the num-

ber of resulting regions proportional to the number of

segments of a non ﬁltered watershed or by statistical

analysis of the attribute variation.

The ﬁgure 4 exempliﬁes the segmentation result

of a ﬁltered image using the color harmony criterion.

This method was capable of segmenting harmonious

groups independently of size, in this case it was able

to isolate a region that was dominated by the color

pink from the rest of the image.

Then, for each segment, it was chosen the tem-

plate and rotation that minimized the intra-region di-

vergence and the hue values where mapped according

Figure 4: Example of harmonic segmentation.

to them. A modiﬁcation proposed by this paper is

to ﬁnd the borders based in the idea that they divide a

bi-modal section of the histogram, in this sense Otsu’s

method (Otsu, 1975) was used.

The ﬁgure 5 shows an result of the application of

the mapping inside each of the harmonic segments in

comparison with the result of the mapping applied on

the whole image. Because of the segmentation, the

clothes of the guy and the boy on the right were not

mapped to pink. That was not the case of the blue hat

that was mapped in the same harmonic region.

5 CONCLUSIONS

Besides attribute operators were extended to color

images, they became more versatile with the exten-

sion of criteria to color information. Color attribute

operators where recently proposed (Sousa Filho and

Flores, 2017) with two criteria successfully been ap-

plied for improvement of image segmentation.

This works presented color harmony as a color cri-

terion, the use of a harmonic distance in the color gra-

dient and the use of Otsu’s method in the hue map-

ping. Also showed the application of the color attri-

bute operator to improve the color harmonization of

images as it allowed to segment the image into maxi-

mal harmonic regions. In the application, a color cor-

rection applied locally on the segments was capable

of harmonizing the image without creating artifacts

from a global correction.

VISAPP 2019 - 14th International Conference on Computer Vision Theory and Applications

370

(a) Sample “23050” (b) Harmonic mapping (c) Harmonic mapping on maximal

harmonic regions

Figure 5: Comparison of the hue mapping with and without the support of the harmony criterion.

Future works include the proposal of impro-

vements to the implementation of color attribute ope-

rators, like the use of a newer representation (Souza

et al., 2015; Xu et al., 2017) or an optimization of the

opening with the use of Viterbi algorithm (Viterbi and

Omura, 1979) and the proposal of new color criteria

for attribute color processing.

ACKNOWLEDGMENT

First author would like to thank Conselho Nacional de

Desenvolvimento Cient

ıﬁco e Tecnol

ogico (CNPq),

Brazil, for the master scholarship. Second author

would like to thank Conselho Nacional de Desenvol-

vimento Cient

ıﬁco e Tecnol

ogico (CNPq), Brazil, for

the post-doctoral scholarship.

REFERENCES

Arbelaez, P., Maire, M., Fowlkes, C., and Malik, J. (2011).

Contour detection and hierarchical image segmen-

tation. IEEE Trans. Pattern Anal. Mach. Intell.,

33(5):898–916.

Baveye, Y., Urban, F., Chamaret, C., Demoulin, V., and Hel-

lier, P. (2013). Saliency-guided consistent color har-

monization. In CCIW, pages 105–118.

Beucher, S. (2014). Labelling operators. Web publication.

Beucher, S. and Meyer, F. (1992). Mathematical Morpho-

logy in Image Processing, chapter 12. The Morpho-

logical Approach to Segmentation: The Watershed

Transformation, pages 433–481. Marcel Dekker.

Borsotti, M., Campadelli, P., and Schettini, R. (1998).

Quantitative evaluation of color image segmentation

results. Pattern Recognition Letters, 19:741–747.

Breen, E. J. and Jones, R. (1996). Attribute Openings, Thin-

nings and Granulometries. 64(3):377–389.

Busin, L., Vandenbroucke, N., and Macaire, L. (2008). Co-

lor Spaces and Image Segmentation. Advances in Ima-

ging and Electron Physics, 151(07):65–168.

Chevreul, M. E. (1967). The principles of harmony and

contrast of colors,(1839). Reprinted, Van Nostrand

Reinhold, New York.

Cohen-or, D., Sorkine, O., Gal, R., Leyvand, T., and Xu,

Y.-q. (2006). Color Harmonization. 1(212):624–630.

Duda, R. O., Hart, P. E., and Stork, D. G. (2001). Pattern

Classiﬁcation. John Wiley and Sons.

Evans, A. N. and Liu, X. U. (2006). A Morphological Gra-

dient Approach to Color Edge Detection. Image Pro-

cessing, IEEE Transactions on, 15(6):1454–1463.

Heijmans, H. J. A. M. (1999). Introduction to Connected

Operators. In Dougherty, E. R. and Astola, J. T., edi-

tors, Nonlinear Filters for Image Processing, pages

207–235. SPIE–The International Society for Optical

Engineering,.

Itten, J. (1961). Kunst der Farbe. Reinhold Pub. Corp.

Lucchese, L. and Mitra, S. K. (2001). Color Image Segmen-

tation: A State-Of-The-Art Survey. Proceedings of the

Indian National Science Academy (INSA-A). Delhi,

Indian: Natl Sci Acad, 67(2):207–221.

Matsuda, Y. (1995). Color design. Asakura Shoten, 2(4):10.

Meyer, F. (2001). Hierarchies of Partitions and Morpholo-

gical Segmentation. In Scale-Space and Morphology

in Computer Vision, volume 2106 of Lecture Notes in

Computer Science 2106, pages 161–182. Springer.

Moon, P. and Spencer, D. E. (1944). Geometric formulation

of classical color harmony. JOSA, 34(1):46–59.

Attribute Operators for Color Images: Color Harmonization based on Maximal Harmonic Segmentation

371

Munsell, A. H. (1969). A grammar of color. Van Nostrand

Reinhold.

Najman, L. and Talbot, H., editors (2013). Mathematical

Morphology. John Wiley & Sons, Inc., Hoboken, NJ,

USA.

Ostwald, W. (1969). The color primer.

Otsu, N. (1975). A threshold selection method from gray-

level histograms. Automatica, 11(285-296):23–27.

Ou, L. C. and Luo, M. R. (2006a). A colour harmony mo-

del for two-colour combinations. Color Research and

Application, 31(3):191–204.

Ou, L. C. and Luo, M. R. (2006b). A colour harmony mo-

del for two-colour combinations. Color Research and

Application, 31(3):191–204.

Rittner, L., Flores, F. C., and Lotufo, R. A. (2010). A Tenso-

rial Framework for Color Images. Pattern Recognition

Letters, 31(4):277–296.

Ruzon, M. A. and Tomasi, C. (2001). Edge, Junction, and

Corner Detection Using Color Distributions. Pattern

Analysis and Machine Intelligence, IEEE Transacti-

ons on, 23(11):1281–1295.

Serra, J. and Vincent, L. (1992). An Overview of Mor-

phological Filtering. Circuits Systems Signal Process,

11(1):47–108.

Sousa Filho, S. and Flores, F. C. (2017). Attribute operators

for color images: Image segmentation improved by

the use of unsupervised segmentation evaluation met-

hods. In International Symposium on Mathematical

Morphology and Its Applications to Signal and Image

Processing, pages 249–260. Springer.

Souza, R., Rittner, L., Lotufo, R., and Machado, R. (2015).

An array-based node-oriented max-tree representa-

tion. In 2015 IEEE International Conference on

Image Processing (ICIP), pages 3620–3624. IEEE.

Vachier, C. (1995). Extraction de caract

eristiques, segmen-

tation d’image et morphologie math

ematique. PhD

thesis, Ecole des Mines de Paris.

Vachier, C. and Meyer, F. (1995). Extinction Value: A New

Measurement of Persistence. In Proceedings of 1995

IEEE Workshop on Nonlinear Signal and Image Pro-

cessing, volume I, pages 254–257. IEEE.

Viterbi, A. J. and Omura, J. K. (1979). Principles of digital

communication and coding. New York : McGraw-

Hill.

Von Goethe, J. W. (1840). Theory of colours, volume 3. Mit

Press.

Xu, Y., Carlinet, E., Geraud, T., and Najman, L. (2017).

Hierarchical segmentation using tree-based shape spa-

ces. IEEE Transactions on Pattern Analysis and Ma-

chine Intelligence, 39(3):457–469.

Zhang, H., Fritts, J. E., and Goldman, S. A. (2003). An

entropy-based objective evaluation method for image

segmentation. In Storage and Retrieval Methods and

Applications for Multimedia 2004, volume 5307, pa-

ges 38–49.

VISAPP 2019 - 14th International Conference on Computer Vision Theory and Applications

372