REAL TIME CLUSTERING MODEL

J. Cheng, M. R. Sayeh

Department of Electrical and Computer Engineering

Southern Illinois University Carbondale, Carbondale, IL 62901, U.S.A.

M. R. Zargham

Department of Computer Science, Southern Illinois University Carbondale, Carbondale, IL 62901, U.S.A.

Keywords: ODE, Clustering, Vector quantization, Real time.

Abstract: This paper focuses on the development of a dynamic system model in unsupervised learning environment.

This adaptive dynamic system consists of a set of energy functions which create valleys for representing

clusters. Each valley represents a cluster of similar input patterns. The system includes a dynamic parameter

for the clustering vigilance so that the cluster size or the quantizing resolution can be adaptive to the density

of the input patterns. It also includes a factor for invoking competitive exclusion among the valleys; forcing

only one label to be assigned to each cluster. Through several examples of different pattern clusters, it is

shown that the model can successfully cluster these types of input patterns and form different sizes of

clusters according to the size of the input patterns.

1 INTRODUCTION

As stated in (Jain, 1988), "Cluster analysis is the

process of classifying objects into subsets that have

meaning in the context of a particular problem." In

other words, clustering is a process of grouping a set

of unlabeled data. As shown in Figure 1, in general,

clustering can be grouped into two types: non-

overlapping (exclusive) and overlapping

(nonexclusive). In non-overlaping, each object input

will be assigned to only one cluster whereas in

overlapping an object can be assigned to more than

one cluster. In this paper we only consider non-

overlapping clustering. Non-overlapping clustering

could lie either intrinsic or extrinsic. In the intrinsic

approach, also called unsupervised learning, a

proximity matrix is the only criteria used. (Proximity

matrix represents relationship between the objects; if

the objects are patterns such matrix could represent

the distance between the patterns). The extrinsic

approach, also called supervised learning, in

addition to proximity matrix, it also uses category

labels on the objects. To notice the difference

between these two approaches, let’s consider a set of

data representing

health condition of normal and

overweight children. Using intrinsic approach, we

can group these children based on these factors and

then try to determine whether overweight plays a

role in academic status. Taking an extrinsic

approach allow us to study the way of separating

normal and overweight children by considering their

health conditions.

We consider intrinsic approaches only. The

intrinsic methods can be

divided into five types:

hierarchical, partitional, grid-based,

artificial neural

networks, and evolutionary (Jain, 1988; MacQueen,

1967; Grossberg, 1976; and Kohonen, 1982).

Comparison of these clustering methods is hard

to do using simulation because of different

implementation of the methods and the data that is

used. It is also hard to do theoretically comparison

of them because they are almost impossible to model

mathematically (Jain, 1988). Furthermore, the

existing models impose architectural complexity

and/or time complexity which prevent them of

having real time response time. To overcome the

real time mathematical modeling problems, we have

proposed a new method which depends solely on

ordinary differential equations (ODE) (Cheng, 2006).

There is no need for IF/THEN logical statements.

Therefore it can be easily implemented on the

analog type devices to take advantage of high-speed

electronics or photonics technologies.

235

Cheng J., R. Sayeh M. and R. Zargham M. (2008).

REAL TIME CLUSTERING MODEL.

In Proceedings of the Tenth Inter national Conference on Enterprise Information Systems - AIDSS, pages 235-240

DOI: 10.5220/0001694002350240

Copyright

c

SciTePress

Figure 1: Different Clustering Methods.

The organization of this paper is as follows:

Section 2 describes the adaptive dynamical model

and devise an additional state for the dynamics of

vigilance parameter

λ

, which plays an important

role in our system; Section 3 illustrates the

performance of the proposed model on several

examples; Finally, Section 4 presents the conclusion.

2 THE CLUSTERING MODEL

The idea behind our model is to store input patterns

on the surface of an energy

function of a dynamic

system. The clusters are represented as valleys on

the surface of an energy function. To demonstrate

this concept, let’s consider an input U in one

dimensional space as shown in Figure 2. Also let W

be a representative of a cluster and is randomly

placed on the surface of the energy function as

shown in Figure 2 by a black dot. The energy

function is represented by the black line, and the

input U is denoted by a gray dot. As the dynamic

progresses, the W moves toward U and forms a

complete valley when it reaches to U. Similarly, in

cases where there are more than one input pattern,

the valleys are created for patterns that are close to

each other.

Figure 2: One-Dimensional Example of Process of

Clustering forming by our model.

Generally, considering an N dimensional space

for input patterns, the V energy function is

constructed to represent M valleys centered at

locations

(

)

T

Njjjj

wwwW ...,

,2,1

= , for

1=j

to M,

representing M clusters for

P

input patterns. The

V energy function is represented as

∑∑

==

⎥

⎦

⎤

⎢

⎣

⎡

−=

P

p

M

j

jp

xV

1

2

1

γ

Here,

2

||||1

1

jp

j

jp

WU

x

−+

=

λ

where

• is the Euclidean norm,

(

)

T

Njjjj

wwwW ...,

,2,1

= , for

1=j

to M, represent

center of valley j,

p

U

is the input pattern p,

M is the number of clusters, and

P is the number of input patterns.

W and U refer to vectors when N >1.

The constant

γ encourages a number of valleys

(clusters) to be formed and the factor

λ

(vigilance

parameter) approximately reflects the radius of the

generated valley.

In order to invoke competitive exclusion among

the valleys, the following competitive C-energy

function is used

∑∑∑

== ≠

=

P

p

M

i

M

is

spip

xxC

11

This function guarantees only one of the valleys

encodes the patterns by achieving its minimal value.

Based on the above two functions, the total R-

energy function is constructed, which is the

summation of the V-energy function and C-energy

function.

ICEIS 2008 - International Conference on Enterprise Information Systems

236

∑∑∑∑∑

== ≠==

+

⎥

⎦

⎤

⎢

⎣

⎡

−=

P

p

M

i

M

is

spip

P

p

M

j

jp

xxxR

111

2

1

ργ

where ρ is a balancing factor.

The

j

λ

, j= 1,..,M, (vigilance parameter)

approximately reflects the radius of the generated

valley j. As

j

λ

gets smaller, valley j becomes wider.

The

),,...,1( Mjx

jp

=

which has a value between 0

and 1, represents the depth of valley j. As the

distance between

p

U

and

j

W

decreases, the value

of

jp

x

moves toward 1 indicating

j

W

as a cluster

for

p

U

. This model is a gradient dynamical system

with R; thus

)(*

4

2

11

ijip

p

j

M

q

M

qs

spqp

P

p

j

ij

wux

xx

w

R

−

⎥

⎦

⎤

⎢

⎣

⎡

+−−=

∂

∂

∑∑∑

=≠=

γλ

()

.

1

1

1

2

⎟

⎠

⎞

⎜

⎝

⎛

−+

=

∑

=

N

i

ijipj

jp

wu

x

λ

where

jp

x

is explicitly given as

Thus, the dynamics of the system can be expressed

as:

)(*

4

2

11

ijip

p

j

M

q

M

qs

spqp

P

p

j

ij

wux

xx

dt

dw

−

⎥

⎦

⎤

⎢

⎣

⎡

+−=

∑∑∑

=≠=

γλ

Now let’s consider system state for the vigilance

parameter λ so that the cluster size can be adaptive

to the size of the input patterns;

λ

adjust

is a

function which can be expressed as

∑∑

==

⎟

⎟

⎠

⎞

⎜

⎜

⎝

⎛

−=

M

j

P

p

jpj

initial

adjust

x

P

1

2

1

*

λ

γ

λ

λ

Here,

initial

λ

is the initial value of

λ

, usually

specified by the user.

The λ dynamics can be derived as

)(*

*

2

1

2

11 1

∑

∑∑ ∑

=

== =

−

⎥

⎦

⎤

⎢

⎣

⎡

−=

∂

∂

N

k

kjkp

p

j

P

p

M

j

P

q

jqj

initial

j

j

adjust

wux

x

P

λ

γ

λ

λ

λ

λ

3 CLUSTERING PERFORMANCE

This section demonstrates simulation results of our

model on two sets and four sets of 2D input patterns

as shown in figures 3, 5, 7, and 10. In Figure 3, there

are two sets of input patterns where each one

includes10 inputs patterns. In Figure 5, there are two

sets of input patterns, one contains 20 and the other

contains10 inputs. In Figures 7 and 10, there are four

sets of input patterns, containing 5, 10, 15 and 20

inputs. Figures 3, 5, and 7 include 4 Ws while Figure

10 includes 6 Ws. Input patterns are denoted as

squares (gray color) and Ws represented as diamonds

(black color). The coordinate one and coordinate

two represent the first and second coordinates of

input patterns (U) and weights (W). Each set of

inputs are randomly generated having a value

between 0 and 1, while weights are placed at some

specific points. Figures 4(a), 6(a), 8(a), 9(a), and

11(a) demonstrate tracking paths and final positions

of the Ws in figures 3, 5, 7, 7 and 10, respectively.

The small circles represent the final position of Ws;

the ones with grey color represent the center of the

clusters formed by winner Ws; and the ones with

white color represent final position of Ws, which

cannot win the competition. Figures 4(b), 6(b), 8(b),

9(b), and 11(b) are the final contour images of the

energy function for each of these cases. The

surfaces of these images represent the V-energy

function in terms of different input values. Each

generated valley is represented by a set of nested

rings with different shades representing different

levels of valley’s depth (black represents the deepest

point and the dark gray represents the surface).

Figure 4 (a) demonstrates the set of obtained clusters

for initial λ equal to 20, and γ equal to 2. As can be

seen, two Ws win the competition to form clusters

for each set of inputs and the other two Ws move

away from the input patterns. As can be observed

from the contour image in Figure 4 (b), two valleys

are formed representing two different clusters.

REAL TIME CLUSTERING MODEL

237

0 0.2 0.4 0.6 0. 8 1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Dimension One (U & W)

Dimension Two (U & W)

Figure 3: Two clusters; there are 20 input patterns

(represented by squares) and 4

Ws (represented by the

diamonds) positioned at the bottom of the inputs.

0 0.5 1 1.5 2 2.5

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

Dimension One (U & W)

Dimension Two (U & W)

(a) Moving path and final position of Ws.

(b) Contour representation.

Figure 4: Final positions of

Ws and contour representation

of the

V-energy function.

Figure 5 (a) demonstrates the set of obtained clusters

for initial λ equal to 20, and γ equal to 2. As can be

seen, two Ws win the competition and form clusters.

The other two Ws move away. In this simulation, λ

values from initial value 20 change to 15.7, 351.5,

260.2, and 29.3 for each of four Ws. Two winning

Ws have two different λ values 15.7 and 29.3 since

the two sets of input patterns have different size. The

other two runaway Ws have λ values as 351.5 and

260.2. The bigger set of input patterns corresponds

to smaller λ values (resulting bigger cluster), and the

smaller set of input patterns corresponds to bigger λ

values (resulting smaller cluster).

Figure 8 (a) demonstrates the set of obtained clusters

for initial λ equal to 20, and γ equal to 4. As can be

seen, four Ws form clusters for each set of inputs.

After simulation stops, λ values from initial value 20

change to 10.1, 11.9, 29.7, and 17.5 forming four

different size clusters.

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Dimension One (U & W)

Dimension Two (U & W)

Figure 5: Two clusters; there are 30 input patterns

(represented by squares) and 4

Ws (represented by the

diamonds) positioned at the bottom.

0 0.5 1 1.5 2

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

Dimens ion One (U & W )

Dimension Two (U & W)

(a) Moving path and final position of Ws.

(b) Contour representation.

Figure 6: Final positions of

Ws and contour representation

of the

V-energy function.

0 0.2 0.4 0.6 0.8 1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Dimension One (U & W)

Dimension Two (U & W)

Figure 7: Four clusters; there are 50 input patterns

(represented by squares) and 4

Ws (represented by the

diamonds) positioned at the bottom.

ICEIS 2008 - International Conference on Enterprise Information Systems

238

0 0.2 0.4 0.6 0.8 1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Dimens ion One (U & W )

Dimension Two (U & W)

(a) Moving path and final position of Ws.

(b) Contour representation.

Figure 8: Final positions of Ws and contour representation

of the

V-energy function.

Figure 9 (a) demonstrates the same experience as

Figure 8 (a) except in this case the initial value for λ

is set equal to 40. After simulation stops, λ values

from initial value 40 change to 24.2, 30.0, 84.4 and

43.8.

0 0.2 0.4 0.6 0.8 1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Dimension One (U & W)

Dimens ion Two (U & W)

(a) Moving path and final position of Ws

xx

yy

5328 (1772340310682865/2251799813685248-yy)

2

))-8532229875571793/72057594037927936/(1

+

-0.5 0 0.5 1 1.5 2

-0.5

0

0.5

1

1.5

2

(b) Contour representation

Figure 9: Final positions of

Ws and contour representation

of the

V-energy function.

Figure 11 (a) demonstrates the set of obtained

clusters for initial λ equal to 25, and γ equal to 4. As

can be seen, out of 6 Ws, four Ws win and form

clusters. After simulation stops, λ values for these

wining Ws become 13.3, 41.1, 15.9, and 23.4.

0 0.2 0.4 0.6 0.8 1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Dimension One (U & W)

Dimens ion Two (U & W)

Figure 10: Four clusters; there are 50 input patterns

(represented by squares) and 4

Ws (represented by the

diamonds) positioned at the right lower corner.

0 0.5 1 1.5

-0.5

0

0.5

1

Dimension One (U & W)

Dimension Two (U & W)

(a) Moving path and final position of Ws.

(b) Contour representation.

Figure 11: Final positions of

Ws and contour

representation of the

V-energy function.

4 CONCLUSIONS

The main purpose of this paper was the introduction

of a novel dynamical system for clustering which

has potential for real-time device realization. We

also devise an additional system state for the

vigilance parameter λ so that the cluster size or the

quantizing

resolution can be adaptive to the size of

REAL TIME CLUSTERING MODEL

239

the input patterns. This reduces the burden of re-

tuning the vigilance parameter for a given input

pattern set and it will also better represent the input

pattern space. These discussions are furthermore

visualized by simulation examples. As shown from

simulation results, our dynamic system can

successfully cluster different input patterns by

dynamically adjust λ values according to the size of

the input patterns in order to form different sizes of

clusters. In this system, self-organizing properties

can be implicitly coded within the system trajectory

structure using only ODE’s. These ODE’s can be

directly implemented in hardware through feed-back

networks by analog electronic or optical

implementation.

REFERENCES

Block, H.-H., 2002. Clustering Methods: From Classical

Models to New Approaches,

Statistics in Transition,

Vol. 5, No. 5, October 2002, pp. 725-758.

Grossberg, S. 1976. Adaptive pattern classification and

universal recoding, II: feedback, expectation, olfaction,

and illusions, Biological Cybernetics, 23, 187-202.

Jain, A. K., & Dubes, R. C. 1988.

Algorithms for

Clustering Data

. New Jersey: Prentice Hall,

Englewood Cliffs.

Kohonen, T. 1982. Self-organized formation of

topologically correct feature maps.

Biological

Cybernetics

, 43, 59-69.

MacQueen, J. B. 1967. Some methods for classification

and analysis of multivariate observations, Proceedings

of 5-th Berkeley Symposium on Mathematical

Statistics and Probability

, University of California

Press. Berkeley, 1:281-297.

Cheng, J., Sayeh, M.R., & Zargham, M.R. 2006. Neural

Net Based Models for Clustering.

International

Journal of Computational Intelligence Theory and

Practice, 1(2),

91-102.

ICEIS 2008 - International Conference on Enterprise Information Systems

240