ARTIFICIAL IMMUNE FILTER FOR VISUAL TRACKING

Alejandro Carrasco E. and Peter Goldsmith

Mechanical Enginering, University of Calgary, 2500 University Drive NW, Calgary, Canada

Keywords: Filtering, visual tracking, immune system, artificial intelligence.

Abstract: Visual tracking is an important part of artificial Vision for robotics. I

t allows robots to move towards a

desired position using real world information. In this paper we present a novel particle filtering method for

visual tracking, based on a clonal selection and a somatic mutation processes used by the natural immune

system, which is excellent at identifying intrusion cells; antigens. This capability is used in this work to

track motion of the object in a sequence of images.

1 INTRODUCTION

Artificial intelligence has found a source of ideas

borrowed from biological systems such as swarms,

ant colonies, neural networks, genetic algorithms

and immune systems. They have been successfully

used in many different areas: control (Macnab,

2000), optimization (Charbonneau, 2002), pattern

recognition (Tashima, 2001), robotics (Ramirez-

Serrano, 2004) and prediction (Connor, 1994). The

immune system is composed of a complex

constellation of cells, organs and tissues, arranged in

an elaborate and dynamic communications network

and equipped to optimize the response against

invasion by pathogenic organisms. The immune

system is, in it simplest form, a cascade of detection

and adaptation culminating in a system that is

remarkably effective, most of the time. It has many

facets, a number of which can change to optimize

the response to these unwanted intrusions (Dasgupta,

2002). The immune system has a series of dual

natures, the most important of which is self - non-

self recognition. The others are: general - specific,

natural - adaptive, innate - acquired, cell_mediated -

humoral, active – passive and primary - secondary.

Parts of the immune system are antigen-specific

(they recognize and act against particular antigens),

systemic (not confined to the initial infection site,

but work throughout the body), and have memory

(recognize and mount an even stronger attack to the

same antigen the next time) (Gilbert, C. , 1994). It

can recognize and remember millions of different

enemies, and it can produce secretions and cells to

match up with and wipe out each one of them. The

secret to its success is an elaborate and dynamic

communications network (de Castro, 2002). Millions

and millions of cells, organized into sets and subsets,

gather like clouds of bees swarming around a hive

and pass information back and forth. The key to a

healthy immune system is its remarkable ability to

distinguish between the body’s own cells and

foreign cells (Bergstrom, 2004). The body’s immune

defences normally coexist peacefully with cells that

carry distinctive “self” marker molecules. But when

immune defenders encounter cells or organisms

carrying markers that say “foreign,” they quickly

launch an attack. In this work, we use the intruder

detection capability of artificial immune systems in

order to track the object in a sequence of images.

2 VISUAL TRACKING

Visual tracking is the action of consistently locating

a desired feature in each image of an input sequence.

The problem is typically complicated by sensor

noise, motion in the scene, motion on the part of the

observer and real-time constraints. The problem can

be further complicated when more than one identical

feature must be tracked, in which case it is up to the

observer to decide the optimal set of

correspondences which are consistent with a priori

assumptions about, and recent observations of, the

behavioural characteristics of the features (Prassler,

1990)(Carlsson, 1990). Given an image

, the problem is to track a sub-image

(object). In a sequence of images the object will be

+

ℵ∈jijiI ,),(

280

Carrasco E. A. and Goldsmith P. (2007).

ARTIFICIAL IMMUNE FILTER FOR VISUAL TRACKING.

In Proceedings of the Fourth International Conference on Informatics in Control, Automation and Robotics, pages 280-285

DOI: 10.5220/0001630102800285

Copyright

c

SciTePress

in different positions, moving in a determined

pattern. Therefore the prediction part of the filter is

needed to predict where the object I(u,v) will be in

the image I(i,j), giving a region of interest to

accelerate the processing of recognizing the object.

Recognizing the object by filtering the clutter and

noise due to change of illumination, shadows, etc. is

the second part of the filter. The use of filters such

as the Kalman filter (Gutman, 1990)(Welch, 2001),

which is based in optimal prediction for linear

system and noise with Gaussian distribution, are

excellent tools to overcome the problems in visual

tracking. Extensions of the Kalman filter for non-

linear systems have been developed such as

Extended Kalman filter (Ribeiro, 2004) and

Unscented Kalman filter (Jeffrey, 1997). Another

algorithm of interest is the condensation

(Conditional Density Propagation) (Isard, 1998),

which is based on computing the Bayes’ rule to a set

of particles (particle filtering). In general the filters

mentioned above can be seen as Bayesian filters,

where the following density distributions are needed

(Isard, 1996) (Grewal, 1993):

)|(

kk

Zxp

: A posteriori density given the measurement.

)|(

1−kk

Zxp

: A priori density.

)|(

1−kx

xxp

: Process density describing the dynamics.

)|(

kk

xzp

: Observation density

Bayes’ Rule is

∫

∫

−

−−−

=

kkkkk

kkkkkkk

kk

dxZxpxzp

dxZxpxxpxzp

Zxp

)|()|(

)|()|()|(

)|(

1

111

(1)

One of the drawbacks in these algorithms is the

assumption of priori density distribution, Gaussian

distribution such in the case of Kalman filter.

Particle filters use Bayes (equation 1) and Monte

Carlo method to approximate the sequence of

probability distribution; these required a large

number of particles to converge towards the

probability distribution. Therefore, the random

sampling is the main drawback, due to in case that

the population is not drawn to represent some of its

statistical features makes a wrong estimation.

Besides, due to the degeneration of the particles

through time, re-sampling mechanisms are used. In

the next section we introduce an artificial immune

system to filter noisy signals and predict the state of

a system.

3 ARTIFICIAL IMMUNE FILTER

(AIF): CLONAL SELECTION

AND SOMATIC MUTATION

The clonal selection theory, by immunologist Frank

Macfarlane Burnet (Burnet, 1978), models the

principles of an immune system. When an antigen is

present in our body, the B-Lymphocyte cells

produce antibodies Ab receptors. Each B cell has a

specific antibody as a cell surface receptor. The

arrangement and generation of antibody genes

occurs prior to any exposure to antigen. When a

soluble antigen is present, it binds to the antibody on

the surface of B cells that have the correct

specificity. These B cell clones develop into

antibody-producing plasma cells or memory cells.

Only B cells, which are antigen-specific, are capable

of secreting antibodies. Memory cells remain in

greater numbers than the initial B cells, allowing the

body to quickly respond to a second exposure of that

antigen, as show in Figure 1 (de Castro, 2002).

Figure 1: Clonal Selection Principle.

The higher affinity comes from a mechanism

that alters the variable regions of the memory cells

by specific somatic mutation. This is a random

process that by chance can improve antigen binding.

This same principle is the inspiration in this work to

produce an artificial immune filter. Initial set of n B-

cells (particles) , representing the

features of our object to track (positions, velocity,

etc), weights , representing its

affinity between the antigens and the antibodies, and

memory cells , are created. In the

beginning our best affine cell to our antigen is our

),...,,(

11 n

xxxX =

),...,,(

21 n

wwwW =

),...,,(

21 m

sssS =

ARTIFICIAL IMMUNE FILTER FOR VISUAL TRACKING

281

initial condition. Therefore we clone and slightly

mutate the cell, using equation (2)

i

k

best

k

i

k

rxx

α

+=

(2)

where r is a random variable normally distributed

and

)1,0(~ Nr ℜ∈

α

is a small constant. The

affinity is integrated by two distance

measurements from our best B cell, before and after

prediction. Equation (3) is the first part of affinity

i

w

(

)

i

k

best

k

i

xxaf −−= exp

1

(3)

The next step k+1 is the prediction part, given by

for nonlinear dynamics and by

for linear dynamics, where A is known

as the transition matrix. After all the cells have been

through the dynamic system, it is time to obtain a

new measurement , which contains a certain level

of noise. Then we apply equation (4) to obtain the

second part of our affinity measurement, where H is

the observation model in the case of a linear system

and

)(

1

i

k

i

k

xfx =

+

i

k

i

k

Axx =

+1

k

z

β

is a constant.

(

)

i

kk

i

Hxzaf

12

exp

+

−−=

β

(4)

ii

i

afafw

21

+=

(5)

Equation (5) calculates the affinity of each B-cell to

the antigen. The m best cells with high affinity will

conform to our memory cells, and the highest

affinity will be the estimation

1+k

x

and our next best

B-cell .

best

k

x

1+

3.1 Application of Artificial Immune

Filter to Noise Rejection

Before applying the artificial immune system to

visual tracking, the filter was tested on a noisy signal

and compared to a Kalman filter. The signal

represents the antigen to be recognized. The best B-

cell that binds the antigen is the estimation of the

state of the signal. The next stage is choosing the

parameter for mutation,

α

. Since the level of

somatic mutation for the cells is a slight change on

our best B-cell, a value equal or less than dt value,

the step time of the system, is a good option,

because it indicates that B-cells could vary

(0.01 for this example) from their real values.

Given a linear stochastic difference equation in the

next form

dt±

kkkk

wbuAxx +

+

=

+1

(6)

kkk

vHxz

+

=

(7)

where

⎥

⎦

⎤

⎢

⎣

⎡

−

=

1

1

dt

dt

A

⎥

⎦

⎤

⎢

⎣

⎡

=

0

0

b

[]

01=H

0

=

k

u

Noise is modelled by

⎥

⎦

⎤

⎢

⎣

⎡

⋅⋅

⋅⋅

=

k

k

k

rdt

rdt

w

15.0

15.0

2

(8)

[

]

)sinh(05.01.0

kk

rv

+

=

(9)

Equation (9) introduce a heavily spike noise with

non zero mean, while equation (8) is a normal

distribution, and r is random noise. Figure 2 shows

the measured position with noise up to 50% of its

maximum value.

Figure 2: Measured position.

Using the proposed algorithm of Figure 3, we

obtained the filtered signal in Figure 4.

ICINCO 2007 - International Conference on Informatics in Control, Automation and Robotics

282

Figure 3: Pseudo-code for Artificial Immune Filter.

Figure 4: Performance of Filters.

It is well known that the uncertainty of the

covariance parameters of the process noise, Q, and

the observation errors, R, has a significant impact on

Kalman filtering performance. Q and R influence the

weight that the filter applies between the existing

process information and the latest measurements.

Errors in any of them may result in the filter being

suboptimal or even cause it to diverge. The

conventional way of determining Q and R requires

good a priori knowledge of the process noises and

measurement errors, which normally comes from

intensive empirical analysis. Besides of the errors

due to covariance parameters, the Kalman filter is

based on the assumption of normal distribution noise

with zero mean. Figure 4 shows the real signal with

no noise and the filtered signal. It can be seen that

the filter affectively attenuated the noise. In this

example we use the following parameter settings for

the Kalman filter,

⎥

⎦

⎤

⎢

⎣

⎡

=

00015.0015.0

015.000015.0

Q

0025.0=R

QP

=

The parameter settings for the AIF were, n=20,

m=5,

[

]

01.001.0001.0

=

α

,

1.0=

β

.

3.2 Visual Tracking using AIF

Tracking an object in a sequence of images is a

challenging problem. An elementary tracking

approach could be to fit a curve, (contour of an

object) to each image in a sequence, and an

estimated curve is therefore required for each image.

Then a fitted curve from one image is the estimation

for the next image. This kind of algorithm will be

affected by fast motion and become sensitive to

distractions. Clutter in the background, either static

or dynamic, noise of the sensor and change of

illumination, are some factors to consider as noise in

an image (Healey, 1994). The tracking performance

can be greatly improved by a filter able to predict

and correct the fitted curve, removing the noise from

the image. Our artificial immune filter is used in this

section to track an object in a sequence of images.

The extension of the artificial immune filter from

single variable to multivariable is straightforward.

The contour of the object is a parametric curve

(

)

(

)

(

)

tIytIxtc ,)(

=

[]

Lt ,0∈

(10)

where t is an independent parameter over the

interval [0,L], and Ix(t) and Iy(t) are known as spline

functions (Foley, 1990). An important aspect to

achieve real time tracking performance has been the

restriction of measurements of the set of

observations Z to a sparse set of lines normal to the

contour of the object, as shown in Figure 5. In this

case the affinity is given by

()

(

)

∑

=

−−=

P

j

i

k

j

k

i

tCzaf

1

2

exp

(11)

where P is the number of searching lines and is

the edge closest to the hypothetical contour

j

k

z

(

)

tC

i

k

.

ARTIFICIAL IMMUNE FILTER FOR VISUAL TRACKING

283

Figure 5: Normal lines of object contour to search the

observation z

i

.

Figure 6 shows a sequence for fast tracking motion

of a ball with clutter added to background. This

experiment used 100 cells and 10 memory cells, in

real time (30 frames per second). In spite of the fast

motion of the ball, the tracker never loses contact

with the ball in a sequence of image, when we

bounced the ball several times against the wall. The

tracker shows the center of the ball with white dots.

Figure 6: Tracking a fast motion ball.

Figure 7 is a group of snapshots from a tracking

sequence of the ball under heavy clutter, dynamic

background and partial occlusion.

Figure 7: Tracking using Artificial Immune Filter.

4 CONCLUSIONS

In this work we introduced a novel filter using a

clonal selection and somatic mutation model of

immune system. The filter does not require

probability distributions or re-sampling, unlike other

particle filters. The artificial immune filter was

tested for signal processing and visual tracking,

showing good performance in both applications. In

the application of visual tracking of the ball, the

filter was able to track fast ball motion in a non-

smooth trajectory (bouncing) and clutter in the

background. Future work will include the adaptation

of parameters and tracking of several objects.

ACKNOWLEDGEMENTS

We would like to thanks to CONACyT for its

support in this research.

REFERENCES

Macnab, C.J.B., D'Eleuterio, G.M.T., May 2000, Discrete-

time Lyapunov design for neuro-adaptive control of

elastic-joint robots, International Journal of Robotics

Research vol. 19, no. 5, pp. 511-525.

Paul Charbonneau, 2002, An Introduction to Genetic

Algorithms for Numerical Optimization, National

Center for Atmospheric Research, Boulder, Colorado.

Koichi Tashima, Zheng Tang, Okihiko Ishizuka, Koichi

Tanno, 2001, An immune network with interactions

ICINCO 2007 - International Conference on Informatics in Control, Automation and Robotics

284

between B cells for pattern recognition. Scripta

Technica, Syst Comp Jpn, 32(10): 31-41.

Ramirez-Serrano, A., and Pettinaro G.C., 2004, "Origami

robotics: swarm robots that can fold into diverse

configurations", Mechatronics2004, pp. 171-182,

Ankara, Turkey, August 30 - September 1.

Jerome T. Connor, R. Douglas Martin, 1994, Recurrent

Neural Networks and Robust Time Series Prediction.

IEEE Transactions on Neural Networks, Vol. 5 , No.

2.

Dasgupta, D., Gonzalez, F., 2002, An immunity-based

technique to characterize intrusions in computer

networks, IEEE Trans. Evol. Comput. 6:1081–1088.

Gibert, C. J. & Routen, T. W. 1994, Associative memory

in an immune-based system. In: Proceedings of the

Twelfth National Conference on Artificial Intelligence

pp. 852–857, Cambridge, MA. AAAI Press/The MIT

Press.

de Castro, L. N. and Von Zuben, F. J., 2002, Learning and

optimization using the clonal selection principle, IEEE

Trans. Evol. Comput. Special issue on artiﬁcial

immune systems, 6:239–251.

Carl T. Bergstrom, Rustom Antia, 2004, How do adaptive

immune systems control pathogens while avoiding

autoimmunity? Department of Biology, University of

Washington.

E. Prassler, J. Scholz, M. Schuster, and D. Schwammkrug,

June 1998, Tracking a large number of moving object

in a crowded environment, Proceedings of the IEEE

Workshop on Perception for Mobile Agents (Santa

Barbara) (G. Dudek, M. Jenkin, and E. Milios, eds.),

pp. 28-36.

S.Carlsson and J.O. Eklundh, 1990, Object detection using

model based prediction and motion parallax,

Proceedings of the European Conference on Computer

Vision (Antibes), Springer-Verlag, pp. 297-206.

Gutman, P., Velger, M., 1990, Tracking Targets Using

Adaptive Kalman Filtering, IEEE Transactions on

Aerospace and Electronic Systems Vol. 26, No. 5: pp.

691-699.

Welch, G, Bishop, G., 2001, An Introduction to the

Kalman Filter, ACM, IGGRAPH 2001, Course8,

www.cs.unc.edu/welch/kalman.

Maria Isabel Ribeiro, 2004, Kalman and Extended Kalman

Filters: Concept, Derivation and Properties. Institute

for Systems and Robotics. Lisboa PORTUGAL.

Simon J. Julier Jeffrey K. Uhlmann, 1997, A New

Extension of the Kalman Filter to Nonlinear Systems.

The Robotics Research Group, Department of

Engineering Science, The University of Oxford.

M. Isard and A. Blake, 1998, CONDENSATION:

Conditional Density Propagation for visual tracking,

International Journal of Computer Vision, Vol. 29, No.

1, pp. 5-28.

M. Isard and A. Blake, 1996, Contour tracking by

stochastic propagation of conditional density. In

European Conference on Computer Vision, volume 1,

pages 343–356.

M. S. Grewal and A. P. Andrews, 1993, Kalman Filtering.

Prentice Hall.

Burnet, F. M., 1978, “Clonal Selection and After”, In

Theoretical Immunology, (Eds.) G. I. Bell, A. S.

Perelson & G. H. Pimbley Jr., Marcel Dekker Inc., 63-

85.

Leandro Nunes de Castro, Fernando J. Von Zuben, 2002,

The Clonal Selection Algorithm with Engineering

Applications. School of Electrical and Computer

Engineering, State University of Campinas, Brazil.

G. Healey, R. Kondepudy, 1994,"Radiometric CCD

camera calibration and noise estimation", IEEE

Transactions on Pattern Analysis and Machine

Intelligence, Vol. 16, No. 3, pp. 267-276.

Foley, J., van Dan, A., Feiner, S., and Hughes, J.,

1990,Computer Graphics: Principles and Practice,

chapter 13, 563-604, Addison-Wesley.

ARTIFICIAL IMMUNE FILTER FOR VISUAL TRACKING

285