STUDY ON IMAGE CLASSIFICATION BASED ON SVM

AND THE FUSION OF MULTIPLE FEATURES

Dequan Zheng, Tiejun Zhao, Sheng Li and Yufeng Li

MOE-MS Key Laboratory of Natural Language Processing and Speech

Harbin Institute of Technology, Harbin 150001,

P. R. China

Keywords:

Image classification, Fusion, Multiple features, SVM.

Abstract:

In this paper, an adaptive feature-weight adjusted image classification method is proposed, which is based on the

SVM and the fusion of multiple features. Firstly, classifier was separately constructed for each image feature, then

automatically learn the weight coefficient of each feature by training data set and the classifiers constructed. At

last, a complexity classifier is created by combining the separate classifier and the corresponding weight

coefficient. The experiment result showed that our scheme improved the performance of image classification and

had adaptive ability comparing with general approach. Moreover, the scheme has certain robustness because of

avoiding the impact brought by various dimension of each feature.

1 INTRODUCTION

With the development of information and

multimedia technology, image has become an

important component of WWW content. It is

necessary to develop intelligent system to retrieve

and categorize images based on their visual contents.

In computer vision domain, image retrieval and

categorization have become researching hot pot for

recent several years (

Vailaya, Figueiredo, etc. 2001).

Different from traditional image retrieval issue,

image categorization can be well pre-defined in

specific domain and specific categories, such as

sports image categorization in the Olympic Games.

Specifically, image categorization is to give label

information for each image. For example, if we

classify images of the Olympic Games, the system

can automatically label each image with sports

categories information, such as swimming,

volleyball, lifting, table tennis, gymnastics, etc. For

image retrieval, image categorization not only can

meet user’s semantic retrieval demand by category-

specific search, but also can reduce retrieval time by

weeding out irrelevant images in the process of

retrieval.

At present, it is the main method of image

categorization that extracting visual features by

image analysis technology firstly, then building

classification model by machine learning algorithm,

at last predicting semantic label information by

classification model. Youna integrated both texture

and four color features to express each image and

implemented sports image categorization by

Bayesian classifier (

Youna, Eenjun, etc. 2004). Chang

and Goh adopted SVM and global visual feature to

implement image categorization (

Chang, Goh, etc.

2003

). As a result of image data’s characteristic,

image visual feature's extraction and expression are

important barriers, which restrict the performance of

image categorization. Generally speaking, one kind

of visual feature isn’t able to express image contents

well. Therefore, researchers usually extract several

kinds of visual feature to express image contents.

But each feature has different importance in an

image. When we adopt several kinds of visual

feature to express image content, there is an obvious

shortcoming if we simply combine multiple visual

features into a feature vector. It is not nimble to

adjust influence coefficient of each feature to image

classification, this shortcoming seriously influences

the effect of image classification.

In this paper, we proposed an adaptive feature-

weight adjusted image categorization algorithm

based on the SVM and the fusion of multiple

features. The algorithm is able to automatically learn

feature-weight coefficient of each feature, which

solves the shortcoming of combining multiple visual

features into a feature vector and improves the

performance of image categorization.

The rest of this paper is organized as follows:

Zheng D., Zhao T., Li S. and Li Y. (2009).

STUDY ON IMAGE CLASSIFICATION BASED ON SVM AND THE FUSION OF MULTIPLE FEATURES.

In Proceedings of the 11th International Conference on Enterprise Information Systems - Artiﬁcial Intelligence and Decision Support Systems, pages

81-85

DOI: 10.5220/0001864200810085

 SciTePress

Section 2 gives a brief introduction of SVM and

multi-class classifier. Section 3 presents our image

categorization algorithm based on the SVM and the

fusion of multiple features. Section 4 shows the

experimental result of our algorithm. Finally this

paper concludes in Section 5.

2 SVM AND MULTI-CLASS

CLASSIFIER

SVM is a very effective binary classification

algorithm (

Burges, 1998), it aims at separating two

classes of training samples in feature space by an

optimal hyper-plane, where the maximum geometric

margin gains.

Given a binary classification problem, where

is a n-dimension vector and

y is the label of the

class that the vector belongs to:

{( , )} . . 1, 2,3 , { 1, 1}

ii i

xy sti Ny==+−

(1)

The searching for optimal separating hyperplane

0=+ bxw

is to maximize the geometric margin

in vector space, subject to:

()1

ywx b+≥

(2)

The solution can be found through a Wolfe dual

problem with Lagrangian multiplied

111

() ( )

NNN

iijijij

iij

Qa yy x x

ααα

===

=− ⋅

∑∑∑

(3)

Subject to 0≥

and 0

∑

In the dual format, the data points only appear in

the inner product. To get a potentially better

representation of the data, the data points are

mapped into the Hilbert inner product space through

a replacement:

() ( ) (, )

ij i j ij

xxxKxx

⋅→ ⋅ =

(4)

Where

()

⋅k is a kernel function. Then we get the

kernel version of the Wolfe dual problem:

111

() ( )

NNN

iijijij

iij

Qa yyK x x

ααα

===

=− ⋅

∑∑∑

(5)

Thus for a given kernel function, the SVM classifier

is given by

() sgn( ( ,) )

ii i

xyKxxb

∗

∑

(6)

The output of the SVM classifier is the binary

classification result. The further the data point is

away from the hyperplane, the more confident the

classifying result on this data point is.

However, as a binary classification algorithm,

SVM would meet problems when comes to multiple-

class image classification problem. Many methods

have been proposed to combine binary classifiers to

form a multi-class classifier. Traditionally, two

replacements are adopted when it comes to multiple-

class situation.

(1) One-against-All: For each category, training

an intra-class classifier which indicates the sample’s

confidence level of its category membership. Then

normalizing their outputs. For each test sample,

these classifiers of all categories are used to gain

outputs, in which the category with the highest

output is selected as the classification result.

(2) One-against-One: For each category, training

n-1 inter-class classifiers between current category

and other categories (supposing there are n

categories). Then normalizing their outputs. For

each test sample, the confidence level of each

category is the majority voting result of these

classifiers. The category with the highest sum of

outputs is selected as the classification result. It

combines the output of all classifiers to form an

integrated prediction.

Hsu and Lin proved that the One-against-One

method had better performance than the one-against-

all method by experiment and theoretical analysis

(

Hsu and Lin, 2002). In this paper, we adopt the One-

against-One method when we construct SVM multi-

class classifier based on individual visual feature.

3 IMAGE CATEGORIZATION

METHOD

The core idea of the image categorization algorithm

based on SVM and the fusion of multiple features is:

in the training stage, we firstly construct SVM multi-

class classifier based on individual visual feature

(such as color feature, texture feature and shape

feature) by the One-against-One method (

Wu, Lin,

etc. 2004

), then the algorithm automatically learns

the weight coefficient of each feature by feature-

weight learning module, which predicts the category

information of every image in training dataset

STUDY ON IMAGE CLASSIFICATION BASED ON SVM AND THE FUSION OF MULTIPLE FEATURES

through the constructed classifiers; in the predicting

stage, we firstly gain the confidence level value of

the image belonging to each category by every SVM

multi-class classifier based on individual visual

feature, then we calculate the confidence level value

of the image belonging to each category by

combining each confidence level value gained and

corresponding feature-weight coefficient, the

category which gains the greatest confidence level

value is the image's category according to the least

error principle.

The image categorization algorithm proposed in

this paper is composed of the model training

algorithm, the learning algorithm of feature-weight

and the predicting algorithm.

3.1 Model Training Algorithm

(1) Input:

training samples of image category

SSS ,,,

…

(2) Training SVM multi-class classifier based on

color feature

color

SVM

by One-against-One

method.

(3) Training SVM multi-class classifier based on

texture feature

texture

SVM

by One-against-One

method.

(4) Training SVM multi-class classifier based on

shape feature

shape

SVM

by One-against-One

method.

(5) Output:

shapetexturecolor

SVMSVMSVM ,,

3.2 Learning Algorithm of

Feature-Weight

(1) Input:

shapetexturecolor

SVMSVMSVM ,,

, training samples of

image category

SSS ,,,

…

(2) Initializing:

shapeweighttextureweightcolorweight _,_,_

with zero.

(3) Predicting category information with

shapetexturecolor

SVMSVMSVM ,,

for each image in

training samples.

(4) If

color

SVM

gives correct category information,

++colorweight _

; if

texture

SVM

gives correct

category information,

textureweight _

; if

shape

SVM

gives correct category information,

++shapeweight _

(5) Normalizing:

shapeweighttextureweightcolorweight _,_,_

(6) Output:

shapeweighttextureweightcolorweight _,_,_

3.3 Predicting Algorithm

(1) Input:

color-feature-vector, texture-feature-vector,

shape-feature-vector.

(2) Gaining confidence level value of the image

belonging to each category by color-feature-

vector and

color

SVM

pcolorpcolorpcolor _,,_,_



(3) Gaining confidence level value of the image

belonging to each category by texture-feature-

vector and

texture

SVM

ptextureptextureptexture _,,_,_



(4) Gaining confidence level value of the image

belonging to each category by shape-feature-

vector and

shape

SVM

pshapepshapepshape _,,_,_



(5) Calculating the confidence level value of the

image belonging to each category by combining

each confidence level value gained and

corresponding feature-weight coefficient.

ppp ,,,

…

pshapeshapeweightptexture

textureweightpcolorcolorweightp

___

×+

(6)

},,,max{arg

21 n

pppLabel 

(7) Output:

Image category label.

4 EXPERIMENT RESULT

AND DISCUSSION

Our image database is consisted of 8 sports

categories with each 100 images: soccer, basketball,

tennis, volleyball, table tennis, swimming, weight

lifting, gymnastics, shooting, and jiujitsu. Then, this

database is divided into 2 sub-databases, one for

training and the other for test. Both of them contain

8 categories with each 50 images.

Two experiments are conducted to verify the

efficiency of our algorithm.

(1) General method: combining color feature,

texture feature and shape feature into one feature

vector, training one SVM multi-class classifier.

(2) Our scheme: firstly, constructing SVM multi-

class classifier on each individual feature, secondly,

ICEIS 2009 - International Conference on Enterprise Information Systems

learning the weight coefficient of each feature by

training dataset and the classifiers constructed,

finally combining each SVM multi-class classifier

based on individual feature and the corresponding

weight coefficient into a complexity classifier.

Since our goal is to verify the efficiency of

image categorization algorithm we proposed, the

feature extraction method is simple and

straightforward, where color histogram (

Panchanathan,

Park, etc. 2000

), texture co-occurrence (Haralick,

Shanmugam, etc. 1973

) and shape invariant moment

(

Yao and Zhang, 2000

) are extracted for each image.

In the experiment, we use binary classifier in

LibSVM tool kits as core classifier to construct

SVM multi-class classifier (

Chang and Lin). Three

criterions are considered: precision, Recall, F-score.

Furthermore, in order to more clearly compare the

two methods, we compute macro-precision, macro-

recall, macro-F-score for each method.

Table 1: Results of general method.

Cat. B So V T

Precision 87.8% 93.5% 83.3% 100%

Recall 86% 86% 70% 88%

F-score 86.9% 89.6% 76.1% 93.6%

Cat. Sw Wl G Sh

Precision 95.7% 71.2% 81.3% 81.0%

Recall 90% 94% 78% 94%

F-score 92.8% 81.0% 79.6% 87.0%

Table 2: Results of our method.

Cat. B So V T

Precision 97.6% 93.3% 80.7% 100%

Recall 84% 84% 84% 86%

F-score 90.3% 88.4% 82.3% 92.5%

Cat. Sw Wl G Sh

Precision 90.3% 79.0% 80% 79.6%

Recall 94% 98% 80% 86%

F-score 92.1% 87.5% 80% 82.7%

Table 3: Comparison of the two methods.

Items Macro-

Precision

Macro-

Recall

Macro-F-

score

General

method

86.73% 86% 86.2%

Our

method

87.56% 87% 87.3%

As demonstrated in table 1 and table 2, our

method improves the performance of image

categorization comparing with general method. And

this improvement is quantitatively demonstrated by

their macro-precision, macro-recall and macro-F-

score comparison in table 3. Furthermore, the

method proposed in this paper has an obvious

advantage of automatically learning each feature’s

feature-weight, so that it has certain adaptive

capacity and adjusted ability, when it trains and

predicts other categories of images.

5 CONCLUSIONS

In this paper, an image categorization algorithm is

proposed to address the shorting of combining all

the features into one feature vector. The algorithm

firstly constructs SVM classifiers based on

individual feature and automatically learns each

feature’s weight coefficient, then combines SVM

classifiers and corresponding weight coefficient into

a complexity classifier. As demonstrated in the

experiments, our method improves the performance

of image categorization.

ACKNOWLEDGEMENTS

This work is supported by the national natural

science foundation of China (No. 60736044) and the

National High-Tech Development 863 Program of

China (No.2006AA010108).

REFERENCES

Vailaya, A., Figueiredo, M. A. T., Jain, A. K., Zhang, H.

J., 2001. Image Classification for Content-Based

Indexing. IEEE Transactions on Image Processing,

Vol.10, 117-130.

Youna, J., Eenjun, H., Wonil, K., 2004. Sports Image

Classification through Bayesian Classifier. Lecture

Notes in Computer Science, Vol.3040, 546-555.

Chang, E., Goh, K., Sychay, G., Wu, G., 2003. CBSA:

Content-Based Soft Annotation for Multimodal Image

retrieval using Bayes Point Machines. IEEE

Transactions on Circuits and Systems for Video

Technology, Vol.13, 26-38.

Burges, J. C., 1998. A Tutorial on Support Vector

Machines for Pattern Recognition. Data Mining and

Knowledge Discovery, Vol.2, 121-167.

Hsu, C. W., Lin, C. J., 2002. A Comparison of Methods

for Multi-class Support vector Machines [J]. IEEE

Transactions on Neural Networks, Vol.13, 415-425.

Wu, T. F., Lin, C. J., Weng, R. C., 2004. Probability

Estimates for Multi-class Classification by Pairwise

Coupling. Journal of Machine Learning Research,

Vol.5, 975-1005.

Panchanathan, S., Park, Y. C., Kim, K. S., Golshani, F.,

STUDY ON IMAGE CLASSIFICATION BASED ON SVM AND THE FUSION OF MULTIPLE FEATURES

2000. The Role of Color in Content-Based Image

Retrieval. International Conference on Image

Processing, Vol.1, 517-520.

Haralick, R. M., Shanmugam, K., Dinstein, I., 1973.

Texture Features for Image Classification. IEEE

Transaction on System Man Cybern, Vol. 3, 610-621.

Yao, Y. R., Zhang, Y. J., 2000. Shape-based image

retrieval using Wavelet and Moment. Journal of Image

and Graphics, Vol. 5, 206-210.

Chang, C. C., Lin, C. J. LIBSVM: A Library for Support

Vector Machines. Software available at http://

www.csie.ntu.edu.tw/~cjlin/libsvm

ICEIS 2009 - International Conference on Enterprise Information Systems