MULTI-REGULARIZATION PARAMETERS ESTIMATION
FOR GAUSSIAN MIXTURE CLASSIFIER BASED
ON MDL PRINCIPLE
Xiuling Zhou
1,2
, Ping Guo
1
and C. L. Philip Chen
3
1
The Laboratory of Image Processing and Pattern Recognition, Beijing Normal University, Beijing, 100875, China
2
Artificial Intelligence Institute, Beijing City University, Beijing, China
3
The Faculty of Science & Technology, University of Macau, Macau SAR, China
Keywords: Gaussian classifier, Covariance matrix estimation, Multi-regularization parameters selection, Minimum
description length.
Abstract: Regularization is a solution to solve the problem of unstable estimation of covariance matrix with a small
sample set in Gaussian classifier. And multi-regularization parameters estimation is more difficult than
single parameter estimation. In this paper, KLIM_L covariance matrix estimation is derived theoretically
based on MDL (minimum description length) principle for the small sample problem with high dimension.
KLIM_L is a generalization of KLIM (Kullback-Leibler information measure) which considers the local
difference in each dimension. Under the framework of MDL principle, multi-regularization parameters are
selected by the criterion of minimization the KL divergence and estimated simply and directly by point
estimation which is approximated by two-order Taylor expansion. It costs less computation time to estimate
the multi-regularization parameters in KLIM_L than in RDA (regularized discriminant analysis) and in
LOOC (leave-one-out covariance matrix estimate) where cross validation technique is adopted. And higher
classification accuracy is achieved by the proposed KLIM_L estimator in experiment.
1 INTRODUCTION
Gaussian mixture model (GMM) has been widely
used in real pattern recognition problem for
clustering and classification, where the maximum
likelihood criterion is adopted to estimate the model
parameters with the training samples (Bishop, 2007)
(Everitt, 1981). However, it often suffers from small
sample size problem with high dimensional data. In
this case, for d-dimensional data, if less than d+1
training samples from each class is available, the
sample covariance matrix estimate in Gaussian
classifier is singular. And this can lead to lower
classification accuracy.
Regularization is a solution to this kind of
problem. Shrinkage and regularized covariance
estimators are examples of such techniques.
Shrinkage estimators are a widely used class of
estimators which regularize the covariance matrix by
shrinking it toward some positive definite target
structures, such as the identity matrix or the diagonal
of the sample covariance (Friedman, 1989);
(Hoffbeck, 1996); (Schafer, 2005); (Srivastava,
2007). More recently, a number of methods have
been proposed for regularizing the covariance
estimate by constraining the estimate of the
covariance or its inverse to be sparse (Bickel, 2008);
(Friedman, 2008); (Cao, 2011).
The above regularization methods mainly
concern various mixture of sample covariance
matrix, common covariance matrix and identity
matrix or constraint the estimate of the covariance or
its inverse to be sparse. In these methods, the
regularization parameters are required to be
determined by cross validation technique. Although
the regularization methods have been successfully
applied for classifying small-number data with some
heuristic approximations (Friedman, 1989);
(Hoffbeck, 1996), the selection of regularization
parameters by cross validation technique is very
computation-expensive. Moreover, cross-validation
performance is not always well in the selection of
linear models in some cases (Rivals, 1999).
Recently, a covariance matrix estimator called
Kullback-Leibler information measure (KLIM) is
developed based on minimum description length
112
Zhou X., Guo P. and Chen C..
MULTI-REGULARIZATION PARAMETERS ESTIMATION FOR GAUSSIAN MIXTURE CLASSIFIER BASED ON MDL PRINCIPLE.
DOI: 10.5220/0003669301120117
In Proceedings of the International Conference on Neural Computation Theory and Applications (NCTA-2011), pages 112-117
ISBN: 978-989-8425-84-3
Copyright
c
 2011 SCITEPRESS (Science and Technology Publications, Lda.)

(MDL) principle for small number samples with
high dimension data (Guo, 2008). The KLIM
estimator is derived theoretically by KL divergence.
And a formula for fast estimation the regularization
parameter is derived. However, since multi-
parameters optimization is more difficult than single
parameter optimization, only a special case that the
regularization parameters are taken the same value
for all dimensions is considered in KLIM. Though
estimation of regularization parameter becomes
simple, the accuracy of covariance matrix estimation
is decreased by ignore the local difference in each
dimension. This will result in decreasing the
classification accuracy of Gaussian classifier finally.
In this paper, KLIM is generalized to KLIM_L
which considers the local difference in each
dimension. Based on MDL principle, the KLIM_L
covariance matrix estimation is derived for the small
sample problem with high dimension. Multi-
regularization parameters in each dimension are
selected by the criterion of minimization the KL
divergence and estimated efficiently by two-order
Taylor expansion. The feasibility and efficiency of
KLIM_L are shown by the experiments.
2 THEORETICAL
BACKGROUND
2.1 Gaussian Mixture Classifier
Given a data set
1
{}
N
ii
D

 x which will be classified.
Assume that the data point in
D is sampled from a
Gaussian mixture model which has k component:
1
,(, ) ( , )
k
jjj
j
pG 



mxxΣ
with 0
j

 ,
1
1
k
j
j




,
(1)
where
T1
/2 1/2
,
11
(,) exp{( ) ( )}
2(2 ) | |
jj j j j
d
j
G


mmmx Σ x Σ x
Σ
(2)
is a general multivariate Gaussian density function.
x
is a random vector, the dimension of
x
is d and
1
{, , }
k
j
jjj


 m Σ is a parameter vector set of
Gaussian mixture model.
In the case of Gaussian mixture model, the
Bayesian decision rule
arg max ( | , )
j
jpj

x is
adopted to classify the vector
x
into class j

with
the largest posterior probability
(|, )pj x . After
model parameters

estimated by the maximum
likelihood (ML) method with expectation-
maximization (EM) algorithm (Redner, 1984), the
posterior probability can be written in the form:
,
ˆ
ˆ
ˆ
(,)
ˆ
(|, )
ˆ
(, )
jjj
G
pj
p



mx Σ
x
x
,
1, 2, ,jk

 .
(3)
And the classification rule becomes:
arg min ( )
jj
jd

 x , 1, 2, ,jk  ,
(4)
where
T1
ˆˆ
ˆ
ˆ
ˆ
()()()ln||2ln
jjjjjj
d


X mmx Σ x Σ
.
(5)
This equation is often called the discriminant
function for the class
j (Aeberhard, 1994).
Since clustering is more general than
classification in the mixture model analysis case, we
consider the general case in the following.
2.2 Covariance Matrix Estimation
The central idea of the MDL principle is to represent
an entire class of probability distributions as models
by a single “universal” representative model, such
that it would be able to imitate the behaviour of any
model in the class. The best model class for a set of
observed data is the one whose representative
permits the shortest coding of the data. The MDL
estimates of both the parameters and their total
number are consistent; i.e., the estimates converge
and the limit specifies the data generating model
(Rissanen, 1978); (Barron, 1998). The codelength
(probability distribution or a model) criterion of
MDL involves in the KL divergence (Kullback,
1959).
Now considering a given sample data set
1
{}
N
ii
D

 x generated from an unknown density
()p X , it can be modelled by a finite Gaussian
mixture density
(, )p

x , where  is the parameter
set. In the absence of knowledge of
()p x
, it may be
estimated by an empirical kernel density estimate
()
h
p x obtained from the data set. Because these two
probability densities describe the same unknown
density
()p x , they should be best matched with
proper mixture parameters and smoothing
parameters. According to MDL principle, the model
parameters should be estimated with minimized KL
divergence
(, )KL h

based on the given data drawn
from the unknown density
()p x
(Kullback, 1959),
MULTI-REGULARIZATION PARAMETERS ESTIMATION FOR GAUSSIAN MIXTURE CLASSIFIER BASED ON
MDL PRINCIPLE
113

()
()ln
(, ) d
(, )
h
h
p
KL h p
p



x
xx
x
(6)
with

1
1
/2 1/2
1
() ,
1
(,)
11
exp ( ) ( )
2
(2 ) | |
N
hih
i
N
T
ih i
d
i
h
pG
N
N









xxxW
xx W xx
W
.
(7)
Here
h
W is a dd dimensional diagonal matrix
with a general form,
12
(, , , )
hd
diag h h hW  ,
(8)
where
i
h , 1, 2, ,id  are smoothing parameters
(or regularization parameters) in the nonparametric
kernel density. In the following this set is denoted
as
1
{}
d
ii
hh

 . The Eq. (6) equals to zero if and only
if
() (, )
h
pp xx.
If the limit
0h  , the kernel density function
()
h
p x becomes a

function, then Eq. (6) reduces
to the negative log likelihood function. So the
ordinary EM algorithm can be re-derived based on
the minimization of this KL divergence function
with the limit
0h  . The ML-estimated parameters
are shown as follows:
1
ˆ
1
ˆ
(| , )
N
jii
i
j
pj
n



m xx,
T
1
ˆˆ
1
ˆˆ
(|,)()()
N
jiijij
i
j
pj
n



mmΣ xx x .
(9)
The covariance matrix estimation for the
limit
0h 
is shown as follows. By minimizing Eq.
(6) with respect to
j
Σ
, i.e., setting
(, )/ 0
j
KL hΣ , the following covariance
matrix estimation formula can be obtained:
T
()
()
ˆˆ
ˆ
(|, )( )( )d
ˆ
ˆ
(|, )d
hjj
j
h
ppj
ppj





mmxxx x x
Σ
xxx
.
(10)
In this case, the Taylor expansion is used for
(|, )pj x at
i
xxwith respect to x and it is
expanded to first order approximation:
T
ˆˆ
ˆ
(|, ) (| , ) ( ) (| , )
iixi
pj pj pj    xxxxx
(11)
with
ˆˆ
(| , ) (|, )|
i
xi x
pj pj


xx
xx.
On substituting the above equation into Eq. (10)
and according to the properties of probability density
function, the following approximation is finally
derived:
ˆ
()
jhj
h Σ W Σ .
(12)
The estimation in Eq. (12) is called as KLIM_L in
the paper, where
ˆ
j
Σ
is the ML estimation
when
0
h 
, taking the form of Eq. (9).
3 MULTI- REGULARIZATION
PARAMETERS ESTIMATION
BASED ON MDL
3.1 Regularization Parameters
Selection
The regularization parameter set h in the Gaussian
kernel density plays an important role in estimating
the mixture model parameter. Different
h
will
generate different models. So selecting the
regularization parameters is a model selection
problem. In the paper the similar method as in (Guo,
2008) is adopted based on MDL principle to select
the regularization parameters in KLIM_L.
According to the principle of MDL, it should be
with the shortest codelength to select a model. When
0h

, the regularization parameters h can be
estimated with the minimized KL divergence
regarding
h with ML estimated parameter
ˆ

,
*
arg min ( )hJh ,
ˆ
() (, )Jh KLh
. (13)
Now a second order approximation for estimating
the regularization parameter
h is adopted here.
Rewrite the
()
J
h as:
0
() () ()
e
J
hJhJh

 ,
(14)
where
0
()() ln (, )d
h
Jh p p

xxx
,
() ()() ln d
ehh
Jh p p

xxx
.
Replacing
ln ( , )p

x with the second order term of
Taylor expansion into the integral of
0
()
J
h and
resulting in the following approximation of
0
()
J
h ,
0
1
1
1
() ln(,)
1
trace( ( ln ( , )))
2
N
i
i
N
hi
i
Jh p
N
p
N




 


xx
x
Wx
(15)
For very sparse data distribution, the following
approximation can be used:
NCTA 2011 - International Conference on Neural Computation Theory and Applications
114

1
1
T1
() () , ,
,
11
ln ( , ) ln ( , )
11
( , )[ ln ln 2 ln | |
22
1
() ()]
2
N
h h ih ih
i
N
ih h
i
ih i
pp G G
NN
d
GN
N










xx xxW xxW
xx W W
xx W xx
.
(16)
Substituting the Eq. (16) into ()
e
J
h , it can be got:
() ()() ln d
1
ln ln 2 ln | |
22 2
ehh
h
Jh p p
dd
N
 

  

xxx
W
.
(17)
So far, the approximation formula of ()
J
h is
obtained:
1
1
( ) trace( ( ln ( , )))
2
1
ln | |
2
N
hi
i
h
Jh p
N
C

  


xx
Wx
W
,
(18)
where C is a constant irrelevant to h .
Let
ln ( , )
dd i
Hp

  
xx
x . Taking partial
derivative of
()
J
h to
h
W and letting it be equal to
zero, the rough approximation formula of
h is
obtained as follows:
1
1
1
()
N
h
i
diag H
N




W .
(19)
3.2 Approximation for Regularization
Parameters
The Eq. (19) can be rewritten as follows:
1
11
11
() ( ln(,))
NN
h i
ii
diag H diag p
NN


     

xx
Wx
with
1
11 T1
11
1T1
11
ln ( , )
{(|,)[ ( )( ) ]
(| , ) ( ) (| , )( ) }
N
i
i
Nk
ijjijijj
ij
kk
ijij iijj
jj
p
pj
pj pj

 





 

 



xx
mm
mm
x
x ΣΣxx Σ
x Σ xxxΣ
Considering hard-cut case (
(| , ) 1or0
i
pj x ) and
using the approximations
T
1
ˆ
(| , )( )( )
N
iijij jj
i
pj n



mmxx x Σ
,
1
(| , )
N
ij
i
pj n



x ,
1
ˆ
jj
I

ΣΣ and
11
1
k
jj
j






ΣΣ,
it can be obtained:
11
()
h
diag

W Σ .
(20)
Suppose the eigenvalues and eigenvectors of the
common covariance matrix
Σ are
k

and
k

,
1
kd

 , where
T
12
(, ,, )
kkk kd
    , then there
exists:
1
d
T
kkk
k





Σ ,
11
1
d
T
kkk
k






Σ .
(21)
Substituting Eq. (21) into Eq. (20) and using the
average eigenvalue
1
()/trace()/
d
i
i
dd 



Σ
to
substitute each eigenvalue of matrix
Σ
(only in the
denominator), it can be obtained:
2
1
2
()
trace ( )
h
d
diag

W Σ
Σ
with
[]
ij d d


Σ
.
Finally, the regularization parameters can be
approximated by the following equation:
2
2
11
trace ( ) 1 1
(,, )
h
dd
diag
d


Σ
W 
.
(22)
3.3 Comparison of KLIM_L with
Regularization Methods
The four methods (KLIM_L, KLIM, RDA
(Regularized Discriminant Analysis, Friedman, 1989)
and LOOC (Leave-one-out covariance matrix
estimate, Hoffbeck, 1996)) are all regularization
methods to estimate the covariance matrix for small
sample size problem. They all consider ML
estimated covariance matrix with the additional
extra matrices.
KLIM_L is derived by the similar way as KLIM
under the framework of MDL principle. Meanwhile,
the estimation of regularization parameters is similar
to KLIM based on MDL principle. KLIM_L is a
generalization of KLIM. Multi-regularization
parameters are included and estimated in KLIM_L
while one regularization parameter is estimated in
KLIM. For every
ii

(
1
id


), if it is taken
by
1
trace( )
d
Σ
, then (trace( ) / )
hd
dW Σ I . It will
reduce to the case of
hd
hWI in KLIM,
where
trace( ) /hd

Σ .
KLIM_L is derived based on MDL principle
while RDA and LOOC are heuristically proposed.
They differ in mixtures of covariance matrix
considered and the criterion used to select the
regularization parameters.
MULTI-REGULARIZATION PARAMETERS ESTIMATION FOR GAUSSIAN MIXTURE CLASSIFIER BASED ON
MDL PRINCIPLE
115

Different computation time costs are required in
the four regularization discriminant methods. The
time costs of them are sorted decreasing in the
following order: KLIM, KLIM_L, LOOC and RDA.
This will be validated by the following experiments.
4 EXPERIMENT RESULTS
In this section, the classification accuracy and time
cost of KLIM_L are compared with LDA (Linear
Discriminant Analysis, Aeberhard, 1994), RDA,
LOOC and KLIM on COIL-20 object data (Nene,
1996).
COIL-20 is a database of gray-scale images of 20
objects. The objects were placed on a motorized
turntable against a black background. The turntable
was rotated through 360 degrees to vary object pose
with respect to a fix camera. Images of the objects
were taken at pose intervals of 5 degrees, which
corresponds to 72 images per object. The total
number of images is 1440 and the size of each image
is 12
8×128.
In the experiment, the regularization parameter
h of KLIM is estimated by trace( ) /hd Σ . The
parameter matrix
h
W of KLIM_L is estimated by
Eq. (22). In RDA, the values of

and

are
sampled in a coarse grid, (0.0, 0.25, 0.50, 0.75, 1.0),
resulting in 25 data points. In LOOC, the four
parameters are taken according to the table in
(Hoffbeck, 1996). Six images are randomly selected
as training samples from each class to estimate the
mean and covariance matrix. And the remaining
images are employed as testing samples to verify the
classification accuracy. Since the dimension of
image data is very high of 128×128, PCA is adopted
here to reduce the data dimension. Experiments are
performed with five different numbers of
dimensions. Each experiment runs 25 times, and the
mean and standard deviation of classification
accuracy are reported as results. The results of
experiment are shown in table 1 table 2.
In the experiment, the classification accuracy of
KLIM_L is the best among the five compared
methods, while the classification accuracy of KLIM
is the second best. The classification accuracy of
LOOC is the worst among the compared methods
except in dimension 80, where the classification
accuracy of LOOC is higher than that of LDA.
Considering the time cost of regularization
parameters estimating, KLIM_L needs a little more
time to estimate the regularization parameters than
KLIM needs, while RDA and LOOC need much
more time than KLIM_L needs. The experimental
results are consistent with the theoretical analysis.
5 CONCLUSIONS
In this paper, the KLIM_L covariance matrix
estimation is derived based on MDL principle for
the small sample problem with high dimension.
Under the framework of MDL principle, multi-
regularization parameters are estimated simply and
directly by point estimation which is approximated
by two-order Taylor expansion. KLIM_L is a
generalization of KLIM. With the KL information
measure, total samples can be used to estimate the
regularization parameters in KLIM_L, making it less
computation-expensive than using leave-one-out
cross-validation method in RDA and LOOC.
Table 1: Mean classification accuracy on COIL-20 object database.
classifier LDA RDA LOOC KLIM KLIM_L
80 81.6(2.6) 87.2(2.2) 82.2(1.8) 87.7(1.8) 90.0(2.2)
70 83.6(2.2) 86.4(1.8) 81.2(2.8) 87.0(1.7) 87.9(1.8)
60 85.0(2.2) 86.8(2.2) 82.8(2.3) 87.7(1.4) 88.1(1.6)
50 85.0(2.5) 86.3(2.4) 80.3(2.9) 87.5(2.0) 87.8(2.3)
40 86.9(1.7) 86.9(1.7) 80.6(3.2) 87.6(1.3) 87.6(1.6)
Table 2: Time cost (in seconds) of estimating regularization parameters on COIL-20 object database.
classifier RDA LOOC KLIM KLIM_L
80 62.0494 1.8286 3.8042e-005 8.1066e-005
70 48.8565 1.5147 3.4419e-005 7.8802e-005
60 34.2189 1.0275 2.9890e-005 5.2534e-005
50 23.9743 0.7324 2.9890e-005 4.9817e-005
40 16.2443 0.4987 3.0796e-005 4.9364e-005
NCTA 2011 - International Conference on Neural Computation Theory and Applications
116

KLIM_L estimator achieves higher classification
accuracy than LDA, RDA, LOOC and KLIM
estimators on COIL-20 data set. In the future work,
the kernel method combined with these
regularization discriminant methods will be studied
for small sample problem with high dimension and
the selection of kernel parameters will be
investigated under some criterion.
ACKNOWLEDGEMENTS
The research work described in this paper was fully
supported by the grants from the National Natural
Science Foundation of China (Project No.
90820010, 60911130513). Prof. Guo is the author to
whom the correspondence should be addressed, his
e-mail address is [e-mail]
REFERENCES
Bishop, C. M., 2007. Pattern recognition and machine
learning, Springer-Verlag New York, Inc. Secaucus,
NJ, USA.
Everitt, B. S., Hand, D., 1981. Finite Mixture
Distributions, Chapman and Hall, London.
Friedman, J. H., 1989. Regularized discriminant analysis,
Journal of the American Statistical Association, vol.
84, no. 405, 165–175.
Hoffbeck, J. P. and Landgrebe, D. A., 1996. Covariance
matrix estimation and classification with limited
training data, IEEE Transactions on Pattern Analysis
and Machine Intelligence, vol. 18, no. 7, 763–767.
Schafer, J. and Strimmer, K., 2005. A shrinkage approach
to large-scale covariance matrix estimation and
implications for functional genomics, Statistical
Applications in Genetics and Molecular Biology, vol.
4, no. 1.
Srivastava, S., Gupta, M. R., Frigyik, B. A., 2007.
Bayesian quadratic discriminant analysis, J. Mach.
Learning Res. 8, 1277
–1305.
Bickel, P. J. and Levina, E., 2008. Regularized estimation
of large covariance matrices, Annals of Statistics, vol.
36, no. 1, 199–227.
Friedman, J., Hastie, T., and Tibshirani, R., 2008. Sparse
inverse covariance estimation with the graphical lasso,
Biostatistics, vol. 9, no. 3, 432–441.
Cao, G., Bachega, L. R., Bouman, C. A., 2011. The Sparse
Matrix Transform for Covariance Estimation and
Analysis of High Dimensional Signals. IEEE
Transactions on Image Processing, Volume 20, Issue
3, 625 – 640.
Rivals, I., Personnaz, L., 1999. On cross validation for
model selection, Neural Comput. 11,863
–870.
Guo, P., Jia, Y., and Lyu, M. R., 2008. A study of
regularized Gaussian classifier in high-dimension
small sample set case based on MDL principle with
application to spectrum recognition, Pattern
Recognition, Vol. 41, 2842~2854.
Redner, R. A., Walker, H. F., 1984. Mixture densities,
maximum likelihood and the EM algorithm, SIAM
Rev. 26, 195
–239.
Aeberhard, S., Coomans, de Vel, D., O., 1994.
Comparative analysis of statistical pattern recognition
methods in high dimensional settings, Pattern
Recognition 27 (8), 1065
–1077.
Rissanen, J., 1978. Modeling by shortest data description,
Automatica 14, 465
–471.
Barron, A., Rissanen, J., Yu, B., 1998. The minimum
description length principle in coding and modeling,
IEEE Trans. Inform. Theory 44 (6), 2743
–2760.
Kullback, S., 1959. Information Theory and Statistics,
Wiley, New York.
Nene, S. A., Nayar, S. K. and Murase, H., 1996. Columbia
Object Image library(COIL-20). Technical report
CUCS-005-96.
MULTI-REGULARIZATION PARAMETERS ESTIMATION FOR GAUSSIAN MIXTURE CLASSIFIER BASED ON
MDL PRINCIPLE
117