AN INTERPOLATION METHOD FOR THE RECONSTRUCTION

AND RECOGNITION OF FACE IMAGES

N. C. Nguyen and J. Peraire

Massachusetts Institute of Technology, 77 Massachusetts Ave, Cambridge, MA 20139, USA

Keywords:

Face reconstruction, face recognition, best points interpolation method, principal component analysis.

Abstract:

An interpolation method is presented for the reconstruction and recognition of human face images. Basic

ingredients include an optimal basis set deﬁning a low-dimensional face space and a set of interpolation points

capturing the most relevant characteristics of known faces. The interpolation points are chosen as pixels

of the pixel grid so as to best interpolate the set of known face images. These points are then used in a

least-squares interpolation procedure to determine interpolant components of a face image very inexpensively,

thereby providing efﬁcient reconstruction of faces. In addition, the method allows a fully automatic computer

system to be developed for the real-time recognition of faces. The advantages of this method are: (1) the

computational cost of recognizing a new face is independent of the size of the pixel grid; and (2) it allows for

the reconstruction and recognition of incomplete images.

1 INTRODUCTION

Image processing and recognition of human faces

constitutes a very active area of research. The ﬁeld

has evolved rapidly and become one of the most suc-

cessful applications of image analysis and computer

vision partly because of availability of many power-

ful methods and partly because of its signiﬁcant prac-

tical importance in many areas such as authenticity

in security and defense systems, banking, human–

machine interaction, image and multimedia process-

ing, psychology, and neurology. Principal component

analysis (PCA) or the Karhunen-Lo

`

eve (KL) expan-

sion is a well-established method for the representa-

tion (Sirovich and Kirby, 1987; Kirby and Sirovich,

1990; Everson and Sirovich, 1995) and recogni-

tion (Turk and Pentland, 1991) of human faces.

PCA approach (Kirby and Sirovich, 1990) for face

representation consists of computing the “eigenfaces”

of a set of known face images and approximating any

particular face by a linear combination of the leading

eigenfaces. For face recognition (Turk and Pentland,

1991), a new face is ﬁrst projected onto the eigenface

space and then classiﬁed according to the distances

between its PCA coefﬁcient vector and those repre-

senting the known faces. There are two drawbacks

with this approach. First, PCA may not handle incom-

plete data well situations in which only partial infor-

mation of an input image is available. Secondly, the

computational cost per image classiﬁcation depends

on the size of the pixel grid.

This paper describes an interpolation method for

the reconstruction and recognition of face images.

The method was ﬁrst introduced in (Nguyen et al.,

2006) for the approximation of parametrized ﬁelds.

The basic ingredient is a set of interpolation points

capturing the most relevant features of known face

images. The essential component is a least-squares

interpolation procedure for the very rapid computa-

tion of the interpolant coefﬁcient vector of any given

input face. The interpolant coefﬁcient vector is then

used to determine which face in the face set, if any,

best matches the input face. A signiﬁcant advantage

of the method is that the computational cost of rec-

ognizing a new face is independent of the size of the

pixel grid, while achieving a recognition rate compa-

rable to PCA. Moreover, the method allows the recon-

struction and recognition of incomplete images.

The paper is organized as follows. In Section 2,

we present an overview of PCA. In Section 3, we ex-

91

C. Nguyen N. and Peraire J. (2007).

AN INTERPOLATION METHOD FOR THE RECONSTRUCTION AND RECOGNITION OF FACE IMAGES.

In Proceedings of the Second International Conference on Computer Vision Theory and Applications - IU/MTSV, pages 91-96

Copyright

c

SciTePress

tend the best points interpolation method (BPIM) in-

troduced in (Nguyen et al., 2006) and apply it to de-

velop an automatic real-time face recognition system.

In section 4, we test and compare our approach with

PCA. Finally, in Section 5, we close the paper with

some concluding remarks.

2 PRINCIPAL COMPONENT

ANALYSIS

2.1 Eigenfaces

An ensemble of face images is denoted by

U

K

= {u

i

},

1 ≤ i ≤ K, where u

i

represents an i-th mean-subtracted

face and K represents the number of faces in the en-

semble. It is assumed that after proper normaliza-

tion and resizing to a ﬁxed pixel grid Ξ of dimen-

sion N

1

by N

2

, u

i

can be considered as a vector in an

N-dimensional image space, where N = N

1

N

2

is the

number of pixels. PCA (Sirovich and Kirby, 1987;

Kirby and Sirovich, 1990) constructs an optimal rep-

resentation of the face ensemble in the sense that the

average reconstruction error

ε

∗

=

K

∑

i=1

u

i

−

k

∑

j=1

(φ

T

j

u

i

)φ

j

2

, (1)

is minimal for all k ≤ K. In the literature (Turk and

Pentland, 1991), the basis vectors φ

j

are referred as

eigenfaces and the space spanned by them is known

as the face space. The construction of the eigenfaces

is as follows.

Let U be the N × K matrix whose columns are

[u

1

,...,u

K

]. It can be shown that the φ

i

satisfy

Aφ

i

= λ

i

φ

i

, (2)

where the covariance matrix A is given by

A =

1

K

UU

T

. (3)

Here the eigenvalues are arranged such that λ

1

≥ . . . ≥

λ

K

. Since the matrix A of size N × N is large, solving

the above eigenvalue problem can be very expensive.

However, if K < N, there will be only K meaning-

ful eigenvectors and we may express φ

i

as

φ

i

=

K

∑

j=1

ϕ

ij

u

j

. (4)

Inserting (3) and (4) into (2), we immediately obtain

Gϕ

i

= λ

i

ϕ

i

, (5)

where G =

1

K

U

T

U is a symmetric positive-deﬁnite

matrix of size K by K. The eigenvalue problem (5)

Figure 1: Eigenfaces and the mean face. The mean face is

on the top left and followed by 11 top eigenfaces, in order

from left to right and top to bottom.

can be solved for ϕ

ij

,1 ≤ i, j ≤ K, from which the

eigenfaces φ

i

are obtained.

We present in Figure 1 the mean face and a few

of the top eigenfaces for a training ensemble of 400

face images extracted from the AT&T database (see

Section 4.1 for details).

2.2 Face Reconstruction

We brieﬂy describe the reconstruction of face images

using PCA and later compare the results with those

obtained using our method. First, we project an input

face u onto the face space Φ

k

= span{φ

1

,...,φ

k

} to

obtain

u

∗

=

k

∑

i=1

a

i

φ

i

, (6)

where for i = 1,...,k,

a

i

= φ

T

i

u . (7)

We also deﬁne the associated error as

ε

∗

= ku− u

∗

k . (8)

Note that the mean face of the ensemble

U

K

should

be added to u

∗

to obtain the reconstructed image; and

that if k is set equal to K, the reconstruction is exact

for all members of the ensemble.

2.3 Face Recognition

We brieﬂy describe the eigenface recognition proce-

dure of Turk and Pentland (Turk and Pentland, 1991).

To classify an input image, one ﬁrst obtains PCA co-

efﬁcients a

i

,1 ≤ i ≤ k, as described above. One then

computes the Euclidean distances between its PCA

coefﬁcient vector a = [a

1

,...,a

k

]

T

and those repre-

senting each individual in the training ensemble. De-

pending on the smallest distance and the PCA recon-

struction error ε

∗

, the image is classiﬁed as belonging

to a familiar individual, as a new face, or a non-face

image. Several variants of the above procedure are

possible via the use of a different classiﬁer such as the

nearest-neighbor classiﬁer and a different norm such

as L

1

norm or Mahalanobis norm (Delac et al., 2005).

It is generally observed that the recognition per-

formance is improved when using a larger k. Typ-

ically, the number of eigenfaces k required for face

recognition varies from O(10) to O(10

2

) and is much

smaller than N. We note that classiﬁcation of an in-

put image requires the evaluation of PCA coefﬁcients

according to (7). The computational cost per image

classiﬁcation is thus at least O(Nk). This cost de-

pends linearly on N and is quite acceptable for a small

number of input images. However, when classiﬁca-

tion of many images is performed at the same time,

PCA approach appears increasingly intractable. Real-

time recognition is thus excluded for large-scale ap-

plications. Other subspace methods such as indepen-

dent component analysis (ICA) (Draper et al., 2003;

Bartlett et al., 2002) and linear discriminant analysis

(LDA) (Etemad and Chellappa, 1997; Lu et al., 2003)

suffer from similar drawbacks.

3 BEST POINTS

INTERPOLATION METHOD

In this section, we extend the best points interpolation

method developed earlier in (Nguyen et al., 2006) to

face reconstruction and recognition. The basic ingre-

dients of the method are a stable interpolation proce-

dure and a set of interpolation points.

3.1 Interpolation Procedure

Let us recall the pixel grid Ξ and the face space

Φ

k

= span{φ

1

,...,φ

k

}. In this space, we shall seek

an approximation of any input image u. Rather than

performing the projection onto the face space for the

best approximation, we pursue an interpolation as fol-

lows.

In particular, we aim to ﬁnd an approximation ˜u ∈

Φ

k

of u via m(≥ k) interpolation points {z

j

∈ Ξ},1 ≤

j ≤ m, such that

˜u =

k

∑

i=1

˜a

i

φ

i

(9)

where the coefﬁcients ˜a

i

are the solution of

k

∑

i=1

φ

i

(z

j

) ˜a

i

= u(z

j

), j = 1,...,m . (10)

We deﬁne the associated error as

˜

ε = ku− ˜uk . (11)

In general, the linear system (10) is over-determined

because there are more equations than unknowns.

10 20 30 40 50 60 70

10

20

30

40

50

60

70

80

90

Figure 2: Distribution of the interpolation points on the

pixel grid for k = 100 and m = 200.

Hence, the interpolant coefﬁcient vector

˜

a =

[ ˜a

1

,..., ˜a

k

]

T

is determined from

C

T

C

˜

a = C

T

c , (12)

where C ∈ R

m×k

with C

ji

= φ

i

(z

j

),1 ≤ i ≤ k,1 ≤ j ≤

m and c = [u(z

1

),...,u(z

m

)]

T

. It thus follows that

˜

a = Bc . (13)

Here the matrix B=

C

T

C

−1

C

T

is precomputed and

stored. Therefore, for any new face u, the cost of

evaluating the interpolant coefﬁcient vector

˜

a is only

O(mk) and becomes O(k

2

) when m = O(k).

Obviously, the approximation quality depends

crucially on the interpolation pixels {z

j

}. Therefore,

it is extremely important to choose {z

j

} so as to guar-

antee accurate and stable interpolation. For instance,

Figure 2 shows the interpolation points for k = 100

and m = 200 obtained using our method described be-

low. We see that the pixels are distributed somewhat

symmetrically with respect to the symmetry line of

the face and largely allocated around main locations

of the face such as eyes, nose, mouth, and jaw.

3.2 Interpolation Points

We proceed by describing our approach for determin-

ing the interpolation points. The crucial observation

is that much of the surface of a face is smooth with

regular texture and that faces are similar in appear-

ance and highly constrained; for example, the frontal

view of a face is symmetric. Moreover, the value of

a pixel is typically highly correlated with the values

of the surrounding pixels. Therefore, a large number

of pixels in the image space does not represent physi-

cally possible faces and only a small number of pixels

may sufﬁce to represent facial characteristics.

To begin, we introduce a set of images, U

∗

K

=

{u

∗

ℓ

},1 ≤ ℓ ≤ K,, where u

∗

ℓ

is the best approximation

to u

ℓ

. It thus follows that

u

∗

ℓ

=

k

∑

i=1

a

ℓi

φ

i

, (14)

where for 1 ≤ i ≤ k,1 ≤ ℓ ≤ K,

a

ℓi

= φ

T

i

u

ℓ

. (15)

We then determine {z

j

},1 ≤ j ≤ m as a minimizer of

the following minimization

min

x

1

∈Ξ,...,x

m

∈Ξ

K

∑

ℓ=1

k

∑

i=1

a

ℓi

− ˜a

ℓi

(x

1

,...,x

m

)

2

(16)

k

∑

i=1

φ

i

(x

j

) ˜a

ℓi

= u

ℓ

(x

j

), 1 ≤ j ≤ m,1 ≤ ℓ ≤ K.

We shall call the z

j

as best interpolation points, be-

cause the points are optimal for the interpolation of

the best approximations u

∗

ℓ

. We refer the reader

to (Nguyen et al., 2006) for details on the solution

procedure.

3.3 Application to Face Recognition

We apply the method to develop a fully automatic

real-time face recognition system involving the gen-

eration stage and the recognition stage. The detailed

implementation of the system is given below:

1. Determine the dimension of the face space k and

then calculate φ

1

,...,φ

k

.

2. Compute and store {z

j

}, B =

C

T

C

−1

C

T

. Re-

call that C

ji

= φ

i

(z

j

),1 ≤ i ≤ k,1 ≤ j ≤ m.

3. For a “gallery” of images

V

K

′

= {v

i

},1 ≤ i ≤ K

′

,

compute

˜

a

i

= B[v

i

(z

1

),...,v

i

(z

m

)]

T

,1 ≤ i ≤ K

′

.

(Note

V

K

′

can be the same or different from

U

K

).

4. For each new face to be classiﬁed u, calculate its

interpolant coefﬁcient vector

˜

a from (13) and ﬁnd

i

min

= arg min

1≤i≤K

′

k

˜

a−

˜

a

i

k . (17)

5. If k

˜

a−

˜

a

i

min

k is less than a chosen threshold, the

input image u is identiﬁed as the individual asso-

ciated with the gallery image i

min

. Otherwise, the

image is classiﬁed as a new individual.

The generation stage (steps 1–2) is computation-

ally expensive, but only performed when the training

set changes. However, the recognition stage (steps

4–5) is very inexpensive: the calculation of

˜

a takes

O(mk); and the nearest-neighbor search problem (17)

which can be solved typically in O(kK

′0.25

) (Andoni

and Indyk, 2006). Hence, if K

′

is in order of O(k

4

)

or less, the computational cost is only O(k

2

). This

is often the case even for large-scale applications; for

example, for a training database of 10

4

images, one

would need more (or many more) than 10 eigenfaces

to achieve acceptable recognition rates.

In summary, the operation count of the recogni-

tion stage is about O(mk). The computational com-

plexity of our system is thus independent of N. As

mentioned earlier, the complexity of PCA-based al-

gorithms is at least O(Nk). Our approach leads to

a computational reduction of N/m relative to PCA.

Since m is typically much smaller than N, signiﬁcant

savings are expected. The savings per image classiﬁ-

cation certainly translate to real-time performance es-

pecially when many face images need to be classiﬁed

simultaneously.

4 EXPERIMENTS

In practice, some applications of face recognition re-

gard the recognition quality more importantly than the

computational performance. Therefore, in order to

be useful and gain acceptance, our approach must be

tested and compared with existing approaches, partic-

ularly here with the PCA.

4.1 Face Database

The AT&T face database (Samaria and Harter, 1994)

consists of 400 images of 40 individuals (10 images

per individual). The images were taken at different

times with variation in lighting, poses, and facial ex-

pressions, with and without glasses. The images were

cropped and resized by us to a resolution of 74 × 90.

We formed a training ensemble of 400 images by us-

ing 200 images of the database, 10 each of 20 differ-

ent individuals, and including 200 mirror images of

these images (Kirby and Sirovich, 1990).

The testing set contains the (200) remaining im-

ages of 20 individuals not belonging to the training

ensemble. We further divide the testing set into the

gallery of 20 individual faces and 180 probe images

containing 9 views of every individual in the gallery.

The recognition task is to match the probe images to

the 20 gallery faces. The fact that the training and test-

ing sets have no common individual serves to assess

the performance of a face recognition system more

critically — the ability to recognize new faces which

are not part of the face space constructed from the

training set.

Figure 3: The reconstruction results for a familiar face. The

BPIM reconstructed images are placed at the top row for

k = 40,80, 120,160 (from left to right) and m = 2k. The

PCA reconstructed images are placed at the second row for

k = 40,80,120, 160 (from left to right). The original face is

shown on the right.

4.2 Results for Face Reconstruction

We ﬁrst present in Figure 3 the reconstruction results

for a face in the training ensemble. The BPIM pro-

duces reconstructions almost as well as PCA: most

facial features captured by the PCA reconstructed im-

ages also appear in the BPIM reconstructed images.

We underline the fact that the interpolation method

requires less than 5% of the total number of pixels

N = 6660, but delivers quite satisfactory results.

To illustrate the use of the interpolation approach

for reconstructing a full image from a partial image,

we consider a face (in the training set) shown at the

bottom right and a mask shown at the top right in Fig-

ure 4. This is a relatively extreme mask that obscures

90% of the pixels in a random manner. Because the

masked face may not have intensity values at all the

best interpolation points, we need to deﬁne a new set

of interpolation points. To this end, we keep the best

interpolation points which coincide with some of the

white pixels of the masked face and replace the re-

maining best pixels with the “nearest” white pixels.

In Figure 4, the reconstructed images using those in-

terpolation points are compared with the PCA recon-

structed images utilitizing all the pixels. Although the

interpolation procedure does not recover the original

face exactly, the construction is visually close to the

“best” reconstruction.

4.3 Results for Face Recognition

We apply the face recognition system developed in

Section 3.4 to classify the probe images. We illus-

trate in Figure 5 the recognition accuracy as a function

of k for the BPIM and PCA. As it may be expected,

the BPIM yields smaller recognition rates than PCA.

However, as k increases, the BPIM gives recognition

rates which are quite comparable to those of PCA for

Figure 4: Reconstruction of a familiar face (bottom right)

from a 10% mask (top right) with only the white pixels.

The reconstructed images are shown at the top row for k =

40,80, 120, 160 (from left to right) and m = 2k. The PCA

reconstructed images are shown at the second row for k =

40,80, 120, 160 (from left to right) with using all the pixels.

Table 1: Computational times (normalized with respect to

the time to recognize a face for k = 10 and m = 20 with the

BPIM) for the BPIM and PCA at different values of k.

k

BPIM PCA

10 1.00 333.30

20 4.20 592.67

30 9.33 873.34

40 15.60 1107.66

50 26.12 1437.35

60 36.47 1708.02

70 47.93 1958.94

80 61.47 2293.73

large enough k: PCA achieves a recognition rate of

74.98%, while PBIM results in a recognition rate of

73.66% for k = 80. In many applications, the small

accuracy loss of only 1.32% is paid off very well by

the signiﬁcant reduction of 6660/160(> 40) in com-

plexity. This is conﬁrmed in Table 1 which shows

the computational times for the BPIM and PCA. The

values are normalized with respect to the time to rec-

ognize a face for k = 10 and m = 20 with the BPIM.

Clearly, the BPIM is signiﬁcantly faster than PCA.

This important advantage is very useful to applica-

tions that requires a real-time recognition capability.

Finally, in order to demonstrate the classiﬁcation

of incomplete images, we consider a random chosen

mask of 10% pixels shown in Figure 6. Next to the

mask, we show a few faces which are correctly rec-

ognized with using the interpolation procedure when

their intensity values are available only at the white

pixels of the mask. Note the interpolation points are

chosen in the same way as before.

0 10 20 30 40 50 60 70 80

0.3

0.35

0.4

0.45

0.5

0.55

0.6

0.65

0.7

0.75

0.8

k

RECOGNITION RATE

PCA

BPIM

Figure 5: Recognition accuracy of PCA and BPIM with in-

creasing the number of eigenfaces k. Note that the BPIM

uses m = 2k best interpolation points.

Figure 6: Recognition of incomplete face images. The 10%

mask on the left is followed by a few faces which are cor-

rectly recognized with using the interpolation procedure.

5 CONCLUSION

We have presented an interpolation method for the

reconstruction and recognition of face images. It is

important to note that PCA uses full knowledge of

the data in the reconstruction process. In contrast,

our method uses only partial knowledge of the data.

Therefore, the method is very useful to the restora-

tion of a full image from a partial image. Based

on the method, we have also developed a fully au-

tomatic real-time face recognition system. The sys-

tem is shown to be able to recognize incomplete im-

ages. Moreover, the computational cost of recogniz-

ing a new face is only O(mk), translating to a saving

of N/m relative to PCA approach. Typically, since N

is O(10

4

) and m is O(10

2

), this implies two orders of

magnitude less expensive computationally than PCA.

The signiﬁcant reduction in time should enable us to

tackle very large problems. Hence, it is imperative

to test our system on a larger database such as the

FERET database. We plan to pursue this direction in

future research.

ACKNOWLEDGEMENTS

We would like to thank Professor A. T. Patera of MIT

for his long-standing collaboration and many invalu-

able contributions to this work. The authors also

thank AT&T Laboratories Cambridge for providing

the ORL face database. This work was supported by

the Singapore-MIT Alliance.

REFERENCES

Andoni, A. and Indyk, P. (2006). Near-optimal hashing al-

gorithms for approximate nearest neighbor in high di-

mensions. In Proceedings of the 47th IEEE Sym. on

Foundations of Computer Science, Berkeley, CA.

Bartlett, M. S., Movellan, J. R., and Sejnowski, T. J. (2002).

Face recognition by independent component analysis.

IEEE Trans. on Neural Networks, 13:1450–1464.

Delac, K., Grgic, M., and Grgic, S. (2005). Independent

comparative study of pca, ica, and lda on the feret

data set. International Journal of Imaging Systems

and Technology, 15(5):252–260.

Draper, B. A., Baek, K., Bartlett, M. S., and Beveridge, J. R.

(2003). Recognizing faces with pca and ica. Computer

Vision and Image Understanding, 91(1-2):115–137.

Etemad, K. and Chellappa, R. (1997). Discriminant analysis

for recognition of human face images. Journal of the

Optical Society of America A, 14(8):1724–1733.

Everson, R. and Sirovich, L. (1995). Karhunen-loeve pro-

cedure for gappy data. Opt. Soc. Am. A, 12(8):1657–

1664.

Kirby, M. and Sirovich, L. (1990). Application of the

karhunen-lo

`

eve procedure for the characterization of

human face. IEEE Transactions on Pattern Analysis

and Machine Intelligence, 12:103–108.

Lu, J., Plataniotis, K. N., and Venetsanopoulos, A. N.

(2003). Face recognition using lda-based algorithms.

IEEE Trans. on Neural Networks, 14(1):195–200.

Nguyen, N. C., Patera, A. T., and Peraire, J. (2006). A best

points interpolation method for efﬁcient approxima-

tion of parametrized functions. International Journal

of Numerical Methods in Engineering. Submitted.

Samaria, F. S. and Harter, A. C. (1994). Parameterisation

of a stochastic model for human face identiﬁcation.

In Proceedings of the 2nd IEEE workshop on Appli-

cations of Computer Vision, pages 138–142, Sarasota,

Florida.

Sirovich, L. and Kirby, M. (1987). Low-dimensional proce-

dure for the characterization of human faces. Journal

of the Optical Society of America A, 4:519–524.

Turk, M. and Pentland, A. (1991). Eigenfaces for recogni-

tion. Journal of Cognitive Neuroscience, 3(1):71–86.