FILTERING AND COMPRESSION OF STOCHASTIC SIGNALS

UNDER CONSTRAINT OF VARIABLE FINITE MEMORY

Anatoli Torokhti and Stan Miklavcic

University of South Australia, Adelaide, Australia

Keywords:

Optimal ﬁltering, Wiener ﬁlter.

Abstract:

We study a new technique for optimal data compression subject to conditions of causality and different types

of memory. The technique is based on the assumption that certain covariance matrices formed from observed

data, reference signal and compressed signal are known or can be estimated. In particular, such an information

can be obtained from the known solution of the associated problem with no constraints related to causality and

memory. This allows us to consider two separate problems related to compression and de-compression subject

to those constraints. Their solutions are given and the analysis of the associated errors is provided.

1 INTRODUCTION

A study of data compression methods is motivated

by the necessity to reduce expenditures incurred with

the transmission, processing and storage of large data

arrays. While the topics have been intensively stud-

ied (see e.g. (S. Friedland, 2006), (Jolliffe, 1986),

(Hua and Nikpour, 1999), (Hua and Liu, 1998),

(A. Torokhti, 2001), (Torokhti and Howlett, 2007),

(T. Zhang, 2001)), a number of related fundamen-

tal questions are still open. One of them concerns

speciﬁc restrictions associated with different types of

causality and memory.

First Motivation: Causality and Memory. Data

compression techniques mainly consist of three op-

erations, compression itself, de-noising and de-

compression (or reconstruction) of the compressed

data. Each operation is implemented by a special ﬁl-

ter. In reality, a value of the output of such a ﬁlter at

time t

is determined from a ‘fragment’ of its input

deﬁned at times t

k−1

,. .. ,t

k−q

. In other words, in

practice both operations are subject to the conditions

of causality and memory.

Our ﬁrst motivation comes from a real-time signal

processing. This implies that the ﬁlters we propose

should be causal with variable ﬁnite memory.

Second Motivation: Reformulation of the

Problem. Let (Ω, Σ,µ) be a probability space, where

Ω = {ω} is the set of outcomes, Σ a σ–ﬁeld of mea-

surable subsets in Ω and µ : Σ → [0, 1] an associated

probability measure on Σ with µ(Ω) = 1.

In an informal way, the data compression prob-

lem we consider can be expressed as follows. Let y ∈

(Ω,R

) be observable data and x ∈ L

(Ω,R

) be a

reference signal that is to be estimated from y in such

a way that, (a) the data y should be compressed to

a ‘shorter’ vector z ∈ L

(Ω,R

)

with r < min{m,n}

and (b) z should be de-compressed (reconstructed) to

a signal

x ∈ L

(Ω,R

) that is ‘close’ to x in some

appropriate sense. Both operations should be causal

and have variable ﬁnite memory. In this paper, the

term ‘close’ is used with respect to the minimum of

the norm (2) of the difference between x and

The problem can be formulated in several alter-

nate ways.

The ﬁrst way is as follows. Let B : L

(Ω,R

) →

(Ω,R

) signify compression so that z = B (y) and

let A : L

(Ω,R

) → L

(Ω,R

) designate data de-

compression, i.e.,

x = A (z). We suppose that B and

A are linear operators deﬁned by the relationships

[B (y)](ω) = B[y(ω)] and [A (z)](ω) = A[z(ω)]

(1)

where B ∈ R

n×r

and A ∈ R

r×m

. In the remainder of

Components of z are often called principal components

(Jolliffe, 1986).

104

Torokhti A. and Miklavcic S. (2009).

FILTERING AND COMPRESSION OF STOCHASTIC SIGNALS UNDER CONSTRAINT OF VARIABLE FINITE MEMORY.

In Proceedings of the 6th International Conference on Informatics in Control, Automation and Robotics - Signal Processing, Systems Modeling and

Control, pages 104-109

DOI: 10.5220/0002206501040109

 SciTePress

this paper we shall use the same symbol to represent

both the linear operator acting on a random vector and

its associated matrix.

We deﬁne the norm to be

kxk

Ω

kx(ω)k

dµ(ω) (2)

where kx(ω)k

is the Euclidean norm of x(ω). Let us

denote by J(A,B), the norm of the difference between

x and

x, constructed by A and B:

J(A, B) = k x− (A◦ B)(y)k

Ω

. (3)

The problem is to ﬁnd B

: L

(Ω,R

) → L

(Ω,R

)

and A

: L

(Ω,R

) → L

(Ω,R

) such that

J(A

) = min

A,B

J(A, B) (4)

subject to conditions of causality and variable ﬁnite

memory for A and B. The problem consists of two

unknowns, A and B.

A second way to formulate the problem, that

avoids a difﬁculty associated with the two unknowns,

is as follows. Let F : L

(Ω,R

) → L

(Ω,R

) be a

linear operator deﬁned by

[F (y)](ω) = F[y(ω)] (5)

where F ∈ R

n×m

. Let rank F = r and

J(F) = kx− F (y)k

Ω

Find F

: L

(Ω,R

) → L

(Ω,R

) such that

J(F

) = min

J(F) (6)

subject to

rank F ≤ min{m, n} (7)

and conditions of causality and variable ﬁnite mem-

ory for F. Unlike (4), the problem (6)–(7) has only

one unknown.

2 STATEMENT OF THE

PROBLEM

The basic idea of our approach is as follows.

Let x ∈ L

(Ω,R

), y ∈ L

(Ω,R

) and z ∈

(Ω,R

), and let A and B be deﬁned as (1) below.

Here, z is a compressed version of x. We assume

that information about vector z in the form of asso-

ciated covariance matrices can be obtained, in partic-

ular, from the known solution (Torokhti and Howlett,

2007) of problem (6)-(7) with no constraints associ-

ated with causality and memory.

In this paper, the data compression problem sub-

ject to conditions of causality and memory is stated in

the form of two separate problems, (8) and (10) for-

mulated below.

We use the following notation: M (r,n,η

) is a set

of causal r × n matrices B with a so-called complete

variable ﬁnite memory η

. The notation M (m, r,η

)

is similar.

Consider

(B) = kz− B(y)k

Ω

Let B

be such that

) = min

(B) subject to B ∈ M (r,n,η

(8)

We write z

= B

(y). Next, let

(A) = kx− A(z

Ω

(9)

and let A

be such that

) = min

(A) subject to A ∈ M (m, r,η

(10)

We denote x

= A

The problem considered in this paper is to ﬁnd op-

erators B

and A

that satisfy minimization criteria (8)

and (10), respectively.

The major differences between the above state-

ment of the problem and the statements considered

below are as follows.

First, A and B should be causal with variable ﬁnite

memory.

Second, it is assumed that certain covariance ma-

trices formed from x, y and z are known or can

be estimated. In particular, such information can

be obtained from the known solution (Torokhti and

Howlett, 2007) of problem (6)-(7) with no constraints

associated with causality and memory. We note that

such an assumption does not look too restrictive in

comparison with the assumptions used in the associ-

ated methods (Hua and Nikpour, 1999)–(Torokhtiand

Howlett, 2007).

Consequently and thirdly, we represent the initial

problem in the form of a concatenation of two new

separate problems (8) and (10).

3 MAIN RESULTS

Let τ

< τ

< ··· < τ

be time instants and α,β, ϑ :

R → L

(Ω,R) be continuous functions. Sup-

FILTERING AND COMPRESSION OF STOCHASTIC SIGNALS UNDER CONSTRAINT OF VARIABLE FINITE

MEMORY

105

pose α

= α(τ

), β

= β(τ

) and ϑ

= ϑ(τ

) are

real-valued random variables having ﬁnite second

moments. We write x = [α

,α

,. .. ,α

]

y =

[β

,β

,. .. ,β

]

and z = [ϑ

,. .. ,ϑ

]

Let

z be a compressed form of data y deﬁned

z = B(y) with

z = [

,. .. ,

]

, and

x be a de-

compression of

z deﬁned by

x = A(

z) with

x =

[

,. .. ,

]

In many applications

, to obtain

for k =

1,...,r, it is necessary for B to use only a limited

number of input components, η

= 1,...,r. A num-

ber of such input components η

is here called a kth

local memory for B.

To deﬁne a notation of memory for the compres-

sor B, we use parameters p and g which are positive

integers such that 1 ≤ p ≤ n and n−r+2≤ g ≤ n.

Deﬁnition 1. The vector η

= [η

,. .. ,η

]

∈ R

is called a variable memory of the compressor B. In

particular, η

is called a complete variable memory if

= g and η

= n when k = n−g+1,...,n. Here, p

relates to the last possible nonzero entry in the bottom

row of B and g relates to the last possible nonzero

entry in the ﬁrst row.

The notation η

= [η

,. .. ,η

]

∈ R

has a sim-

ilar meaning for the de-compressor A, i.e., η

is a

variable memory of the de-compressor A. Here, η

is the jth local memory of A.

The parameters q and s, which are positive inte-

gers such that 1 ≤ q ≤ r and 2 ≤ s ≤ m, are used

below to deﬁne two types of memory for A.

Deﬁnition 2. Vector η

is called a complete variable

memory of the de-compressor A if η

= q and η

= r

when j = s+ r − 1, .. ., m. Here, q relates to the ﬁrst

possible nonzero entry in the last column of A and s

relates to the ﬁrst possible nonzero entry in the ﬁrst

column.

The memory constraints described above imply

that certain elements of the matrices B = {b

}

r,n

i, j=1

and A = {a

}

m,r

i, j=1

must be set equal to zero. In this

regard, for matrix B with r ≤ p ≤ n, we require that

i, j

= 0

if j = p− r+ i+ 1,...,n,

for



p = r,... ,n − 1,

i = 1,. .. ,r

and



p = n,

i = 1,... ,r− 1,

Examples include computer medical diagnostics (Gi-

meno, 1987) and problems of bio-informatics (H. Kim,

2005).

and, for 1 ≤ p ≤ r− 1, it is required that

i, j

= 0



i = 1,...,r− p,

j = 1,...,n,

and



i = r − p+ 1,...,r,

j = i− r+ p+ 1, .. .,n.

For matrix A with r ≤ p ≤ n, we require

i, j

= 0 (11)

ifj = q + i,...,rforq = 1,...,r− 1,i = 1,...,r− q,

and, for 2 ≤ s ≤ m, it is required that

i, j

= 0

if j = s+ i, .. ., r for s = 1,...,m, i = 1,...,s+ r − 1,

The above conditions imply the following deﬁnitions.

Deﬁnition 3. A matrix B satisfying the constraint

(11)–(11) is said to be a causal operator with the

complete variable memory η

= [g,g + 1,...,n]

Here, η

= n when k = n − g + 1,..., n. The set of

such matrices is denoted by M

(r,n,η

Deﬁnition 4. A matrix A satisfying the constraint

(11)–(11) is said to be a causal operator with the

complete variable memory η

= [r − q + 1,...,r]

Here, η

= r when j = q, .. ., m. The set of such ma-

trices is denoted by M

(m,r,η

3.1 Solution of Problems (8) and (10)

To proceed any further we shall require some more

notation. Let

hα

,β

i =

Ω

(ω)β

(ω)dµ(ω) < ∞, (12)

= {hα

,β

m,n

i, j=1

∈ R

m×n

= [β

,. .. ,β

g−1

]

, y

= [β

,. .. ,β

]

, (13)

= [ϑ

,. .. ,ϑ

g−1

]

and z

= [ϑ

,. .. ,ϑ

]

(14)

The pseudo-inverse matrix (Golub and Loan,

1996) for any matrix M is denoted by M

†

. The symbol

O designates the zero matrix.

Lemma 1. (Torokhti and Howlett, 2007) If we de-

ﬁne w

= y

and w

= y

− P

where

= E

†

+ D

(I − E

†

) (15)

with D

an arbitrary matrix, then w

and w

are mu-

tually orthogonal random vectors.

ICINCO 2009 - 6th International Conference on Informatics in Control, Automation and Robotics

106

Let us ﬁrst consider problem (8) when B has the

complete variable memory η

= [g,g+1,..., n]

(see

Deﬁnition 3).

Let us partition B into four blocks K

, S

and

so that B =





, where

= {k

} ∈ R

×(g−1)

is a rectangular matrix,

= {ℓ

} ∈ R

×n

is a lower triangular matrix,

= {s

(1)

} ∈ R

(r−n

)×(g−1)

= {s

(2)

} ∈ R

(r−n

)×n

are rectangular matrices, and n

= n− g+ 1.

We have B(y) =



) + L

)

) + S

)



, where T

+ L

and S

= S

+ S

. Then

(B) = J

(1)

) + J

(2)

), (16)

where J

(1)

) = kz

− [T

) + L

)]k

Ω

(2)

) = kz

−[S

)+ S

)]k

Ω

. By anal-

ogy with Lemma 37 in (Torokhti and Howlett, 2007),

min

B∈M (r,n,η

)

(B) = min

(1)

) + min

(2)

Therefore, problem (8) is reduced to ﬁnding matrices

, L

, S

and S

such that

(1)

) = min

(1)

) (17)

and

(2)

) = min

(2)

). (18)

Taking into account the orthogonality of vectors

and w

and working in analogy with the argument

on pp. 348–352 in (Torokhti and Howlett, 2007), it

follows that matrices S

and S

are given by

= E

†

+ H

(I − E

†

) (19)

and

= E

†

+ H

(I − E

†

), (20)

where H

and H

are arbitrary matrices.

Next, to ﬁnd T

and L

we use the following no-

tation.

For r = 1,2,. .. ,ℓ, let ρ be the rank of the matrix

∈ R

×n

with n

= n− g+ 1, and let

1/2

= Q

w,ρ

(21)

be the QR-decomposition for E

1/2

where Q

w,ρ

∈

×ρ

and Q

w,ρ

= I and R

w,ρ

∈ R

ρ×n

is upper

trapezoidal with rank ρ. We write G

w,ρ

= R

w,ρ

and

use the notation G

w,ρ

= [g

,. .. ,g

] ∈ R

×ρ

where

∈ R

denotes the j-th column of G

w,ρ

. We also

write G

w,s

= [g

,. .. ,g

] ∈ R

×s

for s ≤ ρ to denote

the matrix consisting of the ﬁrst s columns of G

w,ρ

For simplicity, let us denote this G

:= G

w,s

. Next, let

= [1,0,0,0,. ..], e

= [0,1,0,0,. ..], e

[0,0,1,0,. ..], etc. denote the unit row vectors irre-

spective of the dimension of the space.

Finally, any square matrix M can be written as

M = M

∆

+ M

∇

where M

∆

is lower triangular and M

∇

is strictly upper triangular. We write k · k

for the

Frobenius norm.

Theorem 1. Let B ∈ M

(r,n,η

), i.e., the compres-

sor B is causal and has the complete variable mem-

ory η

= [g,g+ 1, .. .,n]

. Then the solution to prob-

lem (8) is provided by the matrix B

, which has the

form B





, where the blocks K

∈

×(g−1)

, S

∈ R

(r−n

)×(g−1)

and S

∈ R

(r−n

)×n

are rectangular, and the block L

∈ R

×n

is lower

triangular. These blocks are given as follows. The

block K

is given by

= T

− L

(22)

with

= E

†

+ N

(I − E

†

) (23)

where N

is an arbitrary matrix. The block L













, for each s = 1, 2,...,n

, is deﬁned by its

rows

= e

†

+ f

(I − G

†

) (24)

with f

∈ R

1×n

arbitrary. The blocks S

and S

are given by

= S

− S

(25)

and (20), respectively. In (25), S

is presented by

(19). The error associated with the compressor B

given by

kz− B

Ω

∑

s=1

∑

j=s+1

†

∑

j=1

1/2

−

∑

i=1

∑

j=1

†1/2

. (26)

Let us now consider problem (10) when the de-

compressor A has the complete variable memoryη

[r− q+ 1,...,r]

(see Deﬁnition 4).

FILTERING AND COMPRESSION OF STOCHASTIC SIGNALS UNDER CONSTRAINT OF VARIABLE FINITE

MEMORY

107

In analogy with our partitioning of matrix B, we

partition matrix A in four blocks K

, S

and S

so that A =





, where

= {k

} ∈ R

q×(r−q)

is a rectangular matrix,

= {ℓ

} ∈ R

q×q

is a lower triangular matrix, and

= {s

(1)

} ∈ R

(m−q)×(r−q)

= {s

(2)

} ∈ R

(m−q)×q

are rectangular matrices.

Let us partition z

so that z





with z

∈

(Ω,R

r−q

) and z

∈ L

(Ω,R

). We also write

= [α

.. ., α

r−q

]

and x

= [α

r−q+1

,. .. ,α

]

and denote by v

∈ L

(Ω,R

r−q

) and v

∈ L

(Ω,R

orthogonal vectors according to Lemma 1 as

= z

and v

= z

− P

where P

= E

†

+ D

(I − E

†

) with D

arbitrary matrix.

We write G

v,s

= [g

,. .. ,g

] ∈ R

q×s

where G

v,s

constructed from a QR-decomposition of E

1/2

, in

a manner similar to the construction of matrix G

w,s

Furthermore, we shall deﬁne G

:= G

v,s

Theorem 2. Let A ∈ M

(m,r,η

), i.e. the de-

compressor A is causal and has the complete variable

memory η

= [r − q+ 1,. .., r]

. Then the solution to

problem (10) is provided by the matrix A

, which has

the form A





, where the blocks K

∈

q×(r−q)

, S

∈ R

(m−q)×(r−q)

and S

∈ R

(m−q)×q

are

rectangular, and the block L

∈ R

q×q

is lower trian-

gular. These blocks are given as follows. The block

is given by

= T

− L

P (27)

with

= E

†

+ N

(I − E

†

) (28)

where N

is an arbitrary matrix. The block L













, for each s = 1,2,...,q, is deﬁned by its rows

= e

†

+ f

(I − G

†

) (29)

with f

∈ R

1×q

arbitrary. The blocks S

and S

are

given by

= S

− S

P, S

= E

†

+ H

(I − E

†

(30)

where

= E

†

+ H

(I − E

†

) (31)

and H

are arbitrary matrices.

The error associated with the de-compressor A

given by

kx− A

Ω

∑

s=1

∑

j=s+1

†

(32)

∑

j=1

1/2

−

∑

i=1

∑

j=1

†1/2

. (33)

4 SIMULATIONS

The following simulations and numerical results illus-

trate the performance of the proposed approach.

Our ﬁlter F

= A

has been applied to compres-

sion, ﬁltering and subsequent restoration of the refer-

ence signals given by the matrix X ∈ R

256×256

. The

matrix X represents the data obtained from an aerial

digital photograph of a plant

presented in Fig. 1.

We divide X into 128 sub-matrices X

∈ R

m×q

with i = 1,...,16, j = 1, .. ., 8, m = 16 and q = 32

so that X = {X

}. By assumption, the sub-matrix

is interpreted as q realizations of a random vec-

tor x ∈ L

(Ω,R

) with each column representing a

realization. For each i = 1,...,16 and j = 1, .. ., 8,

observed data Y

were modelled from X

in the form

= X

•

rand

(16,32)

(ij)

Here, • means the Hadamard product and

rand

(16,32)

(ij)

is a 16× 32 matrix whose randomly-

chosen elements are uniformly distributed in the

interval (0,1).

The proposed ﬁlter F

has been applied to each

pair {X

, Y

}. Each pair {X

, Y

} was processed by

compressors and de-compressors with the complete

variable memory. We denote B

= B

and A

= A

for such a compressor and de-compressor determined

by Theorems 1 and 2, respectively, so that

∈ M

(r,n,η

) and A

∈ M

(m,r,η

)

The database is available in

http://sipi.usc.edu/services/database/Database.html.

ICINCO 2009 - 6th International Conference on Informatics in Control, Automation and Robotics

108

50 100 150 200 250

100

150

200

250

(a) Given reference signals.

50 100 150 200 250

100

150

200

250

(b) Observed data.

50 100 150 200 250

100

150

200

250

with the complete variable memory.

Figure 1: Illustration of simulation results.

where n = m = 16, r = 8, η

= {η

}

k=1

with η



12+ k− 1, if k = 1,...,4,

16, if k = 5, ...,16

and η

= {η

}

j=1

with η



6+ j− 1, if j = 1,2,

8, if k = 3,...,16

. In this case,

the optimal ﬁlter F

is denoted by F

so that

= A

. We write

= max

− F

for a maximal error associated with the ﬁlter F

over

all i = 1,...,16 and j = 1, .. .,8. The compression

ratio was c = 1/2. We obtained J

= 3.3123e+ 005.

The results of simulations a are presented in Fig.

1 (a) - (c).

REFERENCES

A. Torokhti, P. H. (2001). Optimal ﬁxed rank transform of

the second degree. IEEE Trans. Circuits & Syst., II,

Analog & Digit. Signal Processing, 48(3):309–315.

Gimeno, V. (1987). Obtaining the eeg envelope in real time:

a practical method based on homomorphic ﬁltering.

Neuropsychobiology, 18:110–112.

Golub, G. and Loan, C. V. (1996). Matrix Computation.

Johns Hopkins Univ. Press.

H. Kim, G.H. Golub, H. P. (2005). Missing value estimation

for dna microarray gene expression data: local least

squares imputation. Bioinformatics, 21:211–218.

Hua, Y. and Liu, W. Q. (1998). Generalized karhunen-lo`eve

transform. IEEE Signal Process Letters, 5:141–143.

Hua, Y. and Nikpour, M. (1999). Computing the reduced

rank wiener ﬁlter by iqmd. IEEE Signal Processing

Letters, 6(9):240–242.

Jolliffe, I. (1986). Principal Component Analysis. Springer

Verlag.

S. Friedland, A. Niknejad, M. K. H. Z. (2006). Fast monte-

carlo low rank approximations for matrices. Proc.

IEEE Conference SoSE, pages 218–223.

T. Zhang, G. G. (2001). Rank-one approximation to high

order tensors. SIAM J. Matrix Anal. Appl., 23.

Torokhti, A. and Howlett, P. (2007). Computational Meth-

ods for Modelling of Nonlinear Systems. Elsevier.

FILTERING AND COMPRESSION OF STOCHASTIC SIGNALS UNDER CONSTRAINT OF VARIABLE FINITE

MEMORY

109