Approximate Homomorphic Pre-Processing for CNNs

Shabnam Khanna and Ciara Rafferty

a

CSIT, Queen’s University Belfast, U.K.

Keywords:

Homomorphic Evaluation, Approximate Computing, ReLU, Convolutional Neural Networks.

Abstract:

Homomorphic encryption (HE) allows computations on encrypted data, making it desirable for use in privacy-

preserving data analytics. However, HE function evaluation is computationally intensive. Approximate

computing (AC) allows a trade-off between accuracy, memory/energy usage and running time. Polynomial

approximation of the Rectiﬁed Linear Unit (ReLU) function, a key CNN activation function, is explored and

AC techniques of task-skipping and depth reduction are applied. The most accurate ReLU approximations

are implemented in nGraph-HE’s Cryptonets CNN using a SEAL backend, resulting in a minimal decrease in

training accuracy of 0.0011, no change in plaintext classiﬁcation accuracy, and a speed-up of 47%.

1 INTRODUCTION

Data privacy and security is ever more vital with per-

sonal data stored on the cloud. Homomorphic encryp-

tion (HE) enables computation directly on encrypted

data, reducing risk of privacy-loss. HE has great po-

tential for privacy-preserving shared computation and

secure outsourced analysis. Despite progress in re-

cent years regarding computations on encrypted data

((Graepel et al., 2012), (Chen et al., 2018),(Boemer

et al., 2019), (Chou et al., 2020)), homomorphic eval-

uation is much less efﬁcient than unencrypted data

evaluation. Moreover, non-polynomial function eval-

uation is non-trivial, and often not possible in many

HE libraries, where functions typically need to be in

polynomial form.

Many ML applications use non-polynomial func-

tions, e.g. logistic regression- or classiﬁcation-based

neural networks (NNs) (Chen et al., 2018), (Chou et al.,

2020). Neural networks consist of many layers, each

containing non-polynomial functions, e.g. logistic,

exponential, and sine functions, often approximated

as Taylor functions when evaluated homomorphically

(Cheon et al., 2017). Conventionally, higher degree

Taylor polynomials result in a higher accuracy levels,

developing into inefﬁcient, larger depth calculations.

This research investigates non-polynomial function

approximation, to balance performance and accuracy.

Speciﬁcally, the Rectiﬁed Linear Unit (ReLU) function

is targeted, used in Convolutional Neural Networks

(CNNs), and the impact of applying homomorphic-

a

https://orcid.org/0000-0002-3670-366X

friendly, AC-adapted approximations of ReLU to an

(unencrypted) CNN for image classiﬁcation is consid-

ered. In this research the approximate HE scheme,

CKKS (Cheon et al., 2017), is used due to its compu-

tation expressed as ﬂoating point arithmetic.

This paper is structured as follows: Section 2 de-

tails approximate computing. Section 3 introduces

polynomial approximations of ReLU and AC-adapted

approaches implemented homomorphically. Section 4

deploys the most accurate approximations in an open-

source CNN to determine impact on accuracy.

2 AC TECHNIQUES

One main motivation to deploy AC techniques is to en-

sure energy-efﬁcient technology (Barua and Mondal,

2019). Applications where this performance-accuracy

trade-off is acceptable include machine learning, sig-

nal processing, and big data analytics (Barua and Mon-

dal, 2019). In (Khanna and Rafferty, 2020), task skip-

ping and depth reduction are used to adapt logistic

and exponential functions, represented polynomially

using Taylor series approximation and evaluated ho-

momorphically using CKKS in the Microsoft SEAL

Library (SEAL, 2018). This involved skipping the

highest order term, skipping lower order terms, reduc-

ing the depth by replacing the highest order term with

the highest order term in the depth below. For the ex-

ponential function, the depth reduced approximation

had an average error of

2.42

x

10

−5

compared with

an average error of 0.333321 to the Taylor exponen-

710

Khanna, S. and Rafferty, C.

Approximate Homomorphic Pre-Processing for CNNs.

DOI: 10.5220/0012085400003555

In Proceedings of the 20th International Conference on Security and Cryptography (SECRYPT 2023), pages 710-715

ISBN: 978-989-758-666-8; ISSN: 2184-7711

Copyright

c

2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

tial function approximation with no adaptations, with

a 35% speed up. For the logistic function, skipping

the highest order term (

x

9

), provided a 45.5% speed

up with an average error of

2.08

x

10

−6

as opposed

to

1.52

x

10

−7

when no AC technique is applied to

its Taylor series approximation. In this research the

depth reduction process is formalised and applied to

another important non-linear function, ReLU, given its

signiﬁcant usage in CNNs. Our approach is as follows:

1.

Task Skipping (TS): All but the highest and lowest

two terms of the approximation are skipped.

2.

Depth Reduction (DR): The MATLAB polyﬁt

function is used to ﬁnd the depth reduced approxi-

mation (Mathworks, 2021), using least squares to

approximate a polynomial.

3 ReLU

A CNN is made up of several layers, including an acti-

vation layer, which takes a single number as input and

evaluates a non-linear mathematical operation, adding

non-linearity into the network and enabling complex

predictions. Many non-linear functions, such as Sig-

moid, hyperbolic tangent and Softmax, can be used in

an activation layer. One important activation function

is ReLU , deﬁned as:

f (x) = max(0, x)

(Hesamifard

et al., 2019). Since non-linear functions must be repre-

sented polynomially for homomorphic evaluation, this

research follows (Chabanne et al., 2017) and (Hesam-

ifard et al., 2019) for polynomial approximations of

ReLU. AC techniques from Section 2 are applied to

these approximations, comparing run-time and accu-

racy. Although the CNN input values are in the interval

[0,255], they are not necessarily the ReLU input val-

ues. Scaling and batch normalisation processes are

used when working with inputs for CNN layers; a

smaller input range is used, assuming that inputs to the

ReLU layer can be scaled.

Cryptonets (Dowlin et al., 2016) provides an early

research into homomorphic CNNs, using the MNIST

dataset classiﬁcation (LeCun et al., 1998). (Chabanne

et al., 2017) use “polyﬁt” in Python and the Taylor

expansion of Softplus as polynomial approximations

of ReLU with a classiﬁcation accuracy of 99.30% com-

pared to the Cryptonets accuracy of 98.95% (Dowlin

et al., 2016). (Hesamifard et al., 2019) list ﬁve meth-

ods to approximate ReLU: Numerical analysis, Taylor

Series, Standard Chebyshev polynomials, Modiﬁed

Chebyshev polynomials and their approach based on

the derivative of ReLU. They show an encrypted re-

sult of classiﬁcation accuracy of 99.52% and a run-

time of 320s compared to 697s in Cryptonets (Dowlin

et al., 2016). Following this, 12 ReLU approxima-

tions are used in this research, as follows: 5 approx-

imations of different orders using “polyﬁt” from the

Python numpy package (community, 2020); 3 approxi-

mations derived from Taylor polynomials of Softplus,

f (x) = ln(1 + e

x

)

; and 4 Chebyshev approximations

with input range [0, 250].

All 12 approximations are implemented in PAL-

ISADE, with approximations of order at least 4 and

8 being depth reduced and task skipped respectively.

Table 1 shows the performance and average error of

these approximations. To standardise the methodology

for obtaining DR polynomial approximations and max-

imise accuracy, MATLAB polyﬁt is used (Mathworks,

2021), where the highest power required for one depth

below the actual depth of the approximation is spec-

iﬁed. For TS approximations (applied to Chebyshev

polynomial approximations only), skipping various

terms were attempted to ﬁnd the optimal combination.

Note subscript notations

DR

(x)

and

T S

(x)

correspond

to the DR and TS approximations of the preceding

equation. Equation numbers correspond to those in

Table 1. Each coefﬁcient is deﬁned (and rounded) to

2 decimal places for neatness. However, exact coefﬁ-

cient values are used in the evaluation.

3.1 Polyﬁt

Using “polyﬁt” in Python (community, 2020), as in

(Chabanne et al., 2017) give the following approxima-

tions, with equations 4 and 6 being the depth-reduced

approximations of equations 3 and 5 respectively.

a(x) ≈ 0.20 + 0.50x + 0.20x

2

(1)

b(x) ≈ 0.20 + 0.50x + 0.20x

2

− 0.02x

3

(2)

c(x) ≈ 0.15 + 0.50x + 0.30x

2

− 0.00x

3

− 0.04x

4

(3)

c

DR

(x) ≈ 2.34e+06 − 1.84e+05x

+ 3.24e+03x

2

− 19.79x

3

(4)

d(x) ≈ 0.15 + 0.50x + 0.30x

2

+ 0.00x

3

− 0.02x

4

(5)

d

DR

(x) ≈ 1.01e+06 − 7.96e+04x

+ 1.40e+03x

2

− 8.57x

3

(6)

e(x) ≈ 0.12 + 0.5x + 0.37x

2

− 0.04x

4

+ 0.00x

6

(7)

3.2 Taylor Series

The Taylor expansion of the Softplus function,

f (x) =

ln(1+e

x

)

, provides a good approximation of the ReLU

function (Chabanne et al., 2017). ReLU approxima-

tions in Equations 8 to 11 with equations 10 and 12

being the DR approximations of equations 9 and 11 re-

spectively. All these approximations using the Taylor

expansion of Softplus are shown in Figure 1.

Approximate Homomorphic Pre-Processing for CNNs

711

f (x) ≈ 0.64 + 0.5x + 0.13x

2

(8)

g(x) ≈ 0.64 + 0.5x + 0.13x

2

− 0.01x

4

(9)

g

DR

(x) ≈ 3.14e+05 − 2.46e+04x

+ 4.35e+02x

2

− 2.65x

3

(10)

h(x) ≈ 0.64 + 0.5x + 0.13x

2

− 0.01x

4

+ 0.00x

6

(11)

h

DR

(x) ≈ −3.92e+09 + 2.77e+08x

− 4.08e+06x

2

+ 1.66e+04x

3

(12)

-4 -3 -2 -1 0 1 2 3 4

Input Value

-1

0

1

2

3

4

5

Evaluation of ReLU

ReLU Exact

f - Taylor expansion of Softplus, order 2

g - Taylor expansion of Softplus, order 4

gDR - Taylor expansion of Softplus, order 3

h - Taylor expansion of Softplus, order 6

hDR - Taylor expansion of Softplus, order 3

Figure 1: Taylor expansions of Softplus, along with the exact

ReLU function and the DR approximations.

3.3 Chebyshev Approximations

Chebyshev polynomial approximations are obtained

using the approach outlined in (Hesamifard et al.,

2017), using the Sigmoid function to approximate

ReLU. The authors of (Hesamifard et al., 2017) per-

form polynomial interpolation to approximate the Sig-

moid function using the roots of Chebyshev polynomi-

als. In this research the roots of a standard Chebyshev

polynomial are used to obtain Chebyshev approxima-

tions of ReLU for degrees 4, 8, 12, 16 polynomials in

the input range [0, 250]. The assumption is that any

input can be scaled as close to this input range as possi-

ble. DR and TS are applied to Chebyshev polynomial

approximations of orders 8, 12 and 16. For TS, skip-

ping all but the largest order term, the

x

,

x

2

, and the

(in this case non-existent) constant term provided the

most accurate TS approximations of the Chebyshev

approximations of ReLU. The Chebyshev polynomials

and their DR and RS approximations are deﬁned as

follows (as in Figure 2.):

- Chebyshev order 4 approximation:

i(x) ≈(5.00e+40) ∗ (1.0e-41)x + (1.0e-41) ∗ (1.14e+40)x

2

+ (1.0e-41) ∗ (7247)x

3

− (1.0e-41) ∗ (1.81e+38)x

4

(13)

- Chebyshev order 8

j

O

(x)

with depth-DR,

j

DR

(x)

and

TS, j

T S

(x) approximations:

j

O

(x) ≈ 0.50x + 0.12x

2

− 7.50e-16x

3

− 0.00x

4

− 5.96e-18x

5

+ 0.00x

6

− 1.98e-19x

7

− 3.05e-6x

8

(14)

j

DR

(x) ≈ 0.05 + 0.50x + 0.08x

2

− 8.95e-16x

3

(15)

j

T S

(x) ≈ 0.50x + 0.12x

2

− 0.00x

8

(16)

- Chebyshev order 12

k

O

(x)

with DR,

k

DR

(x)

and

TS, k

T S

(x) approximations:

k

O

(x) ≈ 0.50x + 0.13x

2

+ 3.54e-18x

3

− 0.01x

4

+ 1.19e-19x

5

+ 0.00x

6

− 1.26e-19x

7

− 0.00x

8

+ 8.11e-21x

9

+ 4.93e-72x

10

− 1.92e-22x

11

− 7.09e-9x

12

(17)

k

DR

(x) ≈ 0.01 + 0.50x + 1.12e-11x

2

− 3.62e-17x

3

− 0.02x

4

(18)

k

T S

(x) ≈ 0.50x + 0.13x

2

− 7.09e-9x

12

(19)

- Chebyshev order 16

l

O

(x)

with DR,

l

DR

(x)

and TS,

l

T S

(x) approximations:

l

O

(x) ≈ 0.50x + 0.13x

2

− 5.94e-17x

3

− 0.01x

4

+ 2611e-17x

5

+ 3.40e-4x

6

− 5.81e-18x

7

− 2.32e-5x

8

+ 6.99e-19x

9

+ 1.37e-6x

10

− 4.75e-20x

11

− 5.93e-8x

12

+ 1.71e-217x

13

+ 1.57e-9x

14

− 2.52e-23x

15

− 1.86e-11x

16

(20)

l

DR

(x) ≈ 0.00 + 0.5x + 0.12x

2

− 6.52e-17x

3

− 0.00x

4

− 2.77e-19x

5

7.79e-05x

6

+ 2.23e-19x

7

(21)

l

T S

(x) ≈ 0.50x − 0.13x

2

− 1.86e-11x

16

(22)

-4 -3 -2 -1 0 1 2 3 4

Input Value

-1

-0.5

0

0.5

1

1.5

2

2.5

3

3.5

4

Evaluation of ReLU

ReLU Exact

i - Chebyshev, order 4

j - Chebyshev, order 8

jTS - Chebyshev, order 8

jDR - Chebyshev, order 3

k - Chebyshev, order 12

kTS - Chebyshev, order 12

kDR - Chebyshev, order 3

l - Chebyshev, order 16

lTS - Chebyshev, order 16

lDR - Chebyshev, order 7

Figure 2: Chebyshev polynomial approximations, with the

exact ReLU function and TS and DR approximations.

All 12 original approximations (equations 1, 2,

3, 5, 7, 9, 11, 13, 14, 17, 20) are shown in Figure

3. The Chebyshev approximations are more accurate

than other approaches, with the most accurate being

the largest order (16) Chebyshev approximation. Thus,

this is the selected approximation for use in the CNN.

Before applying to a CNN, the ReLU approximations

are implemented in PALISADE to analyse the perfor-

mance/accuracy trade off.

SECRYPT 2023 - 20th International Conference on Security and Cryptography

712

-4 -3 -2 -1 0 1 2 3 4

Input Value

-8

-6

-4

-2

0

2

4

6

Evaluation of ReLU

ReLU Exact

a - Python Polyfit, order 2

b - Python Polyfit, order 3

c - Python Polyfit, order 4

d - Python Polyfit, order 5

e - Taylor expansion of Softplus, order 2

f - Taylor expansion of Softplus, order 4

g - Taylor expansion of Softplus, order 6

h - Chebyshev, order 4

i - Chebyshev, order 8

j - Chebyshev, order 12

k - Chebyshev, order 16

Figure 3: A ﬁgure showing all 12 ‘original’ polynomial

approximations of, and the exact the ReLU function.

3.4 Implementation of ReLU

Approximations in PALISADE

This research was implemented on a Linux VM run-

ning on an Intel Core i5-8250U CPU using the CKKS

HE scheme in the Palisade library v1.10.5 (Palisade,

2021). All values in this section are provided to 4 dec-

imal places. The ring size used is 32768 with 8192 in-

put values in the range [-4,4] and other parameters set

to allow 128-bit security according to the HE Standard

(Albrecht et al., 2018). The ReLU approximations out-

lined in Section 3 are implemented and analysed with

respect to accuracy vs performance. The input of raw

form images will always be consistent (input values

between 0 and 255), but change after each CNN func-

tion evaluation, and when evaluated homomorphically,

the inputs per layer are unknown in their raw form and

as they change vastly layer-by-layer. Thus, having an

approach to standardise the input values is important.

Since the actual error for each input value varies

signiﬁcantly, the average error calculated across the

input range as absolute errors is provided, the impact

of this when running in a CNN may differ. The eval-

uation error is deﬁned as the average error across the

input range for the evaluation of the relevant approx-

imation as compared to the evaluation of the exact

ReLU function. Note two points regarding average

error and amortised running time. Firstly, the ring size

used enables all approximations to be evaluated to the

estimated precision of around 39 bits. However, this

small ring size does pose a larger evaluation error. To

maintain the same parameter settings (including de-

cryption bit accuracy) for all 12 approaches, without

changing the security setting, requires the ring size to

be set as 2048 and thus the slot size (HE batch size)

is 1024. Increasing the ring and slot size signiﬁcantly

decreases the average evaluation error, however this

impacts on the security level and requires further study.

The performance run-time is the time taken when eval-

uating homomorphically and represented as amortised

run-time, i.e. the run-time per each input value. Since

Table 1: Table shows amortised run-time (milliseconds) and

average error for HE implementation for ReLU approxima-

tions implemented in PALISADE. The amortised run-time

for non-HE implementation is omitted as it is instantaneous.

Equation Degree

Amortised

Run-time

Average

Error

(non-HE)

Average

Error

(HE)

Polyﬁt approximations

1 2 50.5 ms 1.7031 x 10

−4

0.3134

2 3 99.2 ms 4.1638 x 10

−5

0.3256

3 4 143.1 ms 8.5913 x 10

−4

1.2198

4 (3 DR) 3 97.1 ms 5.8239 x 10

−4

1.5744

5 5 142.6 ms 1.6167 x 10

−4

0.1695

6 (5 DR) 3 97.4 ms 4.1846 x 10

−5

0.2565

7 6 147.4 ms 1.8176 x 10

−5

0.0847

Taylor Series approximations

8 2 48.7 ms 7.8015 x 10

−5

0.3020

9 4 100.1 ms 8.4485 x 10

−5

0.2503

10 (9 DR) 3 96.5 ms 4.3796 x 10

−5

0.2840

11 6 146.9 ms 6.5515 x 10

−5

0.2121

12 (11 DR) 3 96.2 ms 4.5544 x 10

−5

0.2129

Chebyshev approximations

13 4 92.6 ms 3.8584 x 10

−4

0.8359

14 8 233.3 ms 8.2333 x 10

−5

0.4877

15 (14 DR) 3 96.4 ms 7.0558 x 10

−5

0.4896

16 (14 TS) 8 128.9 ms 2.5636 x 10

−5

0.3576

17 12 306.7 ms 8.2396 x 10

−5

0.4880

18 (17 DR) 4 139.1 ms 8.4559 x 10

−5

0.4880

19 (17 TS) 12 158.7 ms 1.4595 x 10

−5

0.3419

20 16 393.8 ms 8.2397 x 10

−5

0.4897

21 (20 DR) 7 219.5 ms 8.2395 x 10

−5

0.4885

22 (20 TS) 16 166.5 ms 9.7406 x 10

−6

0.3402

the run-time for each function evaluation is roughly the

same, no matter the slot size, understanding the total

run-time across all input slots may be more useful.

In Table 1 the equation number ‘

m

(

n

DR/TS)’,

where

m, n

are numbers, is read as equation

m

, which is

equation

n

with depth-reduction (DR) or task skipping

(TS), e.g. equation 4 (3 DR) is equation 4, the depth-

reduced approximation of equation 3.

Comparing the trends between the evaluation of

the TS, DR and ‘original’ approximations: Although

the TS approach has the same polynomial order as

the ‘original’ approximation, evaluation run-time is

nearly halved, running similarly as DR versions. The

TS approach skips every term between the

x

2

and the

highest powered term of the original approximation

and the DR approach uses ‘polyﬁt’ in MATLAB to

ﬁnd an approximation as close to the original evalua-

tion as possible but at a lower computational depth. As

Table 1 shows, the DR approximations have similar av-

erage error when compared to original approximations.

However, unexpectedly, every TS approximation is, on

average, more accurate than the original and the DR

polynomial approximations. When evaluating the TS

approximations (unencrypted) the average accuracy

is more than double the average accuracy of the orig-

inal polynomial approximation yet when evaluating

Approximate Homomorphic Pre-Processing for CNNs

713

homomorphically is around a third more accurate than

the original approximation. Figure 2 supports these

conclusions across the input range.

It is important to remember that for performance

(non-HE), only the approximation affects the error, yet

when the same polynomial is evaluated homomorphi-

cally, in addition to the run-time error, there exists an

error created by the CKKS scheme itself. Thus the

analysis focuses on comparing the average errors for

the ‘original’ approximations with the approximations

adapted with AC techniques, rather than comparing

the encrypted vs unencrypted average errors.

Figures 1, 2 and 3 provide a more detailed overview

of errors across the input range than the average evalu-

ation error in Table 1. Table 1 shows that for the ﬁrst

and second set of approximations, (Python polyﬁt and

Taylor expansion of Softplus), when DR is applied

to approximations 3 and 5 and 9 and 11, the average

evaluation error on unencrypted evaluation is smaller

than the error of the original approximation. How-

ever, when evaluated homomorphically, the average

error is much larger than the original error, due to the

additional error in the approximate HE scheme itself.

For the Chebyshev approximations, and higher order

approximations, the DR approximations provide a sim-

ilar or slightly larger, average error. However, the TS

approximations provide a slightly smaller evaluation

error than the ‘original’ approximation, makings the

TS approximations favourites to implement.

4 APPROXIMATION OF ReLU

FOR CNNs

To demonstrate the impact of using ReLU approxi-

mations on an application, the Cryptonets network

(Dowlin et al., 2016) is targeted, implemented in Intel’s

nGraph-HE (Boemer et al., 2019), where the original

x

2

activation function replaced with the ReLU function.

This research applies the approaches from Section 3

to this speciﬁc Cryptonets CNN for image classiﬁca-

tion; The implementation of ReLU in Section 3 is on

input ranges [-4,4], however for the CNN used in this

section, the input range is [-255, 255], to maintain the

CNN structure without adding extra scale layers, and

enable fair comparison with previous research. Thus,

the Chebyshev polynomial has changed, as it is input

dependent. However the Chebyshev polynomials re-

main the most accurate ReLU approximations over

the input range. Table 1 and Figure 2 show that the

larger order Chebyshev polynomials and the larger or-

der Taylor expansion of SoftPlus are more accurate

ReLU approximations. DR and TS are applied to all

Chebyshev polynomials (orders 8, 12, 16) and Table 1

and Figure 2 show that TS approximations had higher

evaluation accuracy than DR approximations.

4.1

Results: ReLU CNN Approximations

This research was implemented on a Linux VM run-

ning on an Intel Core i5-8250U CPU using the CKKS

HE scheme in SEAL (SEAL, 2018), and Cryptonets

(Dowlin et al., 2016) implementated in nGraph HE

(Boemer et al., 2019). All ﬁgures are provided to 4

decimal places. Table 2 shows results when the Cheby-

shev order 16 approximation of ReLU and TS and

DR versions, were applied to the Cryptonets CNN

(Dowlin et al., 2016) in nGraph-HE (Boemer et al.,

2019) for image classiﬁcation. This change was made

in all ReLU layers, a total of 4 layers. Table 2 gives

average CNN values after 5 runs for each approach.

The aim is to have the lowest possible reduction in ac-

curacy, assuming an acceptable reduction in accuracy

of 0.5% - 1% These implementations were trained in

unencrypted form and then tested again using unen-

crypted data, using a SEAL backend - an approach

recommended for debugging (Boemer et al., 2019).

Table 2: Classiﬁcation accuracy and run-time of ReLU ap-

proximations (average over 5 runs), based on the Chebyshev

order 16 approximation, applied to the Cryptonets CNN im-

plemented in nGraph-HE.

Chebyshev 16: No change

Task

Skipping

Depth

Reduction

(x

7

)

Depth

Reduction

(x

3

)

Classiﬁcation Accuracy 0.9900 1.0000 0.9600 0.1500

Train (Test) Accuracy 0.9838 0.9827 0.9169 0.1028

Train (Test) Loss 0.0491 0.0541 0.2939 2.3050

Run-time (Train) µs 390.0 201.0 259.0 193.0

Run-time (Test-HE (CPU)) 237.0 84.0 133.0 86.0

% Speed-up (Train) N/A 48% 34% 51%

% Speed-up (Test-HE (CPU)) N/A 46% 29% 29%

Compared to the approximations evaluated in PAL-

ISADE, where TS approximations had a higher evalu-

ation accuracy than DR approximations, when applied

to the CNN the DR approximations provide a higher

classiﬁcation accuracy. This may be because, ﬁrstly,

these results have high variance and need more runs

for a stable average, and secondly, the TS approxima-

tion provides a more accurate CNN classiﬁcation when

used in place of ReLU. These conclusions should be

understood cautiously; due to the structure of vary-

ing CNNs, different ReLU approximations could fare

better in terms of performance. Following Table 1,

although DR approximations have a faster run-time,

the accuracy is unacceptably low. Thus, the TS ReLU

approximation fares best in terms of accuracy and per-

formance speed-up. Considering the estimated perfor-

mance details of an encrypted CNN given in (Hesam-

ifard et al., 2019), and the impact of DR and TS on

encrypted ReLU, shown in Tables 1 and 2, the perfor-

SECRYPT 2023 - 20th International Conference on Security and Cryptography

714

mance of approximated ReLU can be estimated to be

slightly faster than the 320s taken in (Hesamifard et al.,

2019) with a slight accuracy reduction.

5 CONCLUSIONS

This research proposed novel approximations of ReLU

to ensure efﬁcient homomorphic evaluation targeting

CNNs. For unencrypted ReLU approximation, the

most accurate approximation is Chebyshev order 16

(TS). If speed is prioritised, DR Chebyshev approxima-

tions orders 8 and 12 are recommended. The DR ap-

proximation of the Taylor expansion of Softplus (order

4) is also suitable. For encrypted ReLU approximation,

TS Chebyshev polynomial approximations orders 8,

12, 16 provide signiﬁcant speed-up. When applying

Chebyshev approximations (order 16, with DR and

TS) to the CNN, after 5 runs of training and classi-

fying, the TS approximation, equation 22, provides a

48% speed-up in training run-time and a 0.0011 aver-

age decrease in classiﬁcation accuracy. Overall, using

TS and DR approximations in CNNs do not have sig-

niﬁcant negative impact on accuracy and performance.

This research demonstrates signiﬁcant opportunities

for acceleration of HE evaluation using AC techniques,

sacriﬁcing minimal accuracy and helping realise the

potential of HE for large scale data analytics.

ACKNOWLEDGEMENTS

This research was supported partly by a Thales UK

placement - thanks to Adrian Waller and Naomi Farley

for their support.

REFERENCES

Albrecht, M., Chase, M., Chen, H., Ding, J., Goldwasser,

S., Gorbunov, S., Halevi, S., Hoffstein, J., Laine, K.,

Lauter, K., Lokam, S., Micciancio, D., Moody, D.,

Morrison, T., Sahai, A., and Vaikuntanathan, V. (2018).

Homomorphic Encryption Security Standard. Tech-

nical report, HomomorphicEncryption.org, Toronto,

Canada.

Barua, H. B. and Mondal, K. C. (2019). Approximate Com-

puting: A Survey of Recent Trends—Bringing Green-

ness to Computing and Communication. Journal of

The Institution of Engineers (India): Series B.

Boemer, F., Lao, Y., Cammarota, R., and Wierzynski, C.

(2019). ngraph-he: a graph compiler for deep learning

on homomorphically encrypted data. In Palumbo, F.,

Becchi, M., Schulz, M., and Sato, K., editors, Pro-

ceedings of the 16th ACM International Conference on

Computing Frontiers, CF 2019, Alghero, Italy, April

30 - May 2, 2019, pages 3–13. ACM.

Chabanne, H., de Wargny, A., Milgram, J., Morel, C., and

Prouff, E. (2017). Privacy-preserving classiﬁcation

on deep neural network. IACR Cryptol. ePrint Arch.,

2017:35.

Chen, H., Gilad-Bachrach, R., Han, K., Huang, Z., Jalali, A.,

Laine, K., and Lauter, K. (2018). Logistic regression

over encrypted data from fully homomorphic encryp-

tion. BMC Medical Genomics, 11(4).

Cheon, J. H., Kim, A., Kim, M., and Song, Y. S. (2017).

Homomorphic Encryption for Arithmetic of Approx-

imate Numbers. In Takagi, T. and Peyrin, T., editors,

Advances in Cryptology - ASIACRYPT 2017 - 23rd

International Conference on the Theory and Applica-

tions of Cryptology and Information Security, Hong

Kong, China, December 3-7, 2017, Proceedings, Part

I, volume 10624 of Lecture Notes in Computer Science,

pages 409–437. Springer.

Chou, E., Gururajan, A., Laine, K., Goel, N., Bertiger, A.,

and Stokes, J. (2020). Privacy-Preserving Phishing

Web Page Classiﬁcation Via Fully Homomorphic En-

cryption. pages 2792–2796.

community, T. S. (2008-2020). numpy.polyﬁt.

https://numpy.org/doc/stable/reference/generated/

numpy.polyﬁt.html.

Dowlin, N., Gilad-Bachrach, R., Laine, K., Lauter, K.,

Naehrig, M., and Wernsing, J. (2016). CryptoNets:

Applying Neural Networks to Encrypted Data with

High Throughput and Accuracy. Technical Report

MSR-TR-2016-3.

Graepel, T., Lauter, K., and Naehrig, M. (2012). ML Con-

ﬁdential: Machine Learning on Encrypted Data. In

Proceedings of the 15th International Conference on

Information Security and Cryptology, ICISC’12, page

1–21, Berlin, Heidelberg. Springer-Verlag.

Hesamifard, E., Takabi, D., and Ghasemi, M. (2017). Cryp-

toDL: Deep Neural Networks over Encrypted Data.

Hesamifard, E., Takabi, H., and Ghasemi, M. (2019). Deep

Neural Networks Classiﬁcation over Encrypted Data.

In Proceedings of the Ninth ACM Conference on Data

and Application Security and Privacy, CODASPY ’19,

page 97–108, New York, NY, USA. Association for

Computing Machinery.

Khanna, S. and Rafferty, C. (2020). Accelerating Homo-

morphic Encryption using Approximate Computing

Techniques. In Samarati, P., di Vimercati, S. D. C.,

Obaidat, M. S., and Ben-Othman, J., editors, Proceed-

ings of the 17th International Joint Conference on e-

Business and Telecommunications, ICETE 2020 - Vol-

ume 2: SECRYPT, Lieusaint, Paris, France, July 8-10,

2020, pages 380–387. ScitePress.

LeCun, Y., Cortes, C., and Burges, C. J. (1998). The MNIST

database of handwritten digits.

Mathworks (2006-2021). polyﬁt. https://www.mathworks.

com/help/matlab/ref/polyﬁt.html.

Palisade (2021). PALISADE (version 1.10.6). https:

//palisade-crypto.org/.

SEAL (2018). Simple Encrypted Arithmetic Library (release

3.1.0). https://github.com/Microsoft/SEAL. Microsoft

Research, Redmond, WA.

Approximate Homomorphic Pre-Processing for CNNs

715