A DOS ATTACK AGAINST THE INTEGRITY-LESS ESP (IPSEC)

Ventzislav Nikov

Philips TASS and AppTech

Leuven, Belgium

Keywords:

IPSec (ESP) Standard, Denial of Service Attack.

Abstract:

This paper describes a new practical DoS attack that can be mounted against the “encryption-only” conﬁgura-

tion (i.e. without authenticated integrity) of ESP as allowed by IPSec.

This ﬁnding can serve as a strong argument to convince those in charge of the IPSec standardization to improve

it by banning the “encryption-only” conﬁguration from the standard.

1 INTRODUCTION

As illustrated in (M. Bellare, 2004; Bellovin, 1996;

N. Borisov, 2001; Canvel et al., 2003; C. McCubbin,

2000; T. Yu, 2004), it is, today, common knowledge

in the cryptographic community that encryption with-

out authenticated integrity opens the door to various

attacks. More recently, the authors of (K. Paterson,

2005) illustrate this argument on the encryption-only

conﬁguration allowed by IPSec for the secure payload

encapsulation (ESP) (Kent, 2005).

IPSec encryption-only conﬁguration was shown to

be exposed to variety of “initialization vector attacks”

(C. McCubbin, 2000). The authors of (C. McCubbin,

2000) analyze the security risks posed on encryption-

only IPSec when it is used as intermediate layer of the

protocol stack. But their analysis has a limited scope -

just the implication on the next (possible) upper-layer

protocol (e.g. TCP, UDP, IP). We develop further their

idea of properly modifying certain ﬁelds in the pay-

loads of the whole protocol stack (e.g. IP/UDP/RTP

or IP/TCP). In this way we build a more powerful at-

tack.

The authors of (K. Paterson, 2005) have exposed

even more serious weakness of the encryption-only

conﬁguration on IPSec. It has been shown that the

“encryption-only” conﬁguration is subject to a variety

of ciphertext-only attacks, i.e. revealing the encryp-

tion key. The attack consists of two phases: in the

ﬁrst phase the attacker modiﬁes certain ﬁelds in the

ﬁrst few blocks of the ciphertext such that the upper-

layer protocol submits either an error message or the

decryption of the modiﬁed datagram directly to the

attacker’s machine. The second phase proceeds with

further recovery of the plaintext. The attacks pre-

sented in (K. Paterson, 2005) are efﬁcient and have

been proven practical by the authors, i.e. against an

implementation of the IPSec stack in the Linux ker-

nel. Our attack is completely different, since it is

not a chipertext-only attack, the attacker goal is to

mount with minimum efforts a Denial of Service At-

tack (DoS) against the IPSec gateway.

One of the purposes of (K. Paterson, 2005) was

to help closing the gap between the currently known

cryptography theory and the current practices of stan-

dard bodies, implementers and users. Among others,

providing practical efﬁcient attacks against existing

(or in development) standards or products helps the

various communities (theoreticians and practicians)

to work together to establish cryptographically sound

but workable standards and products.

The main contribution of this paper is to present

yet another attack against the “encryption-only” con-

ﬁguration for ESP. This attack has been unearthed

years ago but hasn’t lead to a publication because at

that time it was perceived as of no theoretical inter-

est. However, following the example of (K. Paterson,

2005), we hope to contribute with the publication of

our ﬁnding to help improving the IPSec standard (and

the corresponding implementations).

Our attack is a very practical Denial of Service

(DoS) attack against the IPSec gateway providers.

It only requires a minimal effort from the attacker

while having a devastating potential impact on the

192

Nikov V. (2006).

A DOS ATTACK AGAINST THE INTEGRITY-LESS ESP (IPSEC).

In Proceedings of the International Conference on Security and Cryptography, pages 192-200

DOI: 10.5220/0002095701920200

 SciTePress

gateways performance. We think that this attack is

an even stronger and convincing argument than those

already published to support the banishment of the

“encryption-only” conﬁguration of ESP.

Recall that the ESP conﬁguration is not integrity

protected, thus the option provided in the ESP for re-

play protection makes no sense since it requires ob-

viously data integrity. We stress that our attack is

not just a reply attack since the attacker can devi-

ate/multiply the ﬂow to different targets. We also

assume, as in (C. McCubbin, 2000; K. Paterson,

2005), that the datagrams are not checked after IPSec

processing to see if the correct IPSec policies were

applied; this is the case in the Linux kernel imple-

mentation (as pointed out in (K. Paterson, 2005)).

Although it is commonly known that encryp-

tion without authenticated integrity is dangerous and

more speciﬁcally that ESP without data authentica-

tion leads to all kind of problems there are still secu-

rity IPSec specialists that claim “when IPSec gateway

is protected in addition with a stateful ﬁrewall which

does ﬁlter the packets, most of the described above at-

tacks will not work.” We think that such an approach

is just an attempt to shift the problem, instead of erad-

icate it.

The rest of this paper is organized as follows: Af-

ter an introduction to the IPSec ESP in Sect. 2, we

present the DoS attack itself in Sect. 3.

2 PRELIMINARIES

2.1 Ipsec

IPSec provides security at the level of the IP layer.

The secure encapsulation of the payload of IP data-

grams is part of the IPSec standard and is provided by

ESP (Encapsulating Security Payload, (Kent, 2005)).

ESP provides integrity, authentication and encryption

of the IP datagrams (as an option, it is possible to add

a replay protection).

The encapsulation is connectionless, i.e. it is per-

formed on a per-datagram basis. One common use of

IPSec is to build virtual private networks where IPSec

is conﬁgured to use ESP in tunnel mode. The way

ESP modiﬁes the datagrams in tunnel mode is shown

in the Appendix Fig. 4.

Fig. 4 shows that given a datagram to protect, ESP

creates a new datagram made of, in sequence, a new

IP header, an ESP header, the original datagram (IP

header and Inner Payload), an ESP trailer and an ESP

authentication payload. The detailed description of

the ESP header, trailer and authentication payload is

not provided since they are outside of the scope of our

attack.

IPSec supports different encryption algorithms:

AES (J. Daemen, 2002; NIST, 2002) is the most com-

monly used but other block ciphers as DES (NIST,

1977), 3DES (NIST, 1999), CAST128 (Adams,

1997b; Adams, 1997a), RC5 (Rivest, 1994), IDEA

(X. Lai, 1990), and Blowﬁsh (Schneier, 1993) are also

allowed. All block ciphers are used in CBC mode

(R. Pereira, 1998; S. Frankel, 2003). Additionally,

AES in CTR mode is a valid alternative (Housley,

2004) but in this case the standard states that “AES-

CTR implementations MUST employ a non-NULL

ESP authentication method, since it is trivial to forge

an AES-CTR ciphertext”. As a consequence, the here-

after described attack targets only the block ciphers in

CBC mode. The attack depends only on the block size

n of the cipher, where n =64for all block ciphers ex-

cept for AES where n = 128, 192, 256.

2.2 Cbc Mode

If the to be transmitted data after ESP padding is made

of q blocks, if the plaintext blocks are denoted by

,...,P

, and if e

(·) (d

(·)) denotes the en-

cryption (decryption) of blocks using an n-bit key K

then the CBC mode (NIST, 1980) works as follows.

After a random n-bit IV (Initialization Vector) is gen-

erated, the ciphertext blocks are computed according

to the equations:

= IV, C

= e

i−1

⊕ P

), (1 ≤ i ≤ q).

At the receiver side the plaintext is recovered accord-

ing to the equations:

= C

i−1

⊕ d

), (1 ≤ i ≤ q).

As pointed out in (A. Menezes, 1996; K. Paterson,

2005) a well known weakness of the CBC mode is

the “bit ﬂipping attack”. An attacker can ﬂip (invert)

a speciﬁc bit in the ciphertext block C

i−1

, then this

speciﬁc bit in the recovered plaintext block P

is also

ﬂipped (since P

= C

i−1

⊕ d

)). This allows

an attacker to introduce controlled changes into the

recovered plaintext block P

, but the previous block

i−1

is randomized. Hence the integrity of the IV

in CBC should be protected otherwise uncontrolled

change on the ﬁrst recovered plaintext block P

possible.

2.3 Ip, TCP, Udp and RTP

Datagrams

The hereafter presented attack depends on the way the

IP stack is structured. The presentation is limited to

IPv4 headers as speciﬁed in (Tanenbaum, 2002).

The layout of the IP header is illustrated in the Ap-

pendix Fig. 5. The potential targets of the attack are

the source and the destination IP addresses (32 bits

A DOS ATTACK AGAINST THE INTEGRITY-LESS ESP (IPSEC)

193

each) and the header checksum (16 bits). The ﬁrst 80

bits bits are disregarded and the last optional ﬁelds are

assumed to be absent. Note that, in the absence of the

optional ﬁelds, there is no padding.

There are two commonly used protocols in the

transport layer, a connectionless one - the User

Datagram Protocol (UDP) (Postel, 1980) and a

connection-oriented one - the Transmission Control

Protocol (TCP) (Postel, 1981). Thus an IP data-

gram may contain either UDP or TCP embedded data-

grams. This gives additional targets for the attack.

The TCP header (Postel, 1981) is illustrated in the

Appendix Fig. 6. The source and the destination ports

(16 bits each), the sequence and acknowledgement

number (32 bits each), which together form the ﬁrst

96 bits of the TCP header as well as the checksum

located between 128-th and 144-th bits are additional

important targets of the attack.

The UDP header (Postel, 1980) is illustrated in

Fig. 1. In this case the source and destination ports

as well as the checksum (16 bits each) are additional

targets of the attack.

Destination PortSource Port

Data...

ChecksumLength

3376B\3376F2AM

Figure 1: UDP Segment Format.

Sometimes, another protocol is encapsulated in the

UDP datagrams, e.g. the Real-Time Transport Proto-

col (RTP) (Schulzrinne et al., 2003). A description of

the RTP header is given in the Appendix Fig. 7, giving

yet another target for the attack: the sequence number

ﬁeld located between 16-th and 32-th bits.

This list of protocols and targets is not exhaustive,

the attack is applicable to other protocols. The ratio-

nale behind the choice of those targets is that these are

the ﬁelds that are consulted or manipulated by the IP

gateways in an IP network.

2.4 Internet Checksum

The (internet) checksum (R. Braden, 1988) is used to

discover errors while datagrams are transmitted. The

targets chosen in this paper to illustrate the attack:

the IP header, the TCP and UDP datagrams, all use

checksums to protect their content. The purpose of

the internet checksum is to provide an efﬁcient pro-

tection against transmission errors but not to provide

cryptographic integrity protection of the content.

The outline of the internet checksum is very simple:

the data is presented as a sequence of 16 bit integers,

then the 1’s complement sum of these integers is com-

puted and the checksum is the 1’s complement of this

sum.

The way this checksum is computed allows the fast

incremental update of the checksum when the data

over which it is computed is changed. For example, to

update the checksum C in the IP header m to the new

IP header m



, but without recomputing it over the en-

tire new IP header, the updated checksum C



can be

easily computed as C



= C +(m − m



), as shown

in (T. Mallory, 1990). Note that, to compute the new

checksum, one needs to know m − m



, but neither m,



nor the old checksum C are to be known.

3 A DOS ATTACK

3.1 Fields Manipulation

The choice made by some implementers or users for

the “encryption-only” ESP conﬁguration (i.e. with-

out the “costly” authenticated integrity) is based on

the false belief that conﬁdentiality protection together

with the structure of IP, TCP (or IP, UDP, RTP) data-

grams is enough to detect any data change (malicious

or not). This amounts at making the hypothesis that

any modiﬁcation to the encrypted datagrams will be

detected during the parsing of the decrypted data-

grams (i.e. they are supposedly malformed) or during

the veriﬁcation of the checksums.

This is not true; an attacker can, in fact very eas-

ily, modify the encrypted (in CBC mode) datagrams

in such a way that they are still acceptable for the em-

bedded protocols such as IP, TCP, UDP, RTP. In the

attack presented here, the IV is changed in such a

way that the ﬁrst decrypted block of plaintext P



changed in a predictable way (see Sect. 2.2). More

precisely, the IV is replaced with a new one which

differs only in the positions to be altered in the ﬁrst

block of plaintext.

Consider a conﬁdentiality protection with a block

cipher with length n = 128 (see Fig. 2). In this

case, P

, the ﬁrst block of data, contains the IP header

checksum and the source IP address (see Sect. 2.3).

Now an attacker can change the source IP address to

whatever she/he wants, then using the difference be-

tween the old and the new source IP addresses she/he

can compute the new correct checksum (see Sect. 2.4)

and ﬁnally she/he can compute the new IV corre-

sponding to the altered ESP datagram (the rest of the

data remains the same). When a gateway receives

such a datagram, it accepts it as a valid datagram (i.e.

the IP header is valid) and forwards it further to the

corresponding destination address.

SECRYPT 2006 - INTERNATIONAL CONFERENCE ON SECURITY AND CRYPTOGRAPHY

194

0 80 96 128 160

Checksum

Source IP

Address

Figure 2: n=128.

In the case where n = 192 (see the Appendix

Fig. 8), the attacker has more choices. Assuming

that TCP/IP (or UDP/IP) datagrams are sent, the at-

tacker can also change the destination IP address,

the source and destination ports. But since the TCP

(UDP) checksum is in block P

, the modiﬁcations

brought by the attacker must be such that the TCP

(UDP) checksum does not change, i.e. it is still a valid

TCP (UDP) datagram.

In the case where n = 256, the attacker has even

more choices (see the Appendix Fig. 9). Assuming

that TCP/IP datagrams are sent, the attacker can also

change the source and destination ports from the TCP

datagram as well as the sequence number and the ac-

knowledgement number but again she/he should do

that in such a way that the TCP checksum is not

changed since it is still in block P

. If RTP/UDP/IP

datagrams are sent, the attacker can change the source

and destination ports from the UDP datagram with-

out being restricted anymore to keep unchanged their

UDP checksum, since the checksum is in P

. In ad-

dition the sequence number in the RTP datagram can

also be changed.

The attack cannot be applied when n =64because

the ﬁrst 64 bits of the IP header do not contain any

interesting target ﬁelds (see Fig. 3).

08096160

Checksum

Figure 3: n=64.

3.2 Mounting the Attack

Assume a source device communicates with a desti-

nation device through IPSec gateways A and B (see

the Appendix Fig. 10), using “encryption-only” IPSec

(ESP). Consider the n = 128 case. As shown in

Sect. 3.1, an attacker can intercept a legitimate ESP

datagram, modify the source address, such that the al-

tered ESP datagram and the embedded IP header are

still valid and re-inject the new ESP datagram in the

network. Gateway B will accept this modiﬁed ESP

datagram, extract the IP header and embedded pay-

load (see the Appendix Fig. 4) and forward them to

the destination device. Notice that, for any legiti-

mate ESP datagram, the attacker can generate up to

false source addresses and hence so many false

ESP datagrams, which will have the same content as

the legitimate one but will claim to come from dif-

ferent sources. All of them will be accepted by the

gateway B and forwarded to the destination device. In

this way, the gateway as well as the destination device

are overloaded with junk datagrams. The attacker can

mount this attack on the ﬂy in real-time.

If we assume (as in (K. Paterson, 2005)) that the at-

tacker knows several source IP addresses of machines

which legitimately communicate through the IPSec

gateway, then even stateful ﬁrewall could not defeat

the attack. Moreover it is enough that the attacker

knows a signiﬁcant portion of this IP addresses, which

might be more realistic since it may be that the host

is on a network with network preﬁx known to the at-

tacker, or because it is widely used address preﬁx.

Consider now the n = 192 and n = 256 cases.

In addition to the source address, the attacker can also

alter the destination address along with the source and

destination ports. The rest of the attack is similar to

the n = 128 case, except that the junk datagrams can

be targeted to any destination address. So the attacker

can mount a DoS attack on any selected server in the

Internet.

4 CONCLUSIONS

This paper presents a very strong and practical DoS

attack against the “encryption-only” conﬁguration for

ESP (IPSec). It also shows that, contrary to the in-

tuition, increasing the cipher block size jeopardizes

the security of the protocol even more, i.e. the attack

becomes stronger. We sincerely hope that this attack

will serve as a strong argument to improve the exist-

ing standard by banning the “encryption-only” con-

ﬁguration and will be a compelling example for the

implementors/users of the standard convincing them

not to use such a conﬁguration.

ACKNOWLEDGEMENTS

This work has been prepared in conjunction with the

design and development of the IPSec communication

A DOS ATTACK AGAINST THE INTEGRITY-LESS ESP (IPSEC)

195

suite for the Philips Semiconductors Nexperia Mobile

platform. The author would like to thank Marijke De

Soete, Xavier Serret, Marc Vauclair and Bruno Key-

molen for the valuable discussions.

REFERENCES

A. Menezes, P. van Oorschot, S. V. (1996). Handbook of

applied cryptography. In CRC Press.

Adams, C. (1997a). Constructing symmetric ciphers using

the CAST design procedure. In Designs, Codes, and

Cryptography 12(3) pp. 283–316.

Adams, C. (1997b). The CAST-128 encryption algorithm.

In RFC 2144.

Bellovin, S. (1996). Problem areas for the IP security pro-

tocols. In Usenix Unix Security Symposium, pp. 1–16.

C. McCubbin, A. Selcuk, D. S. (2000). Initialization vector

attacks on the IPSec protocol suite. In WETICE’00,

IEEE Computer Society, pp. 171-175.

Canvel, B., Hiltgen, A., Vaudenay, S., and Vuagnoux, M.

(2003). Password interception in a SSL/TLS channel.

In CRYPTO’03, LNCS 2729, pp. 583–599.

Housley, R. (Jan. 2004). Using advanced encryption stan-

dard (AES) counter mode with IPSec encapsulating

security payload (esp). In RFC 3686.

J. Daemen, V. R. (2002). The design of rijndael: AES - the

advanced encryption standard. In LNCS.

K. Paterson, A. (2005). Cryptography in theory and prac-

tice: The case of encryption in IPSec. In Cryptol-

ogy ePrint Archive, 2005/416, to appear at EURO-

CRYPT’06.

Kent, S. (March 2005). IP encapsulating security payload

(ESP). In Internet-Draft.

M. Bellare, T. Kohno, C. N. (2004). Breaking and provably

repairing the SSH authenticated encryption scheme:

A case study of the encode-then-encrypt-and-MAC

paradigm. In ACM Transactions on Information and

System Security (TISSEC) 7(2), pp. 206–241.

N. Borisov, I. Goldberg, D. W. (2001). Intercepting mobile

communications: The insecurity of 802.11. In MOBI-

COM’01, pp. 180–189.

NIST (1977). Encryption standard (DES). In FIPS PUB 46.

NIST (1980). DES modes of operation. In FIPS PUB 81.

NIST (1999). Triple-DES (3DES). In FIPS PUB 46-3.

NIST (2002). Advanced encryption standard (AES). In

FIPS PUB 197.

Postel, J. (Aug. 1980). User datagram protocol. In RFC

768.

Postel, J. (Sept. 1981). Transmission control protocol. In

RFC 793.

R. Braden, D. Borman, C. P. (Sept. 1988). Computing the

internet checksum. In RFC 1071.

R. Pereira, R. A. (Nov. 1998). The ESP CBC-mode cipher

algorithms. In RFC 2451.

Rivest, R. (1994). The RC5 encryption algorithm. In

FSE’94, pp. 86–96.

S. Frankel, R. Glenn, S. K. (Sept. 2003). The AES-CBC

cipher algorithm and its use with IPSec. In RFC 3602.

Schneier, B. (1993). Description of a new variable-length

key, 64-bit block cipher (blowﬁsh). In FSE’93,

pp. 191–204.

Schulzrinne, H., Casner, S., Frederick, R., and Jacobson, V.

(July 2003). RTP: A transport protocol for real-time

applications. In RFC 3550.

T. Mallory, A. K. (Jan. 1990). Incremental updating of the

internet checksum. In RFC 1141.

T. Yu, S. Hartman, K. R. (2004). The perils of unauthenti-

cated encryption: Kerberos version 4. In NDSS’04.

Tanenbaum, A. (2002). Computer networks. In Prentice

Hall.

X. Lai, J. M. (1990). A proposal for a new block encryption

standard. In EUROCRYPT’90, pp 389-404.

SECRYPT 2006 - INTERNATIONAL CONFERENCE ON SECURITY AND CRYPTOGRAPHY

196

APPENDIX

IP Hdr

Payload

Original IP datagram

Payload

ESP

Trl

ESP

Auth

Encrypted

Authenticated

Datagram with ESP

in tunnel mode

IP Hdr

Payload

New

IP Hdr

New

IP Hdr

Tunneled datagram

ESP

Hdr

Figure 4: ESP in Tunnel Mode.

VERS

1 1 2 3

0 4 8 6 9 4 1

HLEN

Service

Type

Total Length

FLG

Fragment

Offset

TTL Protocol

Header

Checksum

Source IP Address

Destination IP Address

IP Options Padding

Data ...

...

Figure 5: Format of an IP Datagram Header.

A DOS ATTACK AGAINST THE INTEGRITY-LESS ESP (IPSEC)

197

Destination PortSource Port

Sequence Number

Acknowledgment Number

Data

Offset

Reset

Window

Urgent PointerChecksum

Options

Data Bytes

..|.. Padding

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

0123

Figure 6: TCP Segment Format.

time stamp

0 8 16 31

sequence number

CSRC

count

payload

type

synchronization source (SSRC) identifier

contributing source (CSRC) identifiers

Figure 7: RTP Header Format.

0 80 96 128

160

176 192

288 304

160 176 192 208 224

320

Checksum

Source IP

Address

Source Port

Destination Port

Source Port

Destination Port

TCP

UDP

Figure 8: n=192.

SECRYPT 2006 - INTERNATIONAL CONFERENCE ON SECURITY AND CRYPTOGRAPHY

198

0 80 96 128

160

176 192 224 256

288 304

160 176 192 208 224 240 256 352

320

Checksum

Source IP

Address

Source Port

Destination Port

Source Port

Destination Port

TCP

UDP

RTP

Figure 9: n=256.

Source Destination

IPSec

Gateway A

IPSec

Gateway B

Attacker

Figure 10: IPSec.

A DOS ATTACK AGAINST THE INTEGRITY-LESS ESP (IPSEC)

199