A Collaborative Analysis System for Cross-organization Cyber Incident
Handling
Giuseppe Settanni, Florian Skopik, Yegor Shovgenya and Roman Fiedler
Digital Safety and Security Department, AIT Austrian Institute of Technology, Donau-City-Strasse 1, 1220, Vienna, Austria
Keywords:
Cyber Incident Handling, Security Operation Center, Situational Awareness.
Abstract:
Information and Communication Technology (ICT) systems are predominant in today’s energy, finance, trans-
portation and telecommunications infrastructures. Protecting such Critical Infrastructures (CIs) against mod-
ern cyber threats and respond to sophisticated attacks is becoming as complex as essential. A synergistic
and coordinated effort between multiple organizations is required in order to tackle this kind of threats. Inci-
dents occurring in interconnected CIs can be effectively handled only if a cooperation plan between different
stakeholders is in place. Organizations need to cooperatively exchange security-relevant information in or-
der to obtain a broader knowledge on the current cyber situation of their infrastructures and timely react if
necessary. National cyber Security Operation Centers (SOCs), as proposed by the European NIS directive,
are being established worldwide to achieve this goal. CI providers are asked to report to the national SOCs
about security issues revealed in their networks. National SOCs correlate all the gathered data, analyze it and
eventually provide support and mitigation strategies to the affiliated organizations. Although most of these
tasks can be automated, human involvement is still necessary to enable SOCs to adequately take decisions on
occurring incidents and quickly implement counteractions. In this paper we therefore introduce and evalu-
ate a semi-automated analysis engine for cyber incident handling. The proposed approach, named CAESAIR
(Collaborative Analysis Engine for Situational Awareness and Incident Response), aims at supporting SOC
operators in collecting significant security-relevant data from various sources, investigating on reported in-
cidents, correlating them and providing a possible interpretation of the security issues affecting concerned
infrastructures.
1 INTRODUCTION
Advanced multi-stage cyber campaigns are continu-
ously being put in place aiming at disrupting orga-
nizations’ business continuity, exfiltrating sensitive
data, or disturbing critical services. Such elaborated
attacks intrude and leverage multiple systems and net-
works at the same time, therefore localized detection
and isolated reaction are no longer effective. A more
comprehensive and distributed approach is therefore
required (Tankard, 2011).
In fact, recently issued European directives such
as the NIS directive (European Commission, 2013),
explicitly demand for the establishment of collabora-
tion networks among the Member States, in order to
allow secure sharing of relevant incident information
and gain cyber situational awareness both on national
and international level. CI providers are requested
to report cyber security issues occurring on their net-
works to cyber SOCs. These centers are intended to
collect reports, gather intelligence and provide cyber
security support to the affiliated organizations while
preserving their privacy.
Our work is being carried out in the context of
a European funded research project named ECOS-
SIAN
1
(European Control System Security Incident
Analysis Network) that aims at improving the de-
tection and management of highly sophisticated cy-
ber security incidents and attacks against CIs by im-
plementing a pan-European early warning and situa-
tional awareness framework with command and con-
trol facilities (Kaufmann et al., 2014).
In a previous work (Settanni et al., 2015) we pre-
sented and shortly described a blueprint for our cyber
incident analysis system. The system’s objective is
twofold: i) to gain cyber situational awareness of na-
tional CIs by analyzing security intelligence obtained
from multiple sources, ii) to timely and securely dis-
tribute early warnings and advisories in the case of
1
www.ecossian.eu
Settanni, G., Skopik, F., Shovgenya, Y. and Fiedler, R.
A Collaborative Analysis System for Cross-organization Cyber Incident Handling.
DOI: 10.5220/0005688301050116
In Proceedings of the 2nd International Conference on Information Systems Security and Privacy (ICISSP 2016), pages 105-116
ISBN: 978-989-758-167-0
Copyright
c
2016 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
105
detected threats. Here we narrow down the focus
on the acquisition, aggregation and analysis compo-
nents outlined in that work by introducing a semi-
automated system for incident-data analysis. The pro-
posed system gathers security intelligence from mul-
tiple trusted sources, combines and correlates rele-
vant information with reported cyber incidents, and
derives possible conclusions on the occurring security
issues.
An assisted-learning function allows the system to
automatically determine similarities between reported
issues and every other significant resource contained
in the knowledge base in order to ease the analysis
phase. Also, this learning process takes into account
and adapts itself to operator’s feedback. Operators
can train the system by accepting or denying every
automated association (or derived conclusion), scor-
ing their usefulness, and providing comments about
them.
We provide an illustrative scenario that points out
the need for a collaborative analysis system for inci-
dent handling. The main functional requirements for
our approach are derived from this scenario. We il-
lustrate the designed system components highlighting
their functionality; we give an overview on the cur-
rent status of our system prototype implementation,
and we evaluate our approach. We finally discuss how
security intelligence is handled by the introduced sys-
tem by demonstrating how it would address the chal-
lenges outlined in the presented scenario.
This paper is structured as follows. In Section II
we present the ECOSSIAN ecosystem. In Section III
a realistic scenario is described. In Section IV we
collect the main challenges and requirements derived
from the use-case. In Section V we introduce the for-
mal model and the architectural components of our in-
cident analysis system. Section VI reports the imple-
mentation details of the main system components. In
Section VII we evaluate the performance of our sys-
tem, while in Section VIII we highlight how our ap-
proach can be employed to address the issues raised in
the use-case Section, by providing collaborative cyber
incident handling. In Section IX we review the state
of the art of distributed analysis systems for cyber in-
cident handling. We conclude the paper in Section X
with outlook and future work.
2 THE ECOSSIAN ECOSYSTEM
In order to discover and properly handle modern so-
phisticated cyber threats, organizations need to coop-
erate and exchange cyber security information so to
gain knowledge of the current situation of their in-
O-SOC
1A
O-SOC
2A
O-SOC
3A
N-SOC
A
O-SOC
1B
O-SOC
2B
O-SOC
3B
N-SOC
B
O-SOC
1C
O-SOC
2C
O-SOC
3C
N-SOC
C
E-SOC
Figure 1: ECOSSIAN ecosystem.
frastructures (ENISA, 2013a).
Most organizations deploy in-house security pro-
cedures and tools aiming at detecting threats local-
ized within their infrastructures. However, there is a
significant amount of cyber threats impossible to be
revealed without further information gathered from
sources external to the enterprise network. Moreover,
security information is most of the time also sensi-
tive information, therefore the sharing infrastructure
needs to employ mechanisms to protect the shared
data against leakage, misuse and loss of reputation.
Organizations tend to cooperate more between each
other if a trusted third party guarantees for privacy in
the communication (ENISA, 2013b).
To bridge this gap, SOCs such as (German Federal
Office for Information Security, 2014), (National Cy-
ber Security Center - Ministry of Security and Justice
- Netherlands, 2014) and (Government of Canada -
Cyber Security, 2014), are recently being established.
SOCs collect security-relevant information generated
by a multitude of systems, correlate it and try to com-
prehend whether the monitored systems are encoun-
tering anomalies, or are under attack. Once a threat
is identified, a SOC also provides technical support
to mitigate the effects it might cause, and proposes
strategies for solving the issue.
In the ECOSSIAN project, we propose a Pan-
European three-layered approach (Kaufmann et al.,
2014) to protect CIs by detecting cyber incidents and
timely generating and distributing early warnings to
the monitored infrastructures. As depicted in Fig-
ure 1 we foresee three types of SOCs: Organization
SOC (O-SOC), National SOC (N-SOC), and Euro-
pean SOC (E-SOC). At O-SOC level organizations
deploy sensors and tools for threat detection, and re-
port to N-SOCs about incidents that might have cross-
organizational relevance. There are several differ-
ent types of information O-SOCs share with their
respective N-SOC. Data generated by sensors at O-
SOC level can be automatically forwarded to the N-
SOC acquisition module; security relevant informa-
ICISSP 2016 - 2nd International Conference on Information Systems Security and Privacy
106
tion (such as threats, vulnerabilities, observations,
etc.) obtained by analyzing locally detected anoma-
lies, can instead be manually reported by O-SOC op-
erators. N-SOCs are deployed by European Member
States joining the ECOSSIAN network, they are re-
sponsible for gaining cyber situational awareness on
the network on national CIs. Here cyber intelligence
is acquired by analyzing information gathered from
different data sources such as reporting O-SOCs, fed-
erated N-SOCs, publicly available sources, etc. Cyber
incident information aggregation, correlation, clus-
tering and analysis are the main functionalities pro-
vided at this level. Once the evaluation of the analy-
sis results is concluded, mitigation steps, advisories,
or early warnings are sent back to the reporting O-
SOCs. At the highest level an E-SOC performs anal-
ysis of strategic information shared by the different
N-SOCs and distributes advisories to targeted lower
level SOCs.
3 ILLUSTRATIVE SCENARIO
In this section we describe a fictional but realistic sce-
nario of advanced cyber attacks in today’s setting. It
further demonstrates the need for SOCs and coopera-
tion.
WonderLight is the main energy provider in Coun-
tryX, one of the European Member States. The corpo-
rate is structured in five different departments located
in three different sites on the national territory. Three
power plants of different size produce electricity dis-
tributed to more than 40% of the population.
Each chief of department receives a well-crafted
email, apparently sent by the CEO asking the recipi-
ent to click on a link to get access to the last quarterly
financial report. Two recipients (Dr. A and Dr. B)
click on the link and are directed to an empty web
page. Dr. A and Dr. B. close the web browser ig-
noring the effects of their clicks and keep carrying
out their activities. Mrs. C, who received the mes-
sage as well, suspects about the trustworthiness of the
email, but being uncertain of what to do, she deletes
the email from the mail client without clicking on the
link. The remaining two victims (Mrs. D and Dr.
E) are more familiar with cyber-security issues and
since they are aware that the financial report would
not be ready before three weeks, they immediately re-
alize the email they just received is malicious. They
report the security event to the company’s IT depart-
ment in charge, among other tasks, of investigating
cyber-security issues.
The IT department examines the reported issue,
but due to lack of resources (personnel, time, tools)
for a deep inspection, and noticing that the web page
is not reachable anymore when clicking on the re-
ceived link, they simply alert all the company em-
ployees about the occurred phishing attempts and in-
struct them not to click on any link received in similar
emails.
Moreover, Dr. A and Dr. B, by clicking on
the link, inadvertently downloaded through their web
browser a sophisticated malware undetectable by the
antivirus solutions installed on their machines. The
malware exploits a Windows vulnerability, executes
a daemon performing port scanning and communi-
cating with the command and control server through
an encrypted connection. The malicious software de-
tects a list of open ports and hence allows the at-
tackers to gain remote privileged access to the in-
fected computers. The attackers remotely connect
to these machines and thanks to their privileges are
able to browse through the corporate network, explore
privacy-sensitive documents, and get access to the In-
dustrial Control Systems (ICSs) managing the energy
production at the power plants.
The attackers are skilled and motivated activists
aiming at boycotting and damaging WonderLight’s
and other European energy providers’ reputation be-
cause of their questionable environmental policy. Af-
ter three weeks of undetected intrusion, the attackers
have gained a good knowledge of WonderLight’s in-
frastructure. They start maneuvering numerous Pro-
grammable Logic Controllers (PLCs) employed in the
ICSs at the main power plant, and gradually decrease
the energy production until shutting down entire por-
tions of the electrical network. This causes hours of
blackouts in several regions of the country until the
business continuity plan of WonderLight is activated,
the intrusion is detected and the normal energy pro-
duction is restored. Such an event implies a consider-
able loss of revenue and reputation for WonderLight.
If similar APTs target at the same time every other
energy provider in the country, the effects of such dis-
tributed attack can be dramatic (Reichl et al., 2013).
Although modern European States rely on CIs
employing state-of-the art intrusion detection sys-
tems (IDs) and off-the-shelf industrial control sys-
tems, they are not able to prevent the described attack
scenario because they lack of dedicated organization
and especially national cyber SOCs, the CIs do not
cooperate with one another, and they rarely examine
cyber threat intelligence generated by third parties.
Starting from the presented scenario we derive
in the following section the main functional require-
ments for the analysis system, deployed at the N-SOC
in the ECOSSIAN ecosystem, to tackle the outlined
security challenges.
A Collaborative Analysis System for Cross-organization Cyber Incident Handling
107
4 FUNCTIONAL
REQUIREMENTS
Diverse aspects are to be considered when designing
an incident analysis system for a national SOC in the
ECOSSIAN ecosystem.
In order to have a comprehensive picture of the
cyber security situation of the CIs in a country, it is
necessary for the N-SOC to collect relevant informa-
tion from multiple sources.
Publicly available information should be collected
and taken into account to obtain a broad knowledge on
the most recent security events. Information such as
latest available updates and patches, discovered vul-
nerabilities, released security bulletins should be con-
sidered while handling cyber incidents on a national
level.
At organization’s level state-of-the-art intrusion,
threat and anomaly detection systems are required to
be in place aiming at revealing malfunctions, attacks
and deviations from the normal systems’ behavior.
The outcomes generated by such tools (alarms, alerts,
logs, etc.) are analyzed and processed by O-SOC op-
erators and if necessary forwarded to the N-SOC. In
ECOSSIAN we foresee two types of information to
be transmitted from O-SOCs to N-SOC. Manual re-
ports can be filled out by O-SOC operators describ-
ing observed security incident and then sent to the N-
SOC through a secure and privacy-preserving sharing
network; moreover data automatically generated by
sensors
2
deployed in the infrastructure (including IDS
alerts, firewall logs, SIEM alarms, and ECOSSIAN
sensor readings) can automatically be collected by the
N-SOC acquisition module (Settanni et al., 2015).
In order to regulate the information sharing pro-
cess between O-SOCs and the respective N-SOC, re-
porting policies must be in place. Source and destina-
tion entities, formats, timing, frequency, content and
every other detail of the sharing processes has to be
specified within these policies. Only adhering to this
commonly agreed set of rules different organizations
can cooperate effectively in a trusted manner.
Another crucial aspect to take into account is pri-
vacy. Organizations will be prone to cooperate with
one another and share security data only if confiden-
tiality is guaranteed. Data protection is fundamental
for the design of a secure incident sharing network
2
We assume here that EOCSSIAN sensors are deployed
at each organization’s infrastructure and are configured to
generate and periodically transmit (after approval) inter-
preted security data representing the health status of ser-
vices running at the specific infrastructure they monitor.
This data does not need to be further interpreted by the N-
SOC.
and the implementation of an efficient incident anal-
ysis system. Access control, anonymization and en-
cryption functionalities must be therefore enabled.
The process of cyber incident handling can mas-
sively be supported by automated procedures and
software tools for aggregating, classifying and ana-
lyzing frequently generated and high volume incident
data. However, not all the analysis tasks can be car-
ried out without human involvement; complex issues
and manually reported incidents, received less fre-
quently than sensor readings, need to be thoroughly
examined by N-SOC security experts.
The cooperative analysis system we propose in
this paper has the purpose of facilitating the acqui-
sition of national cyber situational awareness, and as-
sisting the decision making process by bringing sig-
nificant intelligence to the attention of N-SOC opera-
tors. To achieve this goal sophisticated visualization
tools demonstrating the current security situation of
the monitored national CIs are required.
5 COOPERATIVE ANALYSIS:
THE LINKING MODEL
In this section we introduce CAESAIR: a Cooperative
Analysis Engine for Situational Awareness & Incident
Response. First a theoretical description of the model
is given, then details on the architectural components
and their functionalities are provided.
5.1 Information Entities
In the following we define the main information en-
tities comprising the data that CAESAIR collects and
generates while processing reported security issues.
We define as Resource any relevant document col-
lected and stored in the system, such as an incident
report from an O-SOC, a security advisory, a forum
post or an email message. Resources are not changed
over the course of their processing at N-SOC. Both
operators and the analysis engine can attribute a Re-
source to clusters and classes of Resources. The set
of existing classes is defined by the N-SOC person-
nel, while the clusters are discovered by the analysis
system during the knowledge base evaluation.
An Artifact identifies a certain concept and an un-
limited number of its text representations (phrases or
regular expressions). The representations are used by
the system to detect the concept in free text, e.g. the
terms “Windows 7” and “ms-win7” identify the same
Artifact MS Windows 7. Two Artifacts may build mul-
tiple one-sided “is-a” relations, such as Linux distri-
ICISSP 2016 - 2nd International Conference on Information Systems Security and Privacy
108
bution to Operating system, or Fedora and Debian to
Linux distribution.
The frequency of Artifacts’ occurrences is one of
the basic metrics used for estimating similarity be-
tween Resources. The more concepts are reflected
as Artifacts, the more information is available about
each Resource.
A Tag is a text label that may be attached to a
Resource or Artifact, showing their connection to a
certain concept that is not explicitly mentioned in
them. For example, reports describing a highly tar-
geted spear phishing attack may be tagged by a N-
SOC operator as “suspected APT” and “social engi-
neering”, although the terms APT or social engineer-
ing do not occur in them. Tags also help the operators
to group Resources and Artifacts in an intuitive and
flexible way.
5.2 Information Importing
Resources are added to CAESAIR either manually by
the O/N/E-SOC personnel or automatically via pre-
configured import interfaces, e.g. web crawlers or re-
mote database APIs.
When a new document is acquired by the analysis
system its text is first indexed to the search engine;
then a Resource object is created in the system, refer-
encing the search index entry. The Resource’s text is
therefore scanned for occurrences of Artifacts known
to the system, or possible new Artifacts. Based on de-
tected Artifacts and the original text itself, the system
attempts to attribute the Resource to one of the known
classes and clusters. Finally, the new Resource is for-
warded for evaluation by an operator, who can con-
firm or reject the system’s suggestions.
Artifacts may be created by the system accord-
ing to predefined rules, or manually by the O/N/E-
SOC operator. All stored Resources will be regularly
scanned for occurrences of newly added Artifacts; if
an Artifact is deleted, records of its occurrences are
also removed.
Furthermore, the system may create new Artifacts
if it discovers a phrase matching a certain rule defined
by the operator, such as “two words beginning with
capital letters”. This behavior differs from detecting
representations of a single Artifact based on a regular
expression: in that case the text match was marked
as an occurrence of the existing artifact, and here we
create a new Artifact and set its first representation
string equal to one that matched our rule.
5.3 Resource Linking Model Definition
CAESAIR’s analysis process aims at identifying all
R1: Vulnerability Report
A2: Linux
A7: Adobe Flash Player
A12: Sandbox escape exploit
A13: Chromium
R2: Vulnerability Confirmation
A3: Windows
A7: Adobe Flash Player
A14: Internet Explorer
R3: Security Advisory
A2: Linux
A3: Windows
A7: Adobe Flash Player
A10: CVE reference
A11: Exploit
R4: Security Advisory
A2: Linux
A12: Sandbox escape exploit
L
1,2
= score(A7)
R1,R2
L
1,3
= score(A2)
R1,R3
+ score(A7)
R1,R3
+ 0,5*score(A11)
R1,R3
L
2,3
=score(A3)
R2,R3
+ score(A7)
R2,R3
L
3,4
=score(A2)
R3,R4
L
1,4
= score(A2)
R1,R4
+ score(A12)
R1,R4
A1: Operating
System
A2: Linux A3: Windows
A6: Software
A7: Adobe
Flash Player
A8: Web
Browser
A10: CVE
reference
A11: Exploit
A12: Sandbox
escape exploit
A13: Chromium
A14: Internet
Explorer
A15:
Vulnerability
Figure 2: Top: Example of a Resource Linking Diagram.
Bottom: Example of an Artifact Relation Diagram.
existing Resources linked to the Resource under ex-
amination or to a given text, deriving possible corre-
lations between Resources, and revealing patterns in
distribution of Resources over time and locations.
The resource linking process is based on interrela-
tions between Resources, which represent documents
containing text, and Artifacts, which represent con-
cepts and their text representations.
If a Resource’s text contains at least one represen-
tation of a certain Artifact, this Artifact is considered
related to the Resource. Each such relation is further
referred to as Occurrence of an Artifact in a Resource.
A Resource may or may not have any arbitrary num-
ber of Occurrences of any Artifacts, as long as each
Occurrence has at least one unique representation in
the Resource’s text (it may be a phrase, word or even
a single character). Figure 2 depicts an example of
Artifacts included in different Resources (upper part
of the figure), and how Artifacts can be related to one-
another (lower part of the figure).
Given a certain Resource r
1
, containing the arti-
facts a
1
, a
2
, a
3
and a
4
, the algorithm calculates its
linkage to the sample Resource r
0
as follows.
Let R be the set of all Resources, and R
0m
the set
of Resources having at least one Artifact in common
with r
0
. Let us assume R
0m
= {r
2
, r
3
, r
4
, r
5
}. Let A
0,1
be a set of Artifacts present both in r
0
and r
1
. Let us
assume it contains the Artifacts a
1
, a
2
and a
3
: A
0,1
=
{a
1
, a
2
, a
3
}
Now for each artifact a
i
in A
0,1
, we define the rat-
ing score as:
score
a
i
r
01
= f
s
(T FIDF(a
i
, r
1
, R
1m
),
T FIDF(a
i
, r
0
, R
0m
), f req(a
i
, R))
(1)
where:
A Collaborative Analysis System for Cross-organization Cyber Incident Handling
109
T FIDF(a, r, Rm) function determining the Term
Frequency-Inverse Document
Frequency weight of an Artifact
a in the resource r considering
the set of resources havinig
mutual artifacts R
m
f req(a, R) function returning the sum of
boolean frequencies of the Ar-
tifact a in the set R of all Re-
sources
f
s
customizable scoring function
We then determine the linkage between r
1
and r
0
as
the sum of scores for each Artifact in A
0,1
:
link
r
01
=
N
i=0
score
a
i
r
01
+ f b (2)
where N is the number of Artifacts that occur in A
0,1
,
and f b is an optional value representing the operator’s
feedback on the goodness of the link (its value can be
grater or smaller than zero).
Figure 2 also shows how four Resources are linked
to one-another. The thickness of the lines connecting
the Resources indicates how significant the Resources
are to one-another and is proportional to the calcu-
lated link.
5.4 Knowledge Base Querying
The knowledge base contains the full and up-to-date
information available at the N-SOC; search and filter-
ing functions make this information available to the
N-SOC personnel and authorized parties.
Two ways of searching are provided:
1. By text: to find Resources containing the speci-
fied text. In this case the results are rated by a
customizable combination of:
number of matches in a single Resource;
number of matches in Resource’s Tags or re-
lated Artifacts;
proximity between the query and found
matches.
2. By Resource: to find Resources that are linked to
the specified sample Resource. In this case the
search results (Resources) are rated by:
text match criteria mentioned in the previous
paragraph;
number of Artifacts that are present in both the
search match and the sample;
cross-combinations of matching Artifacts in
other Resources: a Resource containing match-
ing Artifacts that do not appear in other Re-
sources may be rated higher;
number of Artifacts that do not directly match
with those from the sample, but have a common
parent node in the is-a hierarchy.
Let us assume there are Artifacts represent-
ing some operating systems named MS Windows,
FreeBSD and Debian, whereby FreeBSD and Debian
have an “is-a” relation to an Artifact representing
UNIX-like operating systems. All of the artifacts also
have an “is-a” relation with the Artifact “Operating
system”. Now, if the Resource given as search sam-
ple contains the Artifact “Debian”, then a Resource
containing “MS Windows” will be rated lower than a
Resource having “FreeBSD”. A Resource that has no
common Artifacts with the sample will, as expected,
not receive any rating for this, but can still be rated
higher than others if it better satisfies the text match
criteria.
Filtering allows, instead, to select groups of Re-
sources that exactly satisfy given binary criteria such
as presence of specific Artifacts or words in the Re-
source’s text, or being part of a specific class or clus-
ter.
5.5 Statistical Analysis
The population of Resources in CAESAIR’s knowl-
edge base comprises all information the N-SOC has
about the current security situation. To keep track
of this situation, CAESAIR regularly evaluates the
knowledge base. First it computes statistics describ-
ing a predefined set of Resources’ properties, such as
frequency of Windows software vulnerability reports,
top 100 incident reports sorted by damage level, as
submitted by the victim, etc. Moreover it determines
Resource clusters (based on quantitative properties of
Resources), and then it looks for patterns or anomalies
in relations between the Resources. For instance, an
unprecedented increase in attack reports submitted by
energy sector companies, which the N-SOC person-
nel noticed anyway, can appear correlated with phish-
ing campaigns recently reported by another federated
N-SOC.
6 CAESAIR: SYSTEM
COMPONENTS AND
IMPLEMENTATION
In this section we describe the CAESAIR system
components and provide details on their current soft-
ware implementation.
ICISSP 2016 - 2nd International Conference on Information Systems Security and Privacy
110
LEGEND
Information Flow direction
Bidirectional Information Flow
Provides Interface
Expects Interface
N-SOC
User
Dashboard
O-SOC
Intelligence
Importer
Admin
Dashboard
Analysis engine
Evaluator
Original
Resource
Storage
Metadata
Storage
Search
Index
Input
Processor
OSINT
Importer
N-SOC
GUI
N-SOC
Admin GUI
OSINT
sources access
Figure 3: CAESAIR system components.
6.1 System Components
As shown in Figure 3, CAESAIR consists of the fol-
lowing components:
Importers for both open intelligence (OSINT) and
the data generated within ECOSSIAN;
Original resource (document) storage;
Search index;
Analysis engine, comprising:
Incoming data processor;
Metadata storage;
Evaluator;
Dashboards for the N-SOC personnel.
CAESAIR operates based on incoming Re-
sources, the basic units of information for the sys-
tem. The importers of intelligence data acquire new
documents actively (e.g. by crawling given web re-
sources, databases) or passively from open sources
or O-SOCs through a dedicated interface provided by
importers, either graphical or non-graphical.
Importers validate, sanitize each document’s con-
tents and then forward it to the original resource
storage, where it will be kept with read-only access
for future reference.
A copy of each Resource is saved in the search in-
dex, from where it can be retrieved by Resource ID, or
via full-text search over all properties of a Resource.
The input processor collects Resources from the
original storage and checks them for occurrences of
known Artifacts, or for possible new Artifacts. All
detected occurrences and new Artifacts are saved to
the Metadata storage, whereas the new Resources
are forwarded to the search index.
Metadata storage holds all the data produced
within CAESAIR, such as: Artifacts known to the
system, relations between them and Resources, attri-
bution of Resources to clusters and classes, comments
that N-SOC operators add to Artifacts or Resources.
Evaluator is the core component that helps draw-
ing conclusions from the accumulated data: it peri-
odically, in configurable intervals, queries both the
Metadata storage and the search index to keep track of
interdependencies between Artifacts and Resources.
Using the resource linking model discussed in Section
V, it also estimates similarities between Resources,
attributes them to clusters and suggests classification
for them. On request from an N-SOC user the Eval-
uator also answers the question: ”what Resources
known to the N-SOC are similar to the given one?”.
By running the Linking Model (see previous section)
the Evaluator identifies and prioritizes the Resources
with highest pertinence to the Resource currently an-
alyzed.
The N-SOC dashboard is the primary way for N-
SOC personnel to use the system. It provides a graph-
ical and a programming interface for querying the
analysis system and giving feedback on the queries,
searching for Resources and monitoring patterns in
the accumulated data.
When an N-SOC operator is handling an incom-
ing incident report, the graphical interface will use the
Evaluator’s search and filtering API to display the list
of Resources that seem to be relevant to the current
A Collaborative Analysis System for Cross-organization Cyber Incident Handling
111
context. The operator may narrow down the results
set by setting additional filtering criteria (e.g. only
display the Resources added in March 2015 and from
two specific sources).
Finally, the administrator dashboard allows
manually tuning the system’s parameters and directly
accessing the Metadata storage. It is supposed to be
used for maintenance rather than normal work-flow.
6.2 Implementation
CAESAIR’s storage components (Metadata and Orig-
inal Resource Storage) are based on PostgreSQL
3
database servers. For the search index we employ
Elasticsearch
4
, an open-source search engine that is
optimized for frequently updated dataset and offers a
flexible API for querying and indexing data.
The logic components (Importers, Input Proces-
sor, Evaluator and Dashboards) run by the N-SOC,
are developed as Python modules.
O-SOC intelligence importer will function as a
service available to O-SOCs via an online ticketing
system (Request Tracker
5
) hosted at the N-SOC. The
service will also provide an API, allowing O-SOCs
to interact with CAESAIR using custom software.
The open source intelligence importer incorporates
Python modules for retrieving and parsing data in var-
ious formats from different sources (such as a web-
site, database, file system). It runs on-demand or per-
manently, pushing new data to the Analysis engine
as they appear in the targeted source. The Input Pro-
cessor is triggered by the Open Intelligence Importer
whenever new documents are available; it also ac-
tively checks the Original Resource Storage for up-
dates.
The Evaluator runs periodically to recalculate the
statistics (information on clusters, classes, patterns
detected in Resources and Artifacts) and save it to the
Metadata Storage.
The N-SOC user and administrator dashboards,
implemented as web applications with Python, serve
Javascript-powered client GUI and querying the
Metadata storage, the Search index and the Evalua-
tor. They also validate and forward user commands
and feedback to the Evaluator, and update the Meta-
data storage as needed.
Figure 4 depicts a screen-shot of the main view of
the N-SOC user dashboard. The left panel lists the
Resources to be analyzed. On the right panel a set
of Resources, linked to the one currently selected, are
displayed.
3
http://www.postgresql.org/
4
https://www.elastic.co/
5
https://www.bestpractical.com/rt/
7 SYSTEM EVALUATION
In order to assess the performance and the soundness
of the implemented system we defined relevant quan-
titative and qualitative metrics and we ran a number
of tests to extract measures under different functional
conditions. The evaluation setup, the defined perfor-
mance metrics, the obtained measures and the derived
conclusions are reported in this section.
7.1 Evaluation Setup
All the tests have been executed on a machine with In-
tel(R) Xeon(R) CPU E5-1620 v2 at 3.70GHz 8 cores
and 16GB memory, running Ubuntu 12.04.5 LTS op-
erating system.
To our best knowledge no publicly available
repository of incident reports exists, hence to mea-
sure the performance of our system we imported in
the Resource knowledge base elements from Common
Vulnerabilities and Exposures (CVE)
6
database with
entries registered between 2013 and 2015, including
16 thousand elements.
For the sake of simplicity in this work we lim-
ited the Artifacts extraction to the identification of
only product and platform names within the Re-
sources. We therefore imported the complete Com-
mon Platform Enumeration (CPE)
7
repository (count-
ing around 100 thousand entries), and generated an ar-
tifact (with its different representation) for each entry
of this archive.
7.2 Quantitative Evaluation
To quantitatively evaluate our system we observed its
scalability, in terms of resources required to perform
its operations, considering a variable number of exist-
ing resources and artifacts in the knowledge base. In
close cooperation with the ECOSSIAN partners we
specifically took care about a realistic setup. We fo-
cused on two main timing metrics:
t
Aext
: average time required to extract Artifacts
(CPE entries) relevant to a Resource r
i
t
Link
: average time to link a Resource r
i
to any
other relevant Resource r
j
in the set R
im
8
Two Resources having at least a common Artifact
are linked according to the method described in Sec-
tion 5.3. The scoring function f
s
adopted in our tests
6
https://cve.mitre.org/
7
https://cpe.mitre.org/
8
R
im
is the set of Resources having at least one Artifact
in common with r
i
ICISSP 2016 - 2nd International Conference on Information Systems Security and Privacy
112
Figure 4: Screenshot of one view of CAESAIR’s dashboard.
is the sum. We did not consider any operator’s feed-
back not to bias the evaluation of the automated link-
ing. Hence given an Artifact a
x
present in 2 resources
r
i
and r
j
:
score
a
x
r
i j
= T FIDF(a
x
, r
i
, R
im
)
+T FIDF(a
x
, r
j
, R
jm
)
+ f req(a
x
, R)
(3)
and
link
r
i j
=
N
x=0
score
a
x
r
i j
(4)
Where N is the number of Artifacts present both
in r
i
and in r
j
.
During our tests we imported into the system,
through the CAESAIR importing component, a vari-
able number of randomly selected CVE entries. We
also let the number of imported CPE entries variate,
so to determine how the number of existing Resources
and Artifacts influences the system’s performance.
Figure 5 shows how the average Artifacts extrac-
tion time t
Aext
changes when considering a variable
number of Artifacts (CPEs), and a variable number
of Resources (CVEs). Analyzing this graph we can
observe that the Artifacts extraction function scales
with the number of Artifacts present in the knowledge
base. In fact even considering 100 thousand CPEs the
average time required to extract all the Artifacts from
a Resource is around 10 milliseconds. We can notice
this is true for every number of Resources we consid-
ered.
Figure 5: Average Artifacts extraction time vs number of
CVE Resources vs number of Artifacts (CPE).
Figure 6: Average Linking time vs number of CVE Re-
sources vs number of Artifacts (CPE).
Figure 6 shows how the average Resource linking
time t
Link
changes when considering a variable num-
ber of Resources (CVEs), and a variable number of
Artifacts (CPEs). Observing this graph we can con-
clude that, as predicted, the linking method requires
longer time when the number of Resources to analyze
A Collaborative Analysis System for Cross-organization Cyber Incident Handling
113
grows. Moreover the average Resource linking time
grows with the number of existing Artifacts. This is
due to the fact that the more Artifacts and Resources
are present in the knowledge base, the more queries
the system has to execute to obtain the Resources re-
lated to the one under analysis. Looking at graph in
the case of 10 thousand Resources and 100 thousand
Artifacts, we can observe how, given a Resource, the
linking function needs around 3.5 seconds to obtain
all the Resources in the knowledge base having at
least one mutual Artifact. This is a relatively short
time considering that it corresponds to the time the
operator will need to wait after clicking on a Resource
to obtain all the related Resources. We foresee that
this time could get smaller if we would adopt a less
stringent linking method not based on perfect match-
ing of all the Artifacts Resources contain, but on a
fuzzy search of only the most relevant Artifacts of the
Resource.
7.3 Qualitative Evaluation
In order to perform a qualitative assessment of the de-
signed Resource linking method we adopted the Del-
phi method (Sackman, 1974). A group of 3 experts
with cyber security background and periodically in-
volved in cyber incident handling activities, have been
selected and asked to judge the results of our Re-
source linking method. Statistics on this manual as-
sessment allow a preliminary qualitative evaluation of
our approach.
We let the system extract Artifacts (from the 100
thousands imported CPEs) and generate links for a
set of 10 thousand Resources (CVEs). We consid-
ered a set of 25 different randomly selected linked
Resources (i.e., Resources with at least 1 link) and
assigned them to each expert. The experts graded the
relevance of all the links the system created for each
analyzed Resource, with a score S between 0 (not rel-
evant at all) and 5 (very relevant). The average
¯
S and
the median
˜
S of the scores obtained by each expert are
reported in Table1. The table reports small discrepan-
cies between the evaluation results of the 3 experts.
Indeed the average scores variate in a range between
2.8 and 3.28. Moreover, given the experts’ evalu-
Table 1: Average results of 3 experts’ qualitative evalua-
tion. Analysis based on the links obtained on 25 randomly
selected Resources.
¯
S
˜
S
Expert 1 2.96 3
Expert 2 3.28 4
Expert 3 2.8 3
Average 3.01 3.33
ation, we can derive from a preliminary and rough
analysis that the linking method provides quite accu-
rate matches between existing related Resources. The
qualitative assessment proposed here will be further
validated in the future by the design and the adoption
of an analytical evaluation approach.
8 ILLUSTRATIVE APPLICATION
In this section we validate the presented approach. We
focus on the advantages derived from its deployment
in the ECOSSIAN ecosystem to tackle the security
issues outlined in the scenario in Section 3.
Considering the scenario, if WonderLight (the tar-
geted company) would be part of ECOSSIAN and
it would therefore have a highly qualified O-SOC in
place (required to join the ECOSSIAN ecosystem),
the O-SOC operators would receive notification by
the employees recognizing the suspicious mails, and
using the ECOSSIAN platform they would report a
security incident to the N-SOC including a copy of
the suspicious email.
Let us assume the attack was performed on inter-
national scale, and targeted many European small and
medium energy providers employing a particular se-
ries of PLCs produced by a specific vendor. In this
case the CountryX’s N-SOC receives security issues
not only from the WonderLight’s O-SOC, but also
from other affected energy organizations in Coun-
tryX. The N-SOC uses CAESAIR to optimally prior-
itize and analyze the acquired reports, and correlating
them it derives that a common threat is targeting lo-
cal energy providers. The N-SOC operators generate
hence a security alert and distributes it to all O-SOCs
belonging to the energy sector, asking to those which
didn’t report any issue yet, to inform the N-SOC if
they encountered any similar situation.
Some hours later, the CountryY’s N-SOC learns,
through open source intelligence feeds, about a zero-
day exploit that takes advantages of a newly discov-
ered shell vulnerability and gains privileged remote
access to a system. This N-SOC informs the E-SOC
which opportunely forwards this security information
to all the other N-SOCs.
In order to check if the suspicious emails were
connected to the newly discovered zero-day exploit,
the CountryX’s N-SOC asks the energy O-SOCs to
configure the ECOSSIAN sensors to provide informa-
tion about IP addresses and ports currently active in
their infrastructure. The N-SOC obtains automated
data from the ECOSSIAN sensors, and employing
CAESAIR is able to discover whether the attacker
gained already remote access to the different organi-
ICISSP 2016 - 2nd International Conference on Information Systems Security and Privacy
114
zations’ infrastructure. Affected organizations would
in fact have same suspicious ports and IP addresses in
use.
Our system provides a user-friendly graphical in-
terface that allows the operator to quickly browse
through relevant resources, conveniently ranked by
the analysis engine, and investigate on reported in-
cidents by querying the knowledge base. N-SOC op-
erators are therefore able to easily assess the cyber
security situation of the CountryX by analyzing the
derived intelligence, and will inform the affected or-
ganizations (by distributing tailored advisories to the
respective O-SOCs) on how to mitigate the detected
threat.
Indicators of Compromise (IoCs) will be gener-
ated by the CountryX’s N-SOC and they will be sent
(within an incident report informing about the de-
tected cyber threat) to the E-SOC. The E-SOC will
possibly receive multiple reports related to the same
issue from different affected N-SOCs. After acknowl-
edging the incident the E-SOC will distribute the re-
spective IoCs and inform interested N-SOCs about
it. Receiving N-SOCs will investigate through the O-
SOCs if any organization in their country was affected
by the security issue, and will give their feedback to
the E-SOC. The information collected at E-SOC will
be analysed and will allow to obtain cyber situational
awareness at European level.
This use-case demonstrates how the ECOSSIAN
ecosystem would efficiently address one of the pos-
sible security issues targeting multiple organizations
located in different European countries. By coopera-
tively employing CAESAIR’s outcomes the N-SOCs
operators are able to quickly assess the severity of dis-
tributed threats, contrive possible mitigation steps, al-
low operators to timely react to the security incident
and restrain its impact on their infrastructures.
9 RELATED WORK
Cyber incident handling is becoming a crucial task for
SOCs responsible for the protection of modern CIs.
Increasing amounts of data needs to be properly col-
lected, processed and analyzed to obtain knowledge
on the current security situation and derive proper
countermeasures in case of threats.
Regulatory bodies have recently started issuing
directives (European Commission, 2013), frame-
works (European Commission, 2009), and guidelines
(ENISA, 2013c), in order to raise the awareness on
cyber threats among CI organizations, regulate on the
establishment of required countermeasures, and de-
fine roles and responsibilities of the entities involved
in handling cyber incidents on national and interna-
tional level.
Moreover, several solutions have been proposed
which aim at efficiently process and share threat in-
formation.
In (IBM, 2013) IBM presents X-Force, a system
for automated threat analysis based on dynamic Inter-
net data to gain insight and context in security inci-
dents.
FIDO, an open source system for automatically
analyzing security events and responding to security
incidents is introduced by NETFLIX in (NETFLIX,
2015). FIDO is an orchestration layer that automates
the incident response process by evaluating, assessing
and responding to malware and other detected threats.
A more sophisticated solution is proposed by Air-
bus Defence & Space in (AIRBUS, 2014); Cymerius
is a decision-making engine which operates within a
SOC and permits real-time management of IT sys-
tem security by taking into consideration the organi-
zational aspects of the monitored infrastructure.
ENISA, in cooperation with a group of four Eu-
ropean CERTs, is currently working on an Incident
Handling Automation Project (IHAP) (ENISA, 2015)
which aims at improving the incident handling pro-
cess by increasing automation and providing an easy-
to-set-up and deploy solution for incident response
process. In IHAP CERT teams use a unified Data
Harmonization Ontology to enhance the actionable
reporting and analysis of the collected abuse informa-
tion.
Since 2010 an open community is developing
CRITS (Collaborative Research Into Threats)
9
, an
open source malware and threat repository which
combines an analytic engine with a cyber threat
database. CRITS serves as a repository for attack data
and malware, and provides analysts with a platform
for conducting malware analysis and correlation of hi-
erarchically structured cyber threat information
10
.
All the aforementioned approaches and tools are
based on automated processing of incident data and
do not foresee any human involvement while handling
security incidents. The system we presented in this
paper differs by massively supporting SOCs’ analy-
sis tasks with automated information aggregation and
correlation, but at the same time involves security an-
alysts in the decision making phase. Since CAESAIR
is an assisted-learning system, it also allows analysts
to provide feedback on the soundness of the conclu-
sions derived, and therefore train the engine to dy-
namically adapt to evolving security issues.
9
https://crits.github.io/
10
Open source structured data exchange formats such as
STIX, CybOX and TAXII are used within CRITS.
A Collaborative Analysis System for Cross-organization Cyber Incident Handling
115
10 CONCLUSION AND FUTURE
WORK
In this paper we introduced CAESAIR, a model for
the analysis of cross-organizational cyber incidents
that aims at supporting National Security Operation
Centers in gaining cyber situational awareness and re-
sponding to detected security issues.
Thanks to its self-learning search engine, CAE-
SAIR improves its incident querying capabilities and
hence allows N-SOCs to achieve high incident han-
dling rates.
The system performance evaluation proved the
feasibility and the scalability of our approach by
demonstrating how the main functional tasks pro-
vided by CAESAIR can be implemented and exe-
cuted on real data at a high rate. We eventually
provided a preliminary qualitative assessment of the
CAESAIR Resource linking function adopting the
Delphi method.
Future work include the investigation of advanced
Resource linking procedures (e.g., based on Bayesian
and neural networks), the development and integra-
tion of functionalities for allowing assisted learning
through operator’s feedback, code optimization in or-
der to guarantee improved scalability, and the design
as well as the implementation of an analytical evalu-
ation method for the assessment of the different fore-
seen Resource linking procedures.
ACKNOWLEDGEMENTS
This work was partly funded by the European Union
through the FP7 project ECOSSIAN (607577) and the
Austrian FFG research program KIRAS in course of
the project CIIS (840842)
REFERENCES
AIRBUS (2014). CYMERIUS - Security Manage-
ment Tool. http://www.defenceandsecurity-
airbusds.com/web/guest/1299.
ENISA (2013a). Cybersecurity cooperation: Defending the
digital frontline.
ENISA (2013b). Detect, share, protect. Technical report,
EU Agency for Network and Information Security.
ENISA (2013c). Technical Guideline on Incident Report-
ing.
ENISA (2015). Incident Handling Automation Project.
European Commission (2009). Article 13a of the Regula-
tory Framework for Electronic Communications in the
European Union.
European Commission (2013). Commission proposal for a
directive concerning measures to ensure a high com-
mon level of network and information security across
the union.
German Federal Office for Information Security
(2014). German it crisis management webpage.
https://www.bsi.bund.de/EN/Topics/IT-Crisis-
Management/itcrisismanagement node.html.
Government of Canada - Cyber Security (2014). Cana-
dian cyber incidents response center homepage.
http://www.publicsafety.gc.ca/cnt/ntnl-scrt/cbr-
scrt/ccirc-ccric-eng.aspx.
IBM (2013). Combat the latest security attacks with global
threat intelligence.
Kaufmann, H., Hutter, R., Skopik, F., and Mantere, M.
(2014). A structural design for a pan-european early
warning system for critical infrastructures”. In Elek-
trotechnik und Informationstechnik. Springer.
National Cyber Security Center - Ministry of Security and
Justice - Netherlands (2014). Dutch cyber security
center homepage. https://www.ncsc.nl/english/.
NETFLIX (2015). Introducing FIDO: Automated Security
Incident Response.
Reichl, J., Schmidthaler, M., and Schneider, F. (2013). The
value of supply security: The costs of power outages...
Energy Economics, 36:256–261.
Sackman, H. (1974). Delphi assessment: Expert opinion,
forecasting and group process. An Experiment in
Probabilistic Forecasting.
Settanni, G., Skopik, F., Fiedler, R., and Shovgenya, Y.
(2015). A blueprint for a pan-european cyber incident
analysis system”. In Proceedings of 3rd International
Symposium for ICS and SCADA Cyber Security Re-
search, pages 84–88.
Tankard, C. (2011). Advanced persistent threats and
how to monitor and deter them. Network Security,
2011(8):16–19.
ICISSP 2016 - 2nd International Conference on Information Systems Security and Privacy
116