System, Structures and Processes in Evaluation of

Civil Protection Exercises

Harri Ruoslahti

and Harriet Lonka

ResLab, Laurea University of Applied Sciences, Vanha maantie 9, Espoo, Finland

North Savo Regional Council, Sepänkatu 1, 70100 Kuopio, Finland

Keywords: System, Structures, Processes, Civil Protection Exercises, Exercise Evaluation.

Abstract: Civil protection exercises are activities that simulate real-life emergencies, where participants can practice,

review, and test the exercise system, and its structures and processes. The Union Civil Protection Mechanism

(UCPM) exercises all require systematic evaluation. This article applies the framework of systems theory to

the practical evaluation of civil protection exercises with the research question (RQ): how does systems theory

apply to the evaluation practices of civil protection exercises? The method of this study combines a descriptive

literature review, Delphi workshops, and analysis of expert interview transcripts on the approaches behind

the concepts which are then examined in the context of the evaluation of civil protection exercises.

Organizational structures of exercise systems outline 'what' direct the activities that achieve the goals of the

exercise. Structures are the combinations of relations between the organizational elements that form

organizational activities, and may include rules, roles, methods, technologies, applications, and

responsibilities of the exercise participants. Organizational processes of an exercise include the activities that

establish the goals of the exercise. In an exercise evaluation context, processes focus on how operations and

human interactions are carried out by the people who realize and manage the scenarios of the exercise. This

three-dimensional approach can help address the complex interplay of factors within civil protection exercises.

The contribution of this study is that it clarifies the theoretical background of literature on systems theory,

organizational structures, and processes as they relate to the evaluation of civil protection exercises, which

can have a practical contribution to the training of exercise evaluators.

1 INTRODUCTION

Civil protection exercises simulate real-life

emergencies, so that participants can practice, review,

and test the exercise system, and its structures and

processes, in defined roles (Lausen & Kastner, 2022).

Civil protection exercises are part of the development

of the field civil protection, and for this reason these

exercises require systematic evaluation (Bruns &

Kern, 2022; Bruns, Çelikler & Jonitz, 2022).

Exercises aim at improving preparedness on an

individual level, where exercises provide opportunities

for hands-on practice, on disaster plans and

procedures, while offering constructive criticism; and

on an institutional system level, where a well-designed

exercise may reveal resource and interagency

coordination gaps, clarify roles and responsibilities,

and uncover weaknesses in planning (Beerens, 2021).

https://orcid.org/0000-0001-9726-7956

The Union Civil Protection Mechanism (UCPM)

classifies exercises as discussion based, such as

seminars, workshops, tabletops, and games, and

operations based, which can be e.g., drills, functional

exercises, and full-scale exercises (European

Commission, 2021). Exercises range from relatively

simple exercises that may e.g., involve a small team

to practice a relatively simple drill, to very complex

exercises, where a wide range of organizations

simulate a major emergency (WHO 2017; Lausen &

Kastner, 2022).

Systematically interconnecting evaluation

methods, tools and available evaluators can

contribute to a continuous improvement of the

outcomes of European civil protection exercises

(Heinonen et al. 2024); Understanding these benefits

from a holistic view where civil protection exercises

form a European system that include national and

Ruoslahti, H. and Lonka, H.

System, Structures and Processes in Evaluation of Civil Protection Exercises.

DOI: 10.5220/0013518500004000

Paper published under CC license (CC BY-NC-ND 4.0)

In Proceedings of the 17th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2025) - Volume 2: KEOD and KMIS, pages

315-322

ISBN: 978-989-758-769-6; ISSN: 2184-3228

315

regional systems of systems (Heath, 1998). Exercise

evaluation can include concepts and methodology

coupled with very practical standard operating

procedures (SOP) to help develop more standardized

content and structure for the evaluation of different

exercise formats (Bruns & Kern, 2022).

Evaluation should include a concept, strategies of

data collection, means of analysis and reporting, and

generalizations of their appropriateness to evaluation

areas or problems (Alkin, 1970). Bruns et al. (2022)

propose that three exercise evaluation concepts,

system, structures, and processes can enable to

identify the most suitable evaluation methods with

SOP as a useful guideline for exercise evaluators. The

chosen evaluation focus must help understand the

evaluation process and systematically assist

addressing relevant factors within system, structures,

and processes (Lausen & Kastner, 2022; Bruns et al.,

2022).

This article aims to understand how to apply the

framework of systems theory to the practical

evaluation of civil protection exercises. The research

question (RQ) of this paper is: how does systems

theory apply to the evaluation practices of civil

protection exercises?

The main contribution of this study is that it

deepens the theoretical framework behind the

evaluation foci of system, structures, and processes

proposed by first Heath. (1998) and then Bruns et al.

(2022) and provides practical notions to conduct

evaluation processes in a systematic way and based

on evaluation concepts thinking. The second section

explains the methodology of this study; the third

section how the three concepts from systems

approach have been discussed in academic literature;

the fourth section focuses on the context of civil

protection exercise evaluation; and finally, the last

section draws conclusions and proposes future steps

and studies.

2 METHOD

The method of this study firstly compiles a

descriptive literature review to understand the

theories and approaches behind the concepts of

system, structures, and processes. Secondly, these

concepts are examined as foci in the context of the

evaluation of civil protection exercises.

The database ProQuest Central was used for the

search, which used combinations of the search words:

systems theory; systems theory and evaluation;

organiz(s)ational structures; and organiz(s)ational

processes. Identified articles were first investigated

by reading their title and abstract, after which those

that best explain these concepts in an evaluation or

organizational context were selected and read in full.

The second part of this study has been

complemented by discussing the results of the

literature review with practical iterations in expert

discussions held at meetings and workshops that were

completed in October and November of 2022. The

nature of these meetings is action research (Denzin &

Lincoln, 1994), where a mini-Delphi approach

provided collective debate, individual estimations,

and aggregation of answers (Gallego & Bueno, 2014)

on how systems theory can be applied to the

evaluation of civil protection exercises. These aim of

these meetings and workshops were a systematic path

to develop systematic evaluation of civil protection

exercises (Table 1).

Table 1: Data collection for this study.

Event Date Venue n =

Literature

review

on theory

Oct –

Dec

2022

Search words:

Systems theory

and evaluation

Organiz(s)ational

structures

Organiz(s)ational

rocesses

scientific

articles

Practical

discussions

and

iterations

Nov –

Dec

2022

Teams and Face-

to-face

Mini-Delphi

discussions

based on

literature review

(Grant and

Booth, 2009

)

Ten (10)

events of

twelve

(12)

experts:

Process 2,

Eval. 4,

Method 3,

Trainin

Practical

discussions

and

iterations

Dec

2022

Teams or

Face-to-face Apr

– Jun

2022 under

INEGMA-E

Ten (10)

civil

protection

exercise /

evaluation

experts

Update

literature

review

Jul

2025

Search words:

Evaluation of

civil protection

exercises

four

additional

scientific

articles

The data collection of this study involved a

literature review on theory focusing on the systems

theory, evaluation, and organizational structures and

processes. This phase was conducted by the authors

between October and December of 2022, and it

provided a final sample of 24 scientific articles. The

findings of this literature review were then further

discussed and deepened in a series of ten mini-Delphi

discussions (Grant and Booth, 2009), which included

KMIS 2025 - 17th International Conference on Knowledge Management and Information Systems

316

twelve (12) experts in ten meetings, held between

November and December of 2022, and workshops

involving twelve experts in four fields of expertise:

two (2) of processes, four (4) on evaluation, three (3)

on methods, and two (2) on training (Table 1).

In December of 2022 by transcripts of ten expert

interviews, which had been conducted between April

and June of 2022 with interviewees who have years

of experience in evaluating a variety of civil

protection exercises were examined. In July 2025 an

additional literature search was added (Table 1).

Qualitative data analysis is based on classification of

objects, and is subjective, and cannot be generalized

in the population (Denzin & Lincoln, 1994), as this

research is strongly tied into the context of the

evaluation of European civil protection exercises.

3 SYSTEMS APPROACH IN THE

CONTEXT OF CIVIL

PROTECTION EXERCISES

Literature discusses that evaluation is a social

process, which implies a need for transparent

participatory approaches (Gregory, 2000). One expert

interviewee noted that “the idea of a structured

evaluation with specific objectives is well received

and should be used in future exercises” (INEGMA-E

/ Expert 09, 2022).

3.1 Civil Protection Exercises as

Systems

General systems theory emphasizes that systems are

organized and composed of interdependent

components (Kast & Rosenzweig, 1972). Open

systems are embedded within wider systems, where

e.g. individuals perform within e.g. an exercise

ecosystem (Midgley, 2006; Kast & Rosenzweig,

1972). Any civil protection exercise can be seen as a

system formed by parts/sub-systems such as the

exercise participants, and its stakeholders. General

systems theory can be seen to provide a model of

certain general aspects of reality and as a way of

seeing things which otherwise may become easily

overlooked or bypassed (Von Bertalanffy, 1972).

Some systems information is only partly clear

(Rahimnia, Moghadasian & Mashreghi, 2011), which

can be the case in many exercise systems. Evaluators

can take from Systems Theory the idea that it is

possible to evaluate how well an organization

performs as an open system seeking to thrive in a

turbulent environment (Midgley, 2006).

The Input-Transformation-Output model suggests

that systems receive inputs their environments, that

they transform in some way to export outputs (Kast

& Rosenzweig, 1972; Vos & Schoemaker, 2004).

Feedback, which is information about processes or

outputs of the system, is fed back into the system as

input (Kast & Rosenzweig, 1972). Data quality

ensures that processed data is accurate, timely, and

consistent to make reasonable decisions (Orr, 1998)

“… because human thoughts and behaviours are part

of wider systems of causality, evaluators need to be

very careful about engaging in ‘blame games’”

(Midgley, 2006, p. 17).

Workflow processes and Standard operating

procedures (SOP) have been widely applied in

organizational processes (Hevner & Chatterjee,

2010). Each exercise project forms its own system

within which the exercise participants and

stakeholders act. A systems approach can help

conceptualize exercises, as systems with interrelated

parts that are open to influences from outside the

system (Delphi discussions, 2022; Grunig, Grunig

and Ehling (1992). Exercises may be seen to operate

as open systems (INEGMA-E

, 2022; Crossan, Lane

and White, 1999). Communication spans the

boundaries between system parts, sub-systems, and

the environment of the system (Vos 2017).

Delphi discussion experts saw that a systems

approach may be a useful lens to assess the various

levels of civil protection exercises. Organizational

processes, wider innovation eco-systems and their

actors build a facilitating context for learning

(Hautamäki 2010, Oksanen & Hautamäki 2014).

Organizational learning involves recognizing the

larger systems that an organization or exercise builds

trusting relationships to create commitment among

the stakeholders to support organizational learning

(Senge et al., 2008). Systems can be complex,

networked (Mitleton-Kelly 2003) and dynamic

(Ahrweiler & Keane 2013), and combined with actor

networks (Piperca & Floricel 2012, Steins & Edwards

1999), while an exercise may also fail to achieve its

objectives due to the environmental conditions

(Bruns et al., 2022).

Evaluation, when grounded in real scenarios,

“features the added value of validating theoretical

models and assessing the effectiveness of

technological applications in optimizing emergency

response systems” (Mandirola et al., 2024, p. 15).

Exercise evaluation aims to capture the influencing

factors of the system with multiple evaluation

methods (Heinonen et al., 2024).

System, Structures and Processes in Evaluation of Civil Protection Exercises

317

3.2 Structures Within Civil Protection

Exercises

Organizations consist of elements, relations between

elements, and structure becomes a combination of the

relations between the organizational elements

forming organizational activity (Ahmady, Mehrpour

& Nikooravesh, 2016). Designing temporary project

organizations can be challenging (Shirazi, Langford

& Rowlinson, 1996), as structure can be understood

as the relations between the components of an

organized whole, e.g. relations between jobs,

systems, operating processes, people (Ahmady,

Mehrpour & Nikooravesh, 2016).

Temporary multiorganization seem to follow the

principles of systems theory and distinguish between

managing systems and operating systems (Shirazi et.

al., 1996). Structures in an exercise setting can refer

to the models of the internal relations by which power

relations, reporting, formal communication channels,

responsibilities and decision-making delegation

become clarified (Ahmady, Mehrpour &

Nikooravesh, 2016). Structures can directly influence

organizational learning (Martínez-León & Martínez-

García, 2011), and strategic decision-making

processes (Fredrickson, 1986).

As part of systems, structures have hierarchy as

organizational units have relative ranks, which can be

visualized in an organizational chart, also they have a

functional dimension showing the distance of each

person in organization to central core of organization

(Ahmady, Mehrpour & Nikooravesh, 2016).

Structures may be physical or social, such as

buildings or geographical or relations between

people, positions, units, and sectors (Ahmady,

Mehrpour & Nikooravesh, 2016), while in crisis

situations, structure may become subservient to

turbulent environmental variables (Fredrickson,

1986).

Exercise evaluation may measure and analyse

resolution strategies and possible alternative

measures with resources available to achieve

measurable change within the exercise scenario

(Döring & Bortz, 2026).

Structures lay out who does what, so the exercise

organization can meet its objectives. Structures refer

to the devices and mechanisms by which the exercise

system is operated and managed. Structures outline

how selected activities are directed, what everyone’s

job is, and how it fits within the overall system, as

structures may e.g. clearly define chains of command,

or leave individual actors with high levels of personal

agency.

Exercise structures may be illustrated as diagrams

or organizational charts. “When defining the

objectives, the evaluators should be part of the

planning team, to guarantee good evaluation results”

(INEGMA-E

/ Expert 02, 2022).

European civil protection exercises include

international and national operational structures that

are activated by the Union Civil Protection

Mechanism (UCPM) (Call for Proposals UCPM-

2021-EX, p. 8). These instruments and techniques

may be used operationally in specific cases such as

response capacities and modules (Call for Proposals

UCPM-2021-EX, p. 8). Exercises may provide a

testing ground to implement new tools and

procedures, and effectiveness becomes a focus when

evaluating structures. External influences may be

addressed through systems (Bruns et al., 2022).

3.3 Processes Within Civil Protection

Exercises

Observed organizational processes and structures

help influence and improve managerial behaviours in

practice (Ozbas, 2005). Techniques for representing

processes include flow charts and data-flow diagrams

(Malone et al., 1993). Delegation within hierarchies

helps restrain internal competition and improve the

quality of strategic communication (Ozbas, 2005).

Organizational implications could be made by

collecting appropriately rich datasets of internal

procedures, organizational structure, personnel

usage, etc. (Ozbas, 2005). Process evaluation should

become anchored in practice for performance

measurement and evaluation, and to frame

organizational processes as social practices (Kelly &

Cordeiro, 2020). Strategic communication, which is

vital to allocating resources, may decrease in quality

with increasing levels of integration (Ozbas, 2005).

Recently internalized tacit knowledge can best

contribute to improving the capability to manage

alliances when socialization and externalization

practices make this knowledge is more widely

available (Feller et al., 2013).

“Attention to the collection and meaningful use of

performance information may support improved

student outcomes by shaping goal-setting activities,

providing feedback on performance, and making

targeted changes to organizational processes” (Sun &

Henderson, 2017, p. 1). One expert interviewee

commented that: “follow the structure and its

successful” and that it is “important how evaluation

is integrated in the whole exercise” (INEGMA-E

Expert 01, 2022).

KMIS 2025 - 17th International Conference on Knowledge Management and Information Systems

318

A challenge with measurement is that metrics

across stakeholders may not be comparable to their

relationship to performance (Eccles, Ioannou &

Serafeim, 2014). It may be helpful to involve

historical and institutional knowledge to make

suppositions about the nature of some aspects of

change, rather than attempting to measure it directly

(Carroll, Dobrev & Swaminathan, 2002), as human

agency or societal processes can be difficult to

measure (Gasc-Barbier, 2024).

Processes refer to 'how' structures are managed by

humans, and they include all activities that establish

the goals of the exercise. In an exercise evaluation

context, processes focus on how operations and

human interactions are carried out by the people who

realize and manage the scenarios of the exercise.

When structures consist of the 'what' is used to do

something in the exercise system, processes describe

'how' these structures become managed by the

participants. Analysis of the processes can be made in

the actual implementations of the measure, such as

implementation, problems, and quality perceived by

target groups and the exercise staff (Döring & Bortz,

2016). Exercise evaluation should check how

exercise objectives are achieved, as “clear training

objectives lead to a very clear way of working and

thus a very easy evaluation process” (INEGMA-E

Expert 03, 2022).

The exercise organization is a whole that consists

of unified parts that act in harmony to effectively and

efficiently execute the tasks needed to achieve the

goals of the exercise. Processes focus on the ways in

which operations and human interactions are carried

out by people to realize and manage the exercise

scenario (Russo & Rindone, 2024).

4 HOLISTIC CIVIL

PROTECTION EXERCISE

EVALUATION

The analysis of the data of expert interviews confirms

the approach developed based on the literature

review. Projects gather teams of diverse expertise to

achieve common goals (Canonico et al. 2013),

demonstrating a need to actively manage

communication in all these three phases of knowledge

development with the context of civil protection

exercises (Table 2).

Table 2: Evaluation focus definitions and some examples.

Evaluation

Focus

Issues Examples

System Exercise can be

seen as a system

formed

by parts/sub-

systems

such as the

exercise

participants,

and its

stakeholders.

Mostly designed

by organizers

and exercise

members

Project

management

Exercise

preparation

Scenarios

On-site security

Cultural

influencing

factors

Expert 09:

structured

evaluation with

specific should

be used in

uture exercises

Structures Organizational

structures

of exercise

systems outline

how activities

are directed

to achieve the

goals of the

exercise.

Who does what

so exercise

meets objectives

Participants’

jobs and how

they fit within

overall exercise

system.

Chain of

command

Rules

Roles

Responsibilities

Expert 02:

When defining

the objectives,

the evaluators

should be part

of the planning

team

Expert 01:

it’s important

how evaluation

is integrated in

the whole

exercise

Processes Activities that

establish

goals

Organization

— a whole

consisting

of unified parts

acting in

harmony to

execute tasks to

achieve

goals, both

effectively

and efficiently

Search and

rescue

operations

Firefighting

Deployment of

medical

personnel

Communication

Expert 03: clear

training

objectives

lead to

a very easy

evaluation

process

This data (Table 2) was used to enrich the results

with empirical evidence. System, structures, and

System, Structures and Processes in Evaluation of Civil Protection Exercises

319

processes can be used as a basis to build a framework

that maps the process of evaluating civil protection

exercises holistically in the project flow and focuses

on the points to be considered for an evaluation

concept. The three evaluation concepts, system,

structures, and processes influence each other, “…

there is a tight coupling of systems and processes, and

there are many interdependencies between these

systems and processes” (Vos 2017, p. 23).

There are complex interactions between people,

technologies, and processes, while more and more

modern systems used by civil protection

organizations can be considered cyber-physical

(Linkov et al. 2013) or socio-technical (Amir & Kant

2018) as they combine human activities with

information technologies.

Processes turn inputs through throughput or

transformation into outputs (Katz & Kahn 1978).

Inputs may be resources, and outputs learning,

knowledge creation processes, and ideas, while

throughputs are the interactions between the exercise

actors (Pinho et al. 2014). Feedback is needed (Tran

& Tian, 2013) to create change to the transformation

process and/or future outputs, to maintain a steady

state (Kast & Rosenzweig, 1972).

The framework of system, structures, and

processes can serve as a grounding concept for

methods recommendations for the different types of

exercises. Expert 07 proposes that develop less

generic standard operating procedures (SOP) “could

help during the evaluation process”, while

“evaluators need to know exactly on what details they

have to keep an eye on” (INEGMA-E

/ Expert 02,

2022).

The results indicate that general systems theory

can provide a useful model of aspects of reality by

providing a methodological approach that helps see

things which otherwise may become easily

overlooked or bypassed (Von Bertalanffy, 1972).

Evaluators can reference to systems thinking to

enhance their theory and practice in their usage of

quantitative and qualitative evaluation methods to

provide very useful ways evaluate how well an

organization performs as an open system seeking to

thrive in a turbulent environment (Midgley, 2006).

Standard operating procedures (SOP) and other

workflow processes help guide and structure

evaluation processes (Hevner & Chatterjee, 2010).

Gregory (2000) recommends ensuring that

participation in evaluation is explicitly considered

rather than ignored or implicitly assumed.

5 CONCLUSIONS

The three-dimensional approach of system, structures

and processes can help address the complex interplay

of factors within civil protection exercises. The

contribution of this study is its attempt to clarify the

theoretical and practical basis and background as they

are examined in literature on systems theory, and

organizational structures and processes as they relate

to the evaluation of civil protection exercises.

Structured methods and SOPs, coupled with

evaluator trainings, can help ensure that evaluation is

appropriately focused on correct issues that are based

on the aims and goals of the exercise.

One main research question that deserves further

study is: “Why evaluate?” This will help understand

how evaluation may promote preparedness,

performance, or collaboration. This study provides

theoretical background to the practice of exercise

evaluation. This understanding may be used for

evidence-based policies and planning of

systematically collected and analysed exercise

evaluation results that in turn help provide needed

inputs to improve relevant procedures and policy

processes.

It is important that exercise evaluators can

recognize what factors that are essential in coping

with the exercise scenario. This study recommends

compiling appropriate sets of questions that can be

used to collect relevant data from exercise to exercise,

over time, and types of exercise. These sets of

questions can be coupled with related standard

operating procedures (SOP).

These aim to provide structure to exercise

evaluation. Further study may help understand how

SOPs can also aid evaluator learning, and through

structured evaluation promote learning outcomes

among all exercise participants and help them

communicate what they have learned. Added clarity

provided by SOPs can help identify higher level

development issues. One main contribution of this

study is the addition to the body of knowledge used

to create structured frameworks for evaluation of

system, structures, and processes (Heath, 1998; Bruns

et al., 2022) practical methodology, SOPs, and trained

evaluators for more systematic evaluation processes.

Further study is recommended on the role of

evaluation in promoting learning among exercise

participants. Sharing information on evaluation

methodology and reports to all individual exercise

participants can promote individual learning. Can e.g.

an evaluation questionnaire open participants’

awareness to how their roles tie into the whole.

Structured methodology provides an opportunity to

KMIS 2025 - 17th International Conference on Knowledge Management and Information Systems

320

collect evaluation data from multiple exercises by

multiple case study method. Cases can be exercises

over time, in different locations, or civil protection

across sectors. As the number of exercise evaluations

increases, the use of multiple case study methods can

provide increasing insights to better understand

exercise evaluation of multiple exercises, of either the

same or different type and of re-occurring exercises

over time.

To achieve this, however European-wide data

would have to be securely stored by someone. To

serve this purpose, parts of the data collected

protocols could be standardized across multiple

evaluations, while the other parts of these data

collection protocols could remain customizable to

best reflect the exercise goals.

The development of evaluation methods and tools

should proceed in a coordinated fashion to provide

solutions that are methodologically sound but also

quick and easy to use in all possible, even very harsh

conditions. Today’s technologies permit the

collection of large sets of hard data (position,

communication, video, etc.) the analysis of which

will also need to be considered. One very important

question to be considered the future development and

benefit of European civil protection exercises will

determine the most suitable actor responsible of the

secure storage and dissemination for shared

knowledge and understanding on a European level;

could this be DG-ECHO or who?

REFERENCES

Ahmady, G. A., Mehrpour, M., & Nikooravesh, A. (2016).

‘Organizational structure’. Procedia-Social and

Behavioral Sciences, 230, 455-462.

Ahrweiler, P. & Keane, M. T. (2013). Innovation networks,

Mind & Society, 12(1), 73-90.

Alkin, M. C. (1970). Evaluation theory development.

Evaluation of short-term training in rehabilitation, 9-16.

Amir, S., & Kant, V. (2018). Sociotechnical resilience: A

preliminary concept, Risk Analysis, vol 38, no 1, 8-16.

Beerens, R. J., Tehler, H., & Pelzer, B. (2020). How Can

We Make Disaster Management Evaluations More

Useful? An Empirical Study of Dutch Exercise

Evaluations. International Journal of Disaster Risk

Science, 11, pp. 578-591.

Bruns, H., Çelikler, J., & Jonitz, J. (2022). D2. 2 A

framework for evaluation concepts WP2—

Methodology & SOP.

Bruns, H. & Kern, E-M (2022). Framework for Evaluation

Concepts in European Civil Protection Exercises.

Proceedings of Disaster Research Days 2022.

Carroll, G. R., Dobrev, S. D., & Swaminathan, A. (2002).

Organizational processes of resource partitioning.

Research in organizational behavior, 24, 1-40.

Canonico, P., Söderlund, J., De Nito, E. & Mangia, G.

(2013). Special issue on organizational mechanisms for

effective knowledge creation in projects: Guest

editorial. International Journal of Managing Projects in

Business, 6(2), 223-235.

Chia, R. (1997). Essai: Thirty years on: From

organizational structures to the organization of thought.

Organization Studies, 18(4), 685-707.

Crossan, M.M., Lane, H.W. & White, R.E. (1999). An

organizational learning framework: from intuition to

institution. Academy of Management Review, 24 (3),

522-537.

Denzin, N. K., & Lincoln, Y. S. (1994). Handbook of

Qualitative Research. Sage Publications, Thousand

Oaks, USA.

Döring, N., & Bortz, J. (2016). Forschungsmethoden und

Evaluation in den Sozial- und Humanwissenschaften.

Berlin Heidelberg: Springer-Verlag.

Eccles, R. G., Ioannou, I., & Serafeim, G. (2014). The

impact of corporate sustainability on organizational

processes and performance. Management science,

60(11), 2835-2857.

European-Commission (2021). Union Civil Protection

Mechanism (UCPM) – Technical Guide for UCPM

Full-scale exercises. Available: https://ec.europa.

eu/info/funding-tenders/opportunities/docs/2021-2027/

ucpm/guidance/technical-guide-full-scaleexercises_

ucpm_en.pdf (accessed on 09 February 2022).

Feller, J., Parhankangas, A., Smeds, R., & Jaatinen, M.

(2013). How companies learn to collaborate:

Emergence of improved inter-organizational processes

in R&D alliances. Organization Studies, 34(3), 313-

343.

Fredrickson, J. W. (1986). The strategic decision process

and organizational structure. Academy of management

review, 11(2), 280-297.

Gasc-Barbier, M., Mateos, R. M., Iasio, C., Chanal, A.,

Villatte, A., Bernardie, S., ... & Monserrat, O. (2024).

Crisis exercise in the framework of coastal geohazards:

Experience in the Balearic islands (Spain).

International Journal of Disaster Risk Reduction, 102,

104270.

Gregory, A. (2000). Problematizing participation: A critical

review of approaches to participation in evaluation

theory. Evaluation, 6(2), 179-199.

Grunig, L. A., Grunig, J. E. & Ehling, W. P. (1992). What

is an effective organization. Excellence in public

relations and communication management, 65-90.

Heath, R. (1998). Looking for answers: suggestions for

improving how we evaluate crisis management. Safety

Science, 30, pp. 151 – 163.

Heinonen, J., & Ruoslahti, H. (2024, June). Measuring

Societal Impacts of Cybersecurity. European

Conference on Cyber Warfare and Security (Vol. 23,

No. 1, pp. 189-196).

Heinonen, J., Ruoslahti, H., Lonka, H., Savolainen, J.,

Jokela, J., Hario, P., ... & Bruns, H. (2024, June).

System, Structures and Processes in Evaluation of Civil Protection Exercises

321

Evaluation Methodology for Civil Protection Exercises.

In 2024 IEEE International Conference on Engineering,

Technology, and Innovation (ICE/ITMC), pp. 1-7.

Hevner, A., & Chatterjee, S. (2010). Design research in

information systems: theory and practice (Vol. 22).

Springer Science & Business Media.

INEGMA-E

(2022). Transcripts of ten expert interviews

(E01 - E10). Unpublished. Efforts under project task 2.2.

Kast, F. E., & Rosenzweig, J. E. (1972). General systems

theory: Applications for organization and management.

Academy of management journal,15(4), 447-465.

Katz, D., & Kahn, R. L. 1978. The social psychology of

organizations, Vol. 2, p. 528, Wiley, New York.

Kelly, L. M., & Cordeiro, M. (2020). Three principles of

pragmatism for research on organizational processes.

Methodological innovations, 13(2), 2059799120937242.

Král, P., & Králová, V. (2016). Approaches to changing

organizational structure: The effect of drivers and

communication. Journal of Business Research, 69(11),

5169-5174.

Lausen, L. & Kastner, R. (2022). D2.1 A Framework for

Exercise Types. International Network of Evaluators &

Guideline for a Methodological Approach in Exercise

Evaluation (INEGMA-E2), R-Document, report, 31

March 2022.

Linkov, I., Eisenberg, D. A., Plourde, K., Seager, T. P.,

Allen, J. & Kott, A. 2013. Resilience metrics for cyber

systems. Environment Systems and Decisions, 33(4),

471-476.

Malone, T. W., Crowston, K., Lee, J., & Pentland, B. (1993,

April). Tools for inventing organizations: Toward a

handbook of organizational processes. Proceedings

Second Workshop on Enabling Technologies-

Infrastructure for Collaborative Enterprises (pp. 72-

82), IEEE.

Mandirola, M., Casarotti, C., Morra di Cella, U., Berton, A.,

Rossi, G., Tacconi Stefanelli, C., ... & Lorusso, O.

(2024). A technical-thematic civil protection exercise in

Italy: UAS fleets-based activities supporting

emergency response in seismic scenarios. Applied

Sciences, 14(12), 5306.

Martínez‐León, I. M., & Martínez‐García, J. A. (2011).

The influence of organizational structure on

organizational learning. International Journal of

Manpower.

Midgley, G. (2006). Systems thinking for evaluation. Systems

concepts in evaluation: An expert anthology, 11-34.

Mitleton-Kelly, E. (2003). Ten Principles of Complexity

and Enabling Infrastructures. Mitleton-Kelly, E. (Ed),

Complex Systems and Evolutionary Perspectives on

Organisations: The Application of Complexity Theory

to Organisations, Pergamon. Amsterdam, 23 - 50.

Orr, K. (1998). Data quality and systems theory.

Communications of the ACM, 41(2), 66-71.

Ozbas, O. (2005). Integration, organizational processes,

and allocation of resources. Journal of Financial

Economics, 75(1), 201-242.

Piperca, S. & Floricel, S. (2012). A typology of unexpected

events in complex projects. International Journal of

Managing Projects in Business, 5(2), 248-265.

Rahimnia, F., Moghadasian, M., & Mashreghi, E. (2011).

Application of grey theory approach to evaluation of

organizational vision Grey Systems: Theory and

Application.

Russo, F., & Rindone, C. (2024, July). Planned and

implemented actions by exercises. In International

Conference on Computational Science and Its

Applications (pp. 28-40). Cham: Springer Nature

Switzerland.

Shirazi, B., Langford, D. A., & Rowlinson, S. M. (1996).

Organizational structures in the construction industry.

Construction Management & Economics, 14(3), 199-

212.

Steins, N. A. & Edwards, V. M. (1999). Collective Action

in Common-Pool Resource Management: The

Contribution of a Social Constructivist Perspective to

Existing Theory. Society & Natural Resources, 12:6,

539-557.

Sun, R., & Henderson, A. C. (2017). Transformational

leadership and organizational processes: Influencing

public performance. Public Administration Review,

77(4), 554-565.

Tran, Q., & Tian, Y. (2013). Organizational structure:

Influencing factors and impact on a firm. American

Journal of Industrial and Business Management Vol.3

No.2(2013), Article ID:30242,8 pages.

Von Bertalanffy, L. (1972). The history and status of

general systems theory. Academy of management

journal, 15(4), 407-426.

Vos, M. (2017). Communication in Turbulent Times:

Exploring Issue Arenas and Crisis Communication to

Enhance Organisational Resilience. Jyväskylä

University School of Business and Economics, N:o 40

/ 2017.

Vos & Schoemaker (2004). Accountability of

Communication Management, A Balanced Scorecard

for Communication Quality. Lemma Publishers,

Utrecht, 2004.

World Health Organization (2017). World Health

Organization WHO Simulation Exercise Manual: A

Practical Guide and Tool for Planning. Conducting and

Evaluating Simulation Exercises for Outbreaks and

Public Health Emergency Preparedness and

Response, Available: https://apps.who.int/iris/handle/

10665/254741 (accessed on 15 August 2022).

KMIS 2025 - 17th International Conference on Knowledge Management and Information Systems

322