Exercise Evaluation Creating Knowledge to

Improve Information Resilience

Harriet Lonka

and Harri Ruoslahti

North Savo Regional Council, Sepänkatu 1, 70100 Kuopio, Finland

ResLab, Laurea University of Applied Sciences, Espoo, Finland

Keywords: Information Resilience, National Preparedness, Civil Protection Exercises, Exercise Evaluation.

Abstract: Carrying out and evaluating exercises is typically seen as a very pragmatic means to improve performance of

various security related organizations. In this research, we question this traditional perception of exercising

and broaden the perspective to look at the evaluation of exercises as an integral part of the preparedness related

knowledge management. This study utilizes as its theoretical framework a systemic model for information

resilience created in the context of national preparedness by the Academy of Finland project IRWIN

(Information Resilience in a Wicked Environment). The findings of the EU project INEGMA-E

are used as

study material to unveil the specific features of evaluation systematics in reflection to information resilience.

Exercises are understood as systems and evaluation as a process that captures the influencing factors of this

system. In connection to information resilience, exercise evaluation concepts need to consider e.g. individual

agency and group decision making features, while reflecting them with the development goals of the exercise

system. Improvement of exercise evaluation as a process enhances the creation of validated knowledge for

systemic development of preparedness at all levels. Exercise evaluation can also contribute to improving

information resilience in connection to preparedness related governance and decision making.

1 INTRODUCTION

In the complex and deeply interconnected safety and

security environments of today, new approaches to

enhance cooperation between the many different civil

protection professionals are needed. Yet, appropriate

management of performance of very complex

professional networks can only be done with

accurate, timely and reliable information. Exercises

can be excellent means of providing data for creation

of validated knowledge. Evaluation of exercises is a

crucial element to guide validation of this knowledge.

Carrying out of civil protection exercises (CPE)

or their evaluation should not, however, be conducted

in a vacuum. To be meaningful, this activity should

have clear apprehensible connections to wider

societal targets. In Finland CPEs are often

programmed in the context of preparedness activity

outlined by the Emergency Powers Act (EPA), which

sets demands for the different government branches

develop and maintain their preparedness in case of

times of crises. Preparedness thus covers both

https://orcid.org/0000-0001-9726-7956

proactive actions that help prevent and mitigate

crises, and actions during crises – whether or not

directed by enforcement of the EPA regulated

specific powers.

National Preparedness in Finland is directed by

the Government Resolution on Societal Security

(Lonka 2016, Lonka 2013) that sets the general

guidance for preparedness work at the different levels

of administration. Exercises are a crucial element of

preparedness work. Yet, the way the resolution does

not provide guidelines for carrying out exercises and

further, the gathering and analysis of knowledge

gained from them. This article intends to increase

understanding on how knowledge created by the Civil

Protection Exercises could more efficiently feed in

the knowledge base of national preparedness

planning. The issue is central from the point of view

of improvement of the national security related

knowledge management. This is a specific challenge

as national security affects many stakeholders with

divergent interests and perspectives, as no individual

actor has the resources or information to alone

Lonka, H. and Ruoslahti, H.

Exercise Evaluation Creating Knowledge to Improve Information Resilience.

DOI: 10.5220/0013530100004000

Paper published under CC license (CC BY-NC-ND 4.0)

In Proceedings of the 17th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2025) - Volume 2: KEOD and KMIS, pages

205-212

ISBN: 978-989-758-769-6; ISSN: 2184-3228

205

address preparedness on a societal level (Rantamäki

et al. 2023).

The concept of information resilience in the

wicked environment created by the Academy of

Finland project Information Resilience in a Wicked

Environment (IRWIN 2020-2023) is utilized as a

theoretical framework for this study. In project

IRWIN information resilience is approached as a

holistic concept that is strongly embedded in the

structures, processes, institutions and collective

agency of the democratic system. The project has

contributed considerably to widening the earlier,

rather technical definition of information resilience as

documented in Uusikylä et al. (2024) and Rantamäki

(2024): in the field of public policy, information

resilience is often defined as the capacity of

individuals, communities, and societies to withstand

misinformation, disinformation, and other forms of

information manipulation and recover from them.

The concept can, however, be seen as being wider

than only resisting the false information and

mitigating its effects. An important aspect is that

information resilience includes the ability to operate

amid in-formation uncertainty and imperfection. This

sets a societal target to create robust information

infrastructures that enable people to access, evaluate,

and use information effectively. In the context of the

IRWIN project this constructs a central governance

perspective for understanding information resilience

as a systemic phenomenon which becomes formed in

interaction between actors in an uncertain and

fluctuating environment. (Rantamäki & Jalonen

2022.) According to Rantamäki (2024) information

resilience appears in interaction, during uncertainty

and is characterized by systemicity.

The IRWIN project approaches the issue of

robustness of information resilience from two

perspectives: 1) the ability of the governance and

regulatory system to embrace complexity and create

adaptive, agile, and collaborative forms of interaction

and participation, and 2) the role of citizens as agents

in enhancing or hampering information resilience.

The main research question of the project was how to

redesign systems of governance and regulation,

participation of citizens, and protection of the

knowledge society by understanding information

resilience as a critical element for national

preparedness in a complex environment.

Exercises are traditionally seen as a tool to

practice individuals and organizations to carry out

their civil protection tasks resourcefully (Beerens

2021, Beerens et al. 2021). Thus, evaluation of

exercises often focuses rather narrowly on listing

findings of necessary improvements from the point of

view of a specific organization (Borell and Ericsson

2013, Heath 1998). More seldom is the evaluation

process seen to tackle wider aspects of improving

knowledge management, organizational learning, and

communication in wider civil protection networks

(Wybo 2008).

This article studies exercise evaluation concepts,

as investigated and elaborated in the EU project

INEGMA-E

(International Network of Evaluators &

Guideline for a Methodological Approach in Exercise

Evaluation). The study looks at the potential that

exercise evaluation has to feed in and improve

information resilience in connection to national

preparedness and related to its governance and

decision making. The research question that this

article intends to answer: How can civil protection

exercise evaluation enhance the creation of validated

knowledge on disaster management to improve

information resilience in the context of national

preparedness?

The findings of the INEGMA-E

project are used

as study material on the usefulness and improvement

needs of civil protection exercise evaluation. Thus,

this research is not empirical but a theoretical study

combining the findings of the two projects. The

synthesis is structured in line with the theoretical

framework provided by IRWIN project in the fields

of governance and regulation, individuals’ collective

agency and com-batting mis- and disinformation (see

Uusikylä et al. 2024).

2 METHOD

The goal of this research is to deepen understanding

on how civil protection exercise evaluation can

enhance the creation of validated knowledge on

disaster management and thus improve information

resilience in the context of national preparedness. It

uses the findings of the two recent projects:

INEGMA-E

and IRWIN which complete each other.

INEGMA-E

project serves with findings on how

evaluation of exercises can be improved to produce

applicable knowledge for systemic improvement of

organizational performance. The IRWIN project

adopts a systemic view on information resilience that

enables a shift from a descriptive to an explanatory

analysis of crisis preparedness (Uusikylä et al. 2024).

Combination of these two approaches provides an

integrated view on systemic knowledge gathering of

functional preparedness performance, i.e. the exercise

evaluation process. This knowledge feeds in to

replenishing the theoretical framework of in-

formation resilience in the context of national

KMIS 2025 - 17th International Conference on Knowledge Management and Information Systems

206

preparedness. In this integral picture the exercise-

based knowledge elements offer an enabling factor

for explanatory analysis of crisis preparedness which

the IRWIN project targeted for.

Figure 1 presents the theoretical framework of the

study. In this illustration the original systemic

categories of information resilience in the context of

national security as created by the causal loop

diagram technique (CLD) for analytic purposes of the

IRWIN project have been reduced to four for the sake

of the analytical simplicity (Uusikylä et al. 2024). The

theoretical framework of the IRWIN project provides

a theory base which is used to reflect the findings of

the INEGMA-E

project. The method is theoretical

deductive.

Figure 1: The theoretical framework of the study. Based on

systems model for in-formation resilience in the context of

national security (modified by authors from Uusikylä et al.

2024).

In the framework of the IRWIN project (Figure 1), the

four fields cover different aspects which create

building blocks for information resilience in the

contexts of the collective agency of individuals,

governance and regulatory activities, and countering

malicious information actions. The theoretical

background of these concepts will be presented in the

next chapter.

3 THEORETICAL APPROACH

The theoretical approach of this work (Figure 1) is

based on the framework created in the Academy of

Finland project IRWIN. It provides a conceptual

model to understand the role of information resilience

as a part of national preparedness and security of

supply in a wicked and complex environment. The

framework adopts a systemic view of information

resilience enabling a shift from a descriptive to an

explanatory analysis of crisis preparedness (Uusikylä

et al. 2024).

Information resilience is a systemic phenomenon

manifesting itself in the interaction between actors in

an uncertain environment. The logic of emergence

suggests that the interaction between actors generates

information resilience and influences their

interactions. Information resilience strengthens

information-related agency and reduces information-

related vulnerabilities, such as the effects of

misinformation and disinformation. On the other

hand, information resilience can be defined as the

capacity of individuals, communities, and societies to

withstand and recover from misinformation,

disinformation, and other forms of information

manipulation. The concept also encompasses the

ability to operate amid information uncertainty and

imperfection. Establishing information resilience

involves the development of cognitive skills, critical

thinking, and information literacy. Moreover, a robust

information infrastructure that enables actors to

access, evaluate, and use information effectively is

critical. (Rantamäki et al. 2023, Rantamäki 2024)

Collective agency of citizens plays a central role

in tackling the preparedness for unprecedented crises.

In this perspective individuals are not just users of

information but also active agents who participate in

solving issues (Bifulco 2013). Collective agency is

based on the perceived collective efficacy of

individuals, which is shared belief in collective

capabilities to execute actions (Bandura, 2002).

Collaborative governance is closely connected to

the concept of complexity in the sphere of

governance. It illustrates how parts of a system can

self-organize and create seemingly coordinated and

adaptive behaviour even without centralized

government or leadership (Klijn 2008). Complexity

thinking includes a wide set of concepts that can be

used to explore the dynamics of socio-economic

systems (Cilliers 1998, Byrne & Callaghan 2014).

The complexity leadership theory (CLT) developed

in organizational context underlines the point that the

adaptive space of leadership is created when the

informal and formal side of knowledge confront.

Adaptive regulation includes all aspects which

enable good quality characteristics of regulation.

These can be defined according to EU Impact

Assessment Guidelines as so-called SMART

qualities: regulation should be specific, measurable,

achievable, realistic and time dependent. A

comprehensive and holistic approach to regulatory

development facing complex networks and societal

demands is crucial in the context of national crisis

preparedness (Lonka 2016).

Diverse viewpoints including those of citizens

and practitioners are vital for developing robust

regulation and operational practices for national

preparedness (Santonen & Paasonen 2017). The

Exercise Evaluation Creating Knowledge to Improve Information Resilience

207

regulatory burden, i.e., costs and responsibilities,

should be shared and not accumulated in

governmental agencies (Baldwin et al. 2012). These

aspects underline the importance of whole-of-society

participating in sharing the burden of crisis and crisis

preparedness guided by the regulation.

Counteracting mis- and disinformation is a very

important activity in relation to the information

resilience in the context of national preparedness.

Social media in to-day’s world is a central channel for

distributing misinformation and disinformation and it

is also a key instrument for influencing all societal

activity including politics (Norri-Sederholm et al.

2020). A central part of the modern national security

is the capability of public authorities to detect and

counter disinformation and other intentional efforts to

maliciously influence the flow of information.

Information resilience in this context refers to the

notion that during a crisis as well citizens as various

stakeholders of different critical activities must

receive and understand the central governmental

information quickly and accurately. That is a

prerequisite for them to behave accordingly in their

private sphere and in various societal and public

activities during times of crises.

4 THE CONTEXT OF CIVIL

PROTECTION EXERCISES

In the INEGMA-E

project the evaluation of civil

protection exercises has been viewed as a ground to

train and validate tactics and strategies, to identify

strengths and shortcomings and to stimulate exchange

on best-practices. All in all, these provide an

opportunity for learning and development.

The three pillars of the INEGMA-E

project were:

1) To develop an adequate and versatile evaluation

methodology, addressing the different types of

existing exercises. 2) To explore existing tools,

software solutions and technical tools alike, which

can facilitate data collection throughout an exercise

process. 3) To create an international pool of

evaluators, who meet a certain skill set that is defined

and trained during the project. (Bruns et al. 2022)

Systematically interconnecting evaluation

methods, tools and available evaluators can

importantly contribute to a continuous improvement

of the outcomes of European civil protection

exercises (Heinonen et al. 2024). It is central to

understand these elements from a holistic view

because civil protection exercises create a European

system that includes many national and regional

systems of systems and sub-systems (Heath, 1998;

Heinonen et al. 2024). Exercise evaluation should

have a wider scope and be seen as an instrument for

quality assurance, validation, accountability, and

knowledge gathering in the field of civil protection.

This chapter presents key findings of the

INEGMA-E

project in connection to the need for

development of the evaluation of exercises in a

systemic manner. These findings are then reflected

with the theoretical framework of the IRWIN project.

The INEGMA-E

project analysis was structured

according to the exercise evaluation focus divided in

three entities which are systems, structures, and

processes (Ruoslahti & Lonka 2024). These three

categories are used as a basis for identifying the key

activities which are central goals for practical

exercise evaluation efforts.

Firstly, exercise evaluation targets at capturing the

influencing factors of a system. System in this context

refers to the man-made or natural formations present

in the exercise situation. Such system-influencing

factors may include, for example, project

management, exercise preparation, the scenarios, on-

site security, or cultural factors. All these factors can

be influenced by practical organizers or

administrative level planners of the exercise and its

goals. Possible flaws in project management can lead

to the exercise objectives being lost in the preparation

phase, while poorly prepared and designed locations

can harm the realization of exercise targets in field. In

such situations the failure of the exercise results can

be seen as due to the flaws in the conditions of the

operational environment. (Bruns et al., 2022).

A second category to focus on in the evaluation

are structures referring to devices and mechanisms by

which the exercise system is operated and managed

(Ruoslahti & Lonka 2025). The focus of exercise

evaluation in this category lay in the structures

designed for response management, while evaluation

may measure and analyse resolution strategies as well

as possible alternative measures (Döring & Bortz, p.

1009). The organizational structure of an exercise

determines widely the information flows between

system parts and sub-systems. In centralized

structures decisions would flow top-down, while in

decentralized structures, decision-making becomes

distributed among various levels of the exercise

organization. Structures may either lead to a defined

chain of command, or in other case allow every actor

to have a high level of personal agency (Ruoslahti &

Lonka 2025).

The organizational processes as evaluation foci

refer to how structures are managed. They cover the

activities which establish the goals of the exercise

KMIS 2025 - 17th International Conference on Knowledge Management and Information Systems

208

from the point of view of a specific participating

organization and guide the analysis of how operations

and human interactions are carried out to manage the

exercise scenario. In the context of exercise

evaluation, factors such as communication and

cooperation in realizing exercise scenarios can be

used to scrutinize the exercise system. In the process

approach communication contributes to three phases

of value creation (1) input, (2) throughput and (3)

output (Vos and Schoemaker 2004). Evaluation

within an exercise system, consider its structures and

processes in relation to the requirements of the

exercise scenario. Looking at civil protection

exercises through this lens can help systematize the

evaluation of how these three concepts become

addressed across the exercise and its different

scenarios (Table 1).

Table 1: Input, throughput, and output communication in

the context of civil protection exercises (modified by

authors from Vos and Schoemaker 2004).

Input

communication

Throughput

communication

Output

communication

Reading

materials

(e.g., planning

documents)

Observation

Interview

Survey



Analysis of data

Communication

between

evaluators

FEEDBAC

After event hot

wash

Evaluation

report



As seen in Table 1, the notion of input –

throughput – output – feedback can help make sense

evaluation foci. In an exercise context these can be

seen to form a process cycle of interrelated activities.

Civil protection exercises and their evaluation

projects gather teams of diverse expertise to achieve

common goals that demonstrate a need to actively

manage communication in all these three phases of

knowledge development in the context of civil

protection exercises (Canonico et al. 2013).

Inputs may be resources, and outputs learning,

knowledge creation processes, and ideas (Mitchell &

Boyle 2010), while throughput are the interactions

between the exercise actors (Pinho et al. 2014).

Evolving input, throughput and output

communication can engage various stakeholders and

civil protection experts in knowledge sharing and

adapting participation strategies over time.

5 RESULTS

The added value of this research is that it connects the

concept of civil protection exercise evaluation into a

framework of creating and enhancing information

resilience. The common factor between these two

seemingly remote approaches is systemic knowledge

creation. The systemicity in exercise evaluation refers

to evaluation as a process. This process ideally

combines parts of individual exercises (sub-systems)

to one analytical entity (a system). Furthermore, it can

enhance selected data sets from individual exercises

so that they could be compared between e.g. different

exercise types or over time (Ruoslahti & Lonka

2024). Information resilience is a concept that

particularly underlines the fact that resilience in

connection to the information dimension of crises

requires active networked cooperation and interaction

be-tween different systems. In addition, the study of

Rantamäki (2024) emphasises the importance of

creating an operating culture that accepts information

uncertainty, maintains systemic trust, and proactively

strengthens information-related capabilities in the

context of crises and preparedness.

For this study we chose the specific elements of

the IRWIN project-based systems model to assist

analysis of exercise evaluation features which support

or resist creation of information resilience. These

elements are: 1) shared knowledge and

understanding; 2) decision making capacity and

learning; 3) evidence-based regulation; 4) countering

mis- and disinformation. These elements function in

different manners for different types of exercises and

in connection to different types of exercise evaluation

concepts. The collective agency of individuals and

countering mis- and disinformation plays a key role

in study of the exercise organization processes.

Governance and regulation related elements, on the

other hand, are central for more systemic analysis of

information flows in connection to the civil protection

exercises.

Shared knowledge and understanding are clearly

prerequisites for exercise process to fulfil its tasks.

These are connected to how individuals create

collective agency which is central for different task

forces at the exercise to carry out their organizational

and group related duties. Human interaction,

communication and collaboration are key activities in

successfully fulfilling the tasks related to the exercise

scenario. For the process analysis of exercise

evaluation, understanding the rationale behind

individual actions and decision making that concerns

shared actions becomes central. For this evaluation

Exercise Evaluation Creating Knowledge to Improve Information Resilience

209

focus e.g. individual and group interviews are

important means of gathering information.

The governance and regulation related elements

of the framework, i.e., decision making capacity and

learning as well as enhancing evidence-based

regulation, are elements that enable successful

exercise processes and interconnect the exercise

process to the meta-level expectations and targets of

the exercise as a system of systems. The findings in

relation to these elements feed into different

administrative levels and reveal the development

needs on organizational level and on leadership

matters. These aspects have not been part of the

INEGMA-E

project, the focus of which was more in

the civil protection exercises concentrating on

development of exercise activities of individual civil

protection professional organizations. Evaluation of

wider interconnected systems like the EU MODEX

exercise system, which involve larger variety of

different stakeholders, could benefit from this

approach.

Countering mis- and disinformation will probably

gain more attention in the future as many aspects of

exercise processes, including their evaluation,

become increasingly dependent on digital tools and

database solutions, which may increase systemic

vulnerabilities if providing effective channels for

malicious information that may influence the exercise

process.

Exercise evaluation findings should also, in the

future, provide more information on aspects of cyber

and information security. As digital elements become

a more integral part of exercise scenarios the creation

of information safety and security sensitive exercise

scenarios will be an important challenge. Evaluation

findings can become more useful in enhancing

systemic improvements in the context of information

resilience.

6 DISCUSSION

The three exercise evaluation concepts: system,

structures, and processes influence each other and

form a tight coupling with many interdependencies

(Vos 2017). There are complex interactions between

people, technologies, and processes as well as with

governance systems at the different levels behind and

inside the exercise organizations (Lonka and Wybo

2005). These create complex leadership systems but

also socio-technical ones combining human activities

with information technologies (Wybo and Lonka

2002).

When we study information resilience in

connection to exercise evaluation, the application of

systemic approaches to evaluation tasks becomes

central. It makes a difference, whether an exercise is

studied as an adaptive system or a strictly formalized

activity. The latter approach easily leads to the results

which can only reveal findings already written down

in the exercise plan (Wybo 2008). At the same time,

it may strongly diminish the sphere of individual

agency and innovativeness in the exercise situation.

We claim that understanding the exercise as a

system opens avenues for wider applicability of

evaluation findings. The concept of information

resilience supports the analysis of exercise evaluation

findings in connection to networks of participants and

stakeholders, improvement of their knowledge

exchange and mutual learning and in developing

exercise processes as individual cases or as combined

cases over time (Ruoslahti & Lonka 2025, Yin 1984).

The real challenge for exercise evaluation lies in

the actual planning of the exercises. According to

Wybo (2008) the simulation, which an exercise is,

might assist in enhancing the learning from the

extraordinary situations – which crises are. There are,

however, important obstacles for this learning. “… in

some contexts, all members of the organization are

invited to integrate safety within their professional

identity. During degraded situations, experts tend to

consider adaptations and innovations as a ‘normal

way’ to use their skills and experience when they

must cope within a risky situation. By doing so, they

will be reluctant to talk about it and this ‘expertise in

action’ will not be shared and learned by other

players.” The question remains, how to enable

adaptive reactions at the exercise situations and how

to be able effectively use these experiences as lessons

learned in the evaluation process.

For the lessons learned in connection to

collaborative governance and adaptive regulation, the

system guiding the exercise scenario work as well as

evaluation concepts play a central role. Again, the

planning phase of the exercise becomes crucial, and

it would be preferable to involve as many

stakeholders to the exercise planning as possible

(Lonka and Wybo 2005). If we target the exercise

evaluation to provide useful feedback for governance

and regulation, we need to involve appropriate

stakeholders to both scenario and exercise planning

process. In some sense exercise planning and

evaluation are similar processes as stakeholder

hearings in drafting laws, where a participatory

approach can bring along crucial information for

implementation (Keinänen et al. 2024).

KMIS 2025 - 17th International Conference on Knowledge Management and Information Systems

210

7 CONCLUSIONS

This article has discussed on how evaluation of civil

protection exercises can be used to enhance the

creation of validated knowledge for improving

information resilience in the context of national

preparedness. Information resilience is understood as

a holistic concept that is strongly embedded in the

structures, processes, institutions, and collective

agency of the democratic system (Uusikylä et al.

2024). The specific value of the concept of

information resilience in the scope of this research is

that it helps to focus on characteristics of knowledge

and its formation in uncertain and fluctuating

environment and in interaction between groups and

individuals (Rantamäki & Jalonen 2022, Rantamäki

et al. 2023).

When analysing civil protection exercise

evaluation from aspects of systems, structures and

processes we can discern the specific role of creation,

use and flow of knowledge and information in each

of them. At the systems level the evaluation attention

is focused on exercise administrative level like

project management, exercise preparation, scenarios,

on-site security decisions and general cultural factors.

The critical actors at this level are practical organizers

and administrative level planners setting the goals for

the exercise and refining the exercise plans. The role

of information and knowledge is clearly very similar

as in general governance work in planning,

organization and decision making.

At the structures level the evaluation focuses on

devices and mechanisms by which the exercise

system is operated and managed. Examples of these

are response management, resolution strategies and

decisions on possible alternative measures to solve

the situations in the field. At this level the exercise

evaluation concepts need especially to consider

individual agency, group decision making and

mechanisms targeting to fulfil the systemic goals of

the exercise.

On an organization level, evaluation focuses on

how structures are managed. This includes activities

which establish the goals of the exercise from the

point of view of a specific participating organization

and guide the analysis on how operations and human

interactions are carried out to manage the exercise

scenario. The organizational structure of an exercise

determines widely the information flows between

system parts and sub-systems. In the context of the

exercise evaluation at this level, factors such as

communication and cooperation in realizing the

exercise scenario become central.

The key finding of this study is that the concept of

information resilience can serve as a useful guideline

in creating the analytical frameworks for civil

protection exercise evaluation. It is obvious that the

information revealed by the exercise evaluation can

and should be utilized also wider than only for the

feedback and improvement of the organizations

whose activities have been the primary focus of an

exercise. Civil protection activity is not only

operational work but does have bearing to the wider

perspectives of the societal and even national

preparedness. We should improve the usefulness of

exercise evaluation process to better provide such

preparedness related information.

Further study is recommended with actual

interaction data to help validate the effectiveness of

the discussed features for further practical

implementation and to examine potential for stronger

real-world impacts.

REFERENCES

Baldwin, R., Cave, M. & Lodge, M. (2012). Understanding

Regulation: theory, strategy, and practice. Oxford

University Press.

Bandura, A. (2002). Social Cognitive Theory in Cultural

Context. Applied Psychology, 51.

Beerens, R. J. (2021). Improving disaster response

evaluations: Supporting advances in disaster risk

management through the enhancement of response

evaluation usefulness. Lund University, Division of

Risk Management and Societal Safety, Faculty of

Engineering, Lund.

Beerens, R. J., Tehler, H., & Pelzer, B. (2020). How Can

We Make Disaster Management Evaluations More

Useful? An Empirical Study of Dutch Exercise

Evaluations. International Journal of Disaster Risk

Science, 11, pp. 578-591.

Bifulco, L. (2013). Citizen participation, agency and voice.

European Journal of Social Theory, 16 (2), 174-187.

Borell, J., & Eriksson, K. (2013). Learning effectiveness of

discussion-based crisis management exercises.

International Journal of Disaster Risk Reduction, 5, pp.

28-37.

Bruns, H. & Kern, E-M. (2022). Framework for Evaluation

Concepts in European Civil Protection Exercises.

Proceedings of Disaster Research Days 2022, October

13th to 14th at the University of Innsbruck.

Byrne, D. & Callaghan, G. (2014). Complexity Theory and

Social Sciences. The State of the Art. Routledge, New

York, NY.

Canonico, P., Söderlund, J., De Nito, E. & Mangia, G.

(2013). Special issue on organizational mechanisms for

effective knowledge creation in projects: Guest

editorial. International Journal of Managing Projects

in Business, 6(2), 223-235.

Exercise Evaluation Creating Knowledge to Improve Information Resilience

211

Cilliers, P. (1998). Complexity and Postmodernism.

Understanding Complex Systems. Routledge.

Döring, N., & Bortz, J. (2016). Forschungsmethoden und

Evaluation in den Sozial- und Humanwissenschaften.

Berlin Heidelberg: Springer-Verlag.

Heinonen, J., Ruoslahti, H., Lonka, H., Savolainen, J., Jorma

Jokela, J., Hario, P., Kivinen, L., Lausen, L., Kastner, R.,

Bruns, H. (2024). Evaluation methodology for civil

protection exercises. 2024 IEEE International

Conference on Engineering, Technology, and Innovation

(ICE/ITMC), Funchal, Portugal, 2024, pp. 1-7.

Klijn, E. H. (2008). Governance and governance networks

in Europe. Public Management Review, 10:4, 505–525.

Keinänen, A., Lonka, H., & Jukka, L. (2024). Participatory

Regulation as a Tool for Tackling Complex Security-

Related Challenges. In Information Resilience and

Comprehensive Security: Challenges and Complexities

in Wicked Environments (pp. 163-177). Cham:

Springer Nature Switzerland.

Lonka, H. (2013). National security strategy as a

background to disaster planning: A Finnish Case Study.

Conference proceedings, IIIRR 2013 Conference.

QUT, Brisbane, Australia.

Lonka, H. (2016). Governmental Security Strategy as a

Shaper of Legislation. Publications of the UEF,

Dissertations in Social Sciences and Business Studies,

no 129.

Lonka, H.& Wybo, J-L. (2005). Sharing of experiences. A

method to improve usefulness of emergency exercises.

Int. Journal of Emergency Management Vol.2, No. 3,

2005.

Mitchell, R. & Boyle, B. (2010). Knowledge creation

measurement methods. Journal of Knowledge

Management, 14(1), 67-82.

Norri-Sederholm T., Norvanto E., Talvitie-Lamberg K.,

Huhtinen A-M. (2020). Social Media as the Pulse of

Misinformation and Disinformation in National

Security. Moehlecke de Baseggio E.& al. (Eds.) Social

Media and the Armed Forces. Advanced Sciences and

Technologies for Security Applications. Springer, pp.

207-225.

Olsén, M., Hallberg, N., & Mattsson, K. D. (2019). Who

Learns from Crisis Management Exercises: An

Explorative Study. 16th ISCRAM Conference.

València.

Pinho, N., Beirão, G., Patrício, L. & P. Fisk, R. (2014).

Understanding value cocreation in complex services

with many actors. Journal of Service Management,

25(4), 470-493.

Rantamäki, Aino (2024). Hallintaa epävarmuudessa –

informaatioresilienssi kriiseissä ja niihin

varautumisessa. Acta Wasaensia 530. University of

Vaasa. Academic Dissertation 22 March 2024.

https://urn.fi/URN:ISBN:978-952-395-133-4

Rantamäki, Aino (2023). “Kun talo on tulessa, on

keskustelua vaikea käydä” – Tapaustutkimus

informaatioresilienssin ilmaantumisesta

koronapandemian aikana [“When the house is on fire,

it is difficult to have a conversation” – Case study on

the emergence of information resilience during the

COVID-19 pandemic], Hallinnon tutkimus, 42 (3):

284-302. https://doi.org/10.37450/ht.119755

Rantamäki, A., Uusikylä, P. & Jalonen, H. (2023).

Information Resilience in Networks: An Analysis of a

National Security Legislation Evidence Base.

Scandinavian Journal of Public Administration, Vol:

Articles in Press. DOI: https://doi.org/10.

58235/sjpa.2023.14068

Rantamäki, A. & Jalonen, H. (2022). Hallinnon

informaatioresilienssiä etsimässä – Tutkimusmatka

käsitteen juurille [Searching for information-resilient

governance – A journey to the roots of the concept],

Hallinnon tutkimus, 41 (1): 35–51. https://doi.

org/10.37450/ht.107861

Ruoslahti, H. & Lonka, H. (2025). System, Structures and

Processes in Evaluation of Civil Protection Exercises.

Accepted as short paper to KMIS 2025

Santonen, T. & Paasonen, J. (2017). Evaluating the

adequacy of private security industry regulation in

Finland. Security Journal 30, 585–604.

Uusikylä, P., Jalonen, H. & Jokipii, A. (Eds.) (2024).

Information Resilience in a Complex System of

National Security. Palgrave Macmillan (in print).

Vos, M. (2017). Communication in Turbulent Times:

Exploring Issue Arenas and Crisis Communication to

Enhance Organisational Resilience. Jyväskylä

University School of Business and Economics, N:o 40.

Vos M. & Schoemaker H. (2004). Accountability of

Communication Management, A Balanced Scorecard

for Communication Quality. Lemma Publishers,

Utrecht, 2004.

Wybo, J-L (2008). The Role of Simulation Exercises in the

Assessment of Robustness and Resilience of Private or

Public Organizations. Hans J. Pasman and Igor A.

Kirillov (Ed.) (2008): Resilience of Cities to Terrorist

and other Threats - Learning from 9/11 and further

Research Issues, Pages 491-507. DOI: 10.1007/978-1-

4020-8489-8_23

Wybo, J-L. & Lonka, H. (2002). Emergency management

and information society: how to improve synergy? Int.

Journal of Emergency Management, Vol. 1, No. 2, pp.

183-190.

KMIS 2025 - 17th International Conference on Knowledge Management and Information Systems

212