Risk Analysis Techniques for ERP Projects

based on Seasonal Uncertainty Events

Paulo Mannini

, Edmir P. V. Prado

, Alexandre Grotta

1,2 c

and Leandro Z. Rezende

IS Post-graduation Program (PPgSI), University of São Paulo (USP), São Paulo, Brazil

IS Graduation Program, Federal Institute of Sao Paulo (IFSP), São Paulo, Brazil

Keywords: ERP, Risk Analysis, Seasonal Uncertainty Events.

Abstract: Risk management is fundamental in order to increase Enterprise Resource Planning (ERP) project success

rate in order to plan, prevent and react to risks and uncertainties. But based on the literature review, we

identified a few studies relating to both seasonal uncertainty events (SUE) and ERP projects. Given this

context, this research objective is to analyse the most appropriate risk assessment techniques for ERP projects

based on SUE. In order to achieve this goal, we performed and Systematic Review of Literature and we

applied the Delphi technique with Project Management Professionals and Enterprise Directors. According to

the SLR result, we identified 16 techniques that are more suitable to deal with SUE on ERP projects. After

the Delphi panels perspective, six techniques pointed out as the most suitable for these projects. In addition,

we identified that not all techniques described by the literature converged with the researched context reality.

These findings are very relevant for both the Academia and the Industry to scaffolding SUE on ERP projects.

1 INTRODUCTION

Enterprise Resource Planning Systems (ERP)

systems are computer information systems designed

to process organizational transactions and enable

real-time planning, production and response to

consumers (Amid et al., 2012). Many organizations

have been implementing ERP systems via ERP

projects since the 1990s. ERP systems aim to achieve

enterprise uniformity between information systems

and the real business towards making the

organizations more competitive (Rajagopal, 2002).

Anyhow, ERP projects are a major concern to

organizations (Amid et al., 2012). For instance, that

are indications that some ERP projects might have a

bad reputation of being very costly and ineffective for

organizations (Motwani et al., 2005) including the

underdevelopment countries context. Part of these

claims is related to poor risk planning and control

over processes in ERP projects, resulting in negative

effects on the project outcomes (Tsai et al., 2009).

https://orcid.org/0000-0003-3563-4356

https://orcid.org/0000-0002-3505-6122

https://orcid.org/0000-0003-2549-138X

https://orcid.org/0000-0002-5543-9155

An ERP project is often complex and risky: it

requires a large investment across, it takes a long time

to the concluded. An ERP project also carries a high

risk to the organization (Qi & Zhu 2012; Aloini et al.,

2012): one of the top reasons is that managers do not

take into account proper manners to analyze all the

risks involved into an ERP project. Even further,

seasonal uncertainty events (SUE) are misconceived

(Schmidt et al., 2001) on ERP projects. One SUE

example on ERP projects is the freezing period, a

period of time when no software updates are allowed

except for the emergency ones (Neubarth et al. 2016).

However, despite the importance of SUE in ERP

projects risk analysis, there are few studies in the

literature relating to seasonality and project risk

management. Given this context, this research aims to

answer the following research question: What are the

most appropriate techniques to analyze risks in ERP

projects influenced by SUE? Thus the research main

objective is to analyze the most appropriate risk

assessment techniques for ERP projects based on

SUE. We defined two goals to achieve this objective:

Mannini, P., Prado, E., Grotta, A. and Rezende, L.

Risk Analysis Techniques for ERP Projects based on Seasonal Uncertainty Events.

DOI: 10.5220/0009420601730180

In Proceedings of the 22nd International Conference on Enterprise Information Systems (ICEIS 2020) - Volume 1, pages 173-180

ISBN: 978-989-758-423-7

173

(1) identify the techniques used for risk analysis in

ERP projects via a Systematic Literature Review

(SLR); (2) analyze the importance of identified

techniques to SUE via the Delphi technique.

This research has the other five sections. In the

next section, we present the theoretical bases used by

this research. In Section 3, we present the research

methodology that provides a research overview. In

Section 4, we present the data collection procedures.

In Section 5, we present and discuss the research

results, followed by the last section: conclusion.

2 THEORETICAL BASES

This section defines the main concepts used by this

research: ERP, project risk management, seasonality,

and risk techniques and methods.

(1) ERP. ERP systems were originally deployed to

facilitate manufacturing and business processes. Over

time, they evolved to include all organization

processes, such as sales, marketing, and human

resources. Companies are now using ERP via web and

mobile solutions in order to connect the entire value

chain, including their suppliers (Rainer and Cegielski,

2012). Additionally, there is a lack of employees’

knowledge regarding what an ERP system is and how

to operate it (Motwani et al., 2005).

(2) Project Risk Management. There are several

definitions of risk in the literature. One of the most

accepted definition is given by the Project

Management Institute (PMI): considers negative risks

as threats and positive risks as opportunities. For

PMI, project risk management as an area of expertise

that encompasses seven processes: plan risk

management, identify risks, perform a qualitative risk

analysis, perform a quantitative risk analysis, plan

risk responses, implement risk response, and monitor

risks (PMI, 2017). Another acceptable definition of

risks is the deviations from expectations, caused by

uncertainties that impact objectives positively or

negatively (ISO 2016). Anyhow, other organizations

consider the term risk as something negative only

(OGC, 2009; IPMA, 2015). Project managers often

lack knowledge about formal methods for project risk

management planning (Globerson and Zwikael

2002).

(3) Seasonality and SUE. Seasonality is a

periodic variation that presents a constant long-term

pattern. These variations are repeated, such as

annually, semi-annually or quarterly. An example of

seasonality may be a sales increase during the

Christmas season (Passari, 2003). Thus, SUE are

those uncertainties that might have a higher

probability to occur during certain periods know as

seasonally (Acebes et al. 2014). However, a project

risk analysis method must consider several other

aspects. Thus, we consider the seasonality effects on

projects according to three different types.

i. External Environment. For instance, the

winter might be considering an external

environment factor given that, for instance,

the snow might affect a construction project

or even the in-person meetings of an ERP

project (Acebes et al. , 2014).

ii. Products and Services. Seasonality might

affect the way people sell and acquire

products or services, in which the ERP

system might be prepared for those needs

(Mattsson, 2010).

iii. Processes and Operations. ERP system

freezing as an example of the most frequent

IT seasonality, and it affects the process and

company operations (Prado et al., 2017).

(4) Risk Techniques and Methods. The

technique might be defined as the manner that

technical details are addressed in order to achieve the

desired result. The method might be defined as a

systematic procedure, and an inquiry mode applied to

a particular discipline. Thus, techniques are applied

by humans and might utilize one or more methods

towards producing the desired result (PMI, 2017).

Regarding SUE and risk analysis techniques, we

can classify existing methods and their tools to fit into

four technique categories. The qualitative risk

analysis process techniques were based on (PMI,

2017) for RC. The categories DS, DP, and IP were

defined based on (Stair & Reynods, 2017), as follows:

o Risk Identification: groups risk identification

techniques. They are split into two categories:

Risk Categorization (RC) and [Risk] Data

Source (

DS);

o Risk Calculation: groups how risks are

calculated. It is identified by one single category,

the [Risk] Data Processing (DP);

o Risk

Presentation: groups how risk data is

presented and shared among the project team It

is identified by one single category, the [Risk]

Information Presentation (IP).

3 RESEARCH METHODOLOGY

This section defines the research methodology and

phases. In the first phase, we conducted an SLR. At the

second phase, we conduct a Delphi technique, two

rounds were needed. The conduction and the results of

the research are described in the following sections.

ICEIS 2020 - 22nd International Conference on Enterprise Information Systems

174

3.1 Research Phases

This is an exploratory study, based on qualitative data

(Creswell, 2019) applied to ERP system

investigation. It aims to contribute with new findings

regarding the gap in ERP projects regarding SUE.

This research had two phases as outlined in Figure 1.

Figure 1: Research phases.

The first phase (SLR) has the objective to identify

and classify the techniques used to analyze project

ERP project risks according to the literature. In the

second phase, were ranked these techniques

according to the prism of a Delphi technique. The

methodology used by these two phases is detailed at

the next two subsections as follows.

3.2 Systematic Literature Review

The SLR was performed based on (Kitchenham,

2009). The data were acquired from four different

search engines: Scopus, Science Direct, IEEE, and

ACM, covering a state-of-art period from 2012 to

2019. The specific query strings for each search

engine were created according to the following

keyword: ("Project risk analysis" OR "Project Risk

Assessment") AND "Project Risk Management";

(quali* OR quanti*) AND ("Risk Assessment" OR

"Risk Analysis") AND project AND "Risk

Management".

The protocol for quality criteria was based on

(Kitchenham, 2009) and is described as follow:

(1) Exclusion Criteria: i) duplicated articles; ii)

articles whose access were not free; iii) articles that

did not deal with project risk management; iv) articles

that did not address qualitative and/or quantitative

aspects of risk analysis; v) articles that did not provide

details techniques details or characteristics;

(2) Inclusion Criteria: studies that had

techniques for assessment and/or application in risk

analysis projects. We then read the articles to collect

the information according to the details given in the

next subsection.

3.3 Delphi Technique

According to Skulmoski et al. (2007) and Skinner et

al. (2015) Delphi technique is an appropriate

technique for acquiring expert recommendations

when addressing a research problem in the IT field. It

is suitable for ranking technology issues of new IT

product development projects, like the one proposed

in this research.

According to Dalkey and Helmer (1963), the

Delphi technique uses a group of experts, is based on

pre-established criteria, uses multiple rounds of

questioning with these experts, through questionnaire

or interview, and is applied individually to avoid

direct confrontation between them. Skinner et al.

(2015) state that there is no limitation on the number

of experts but should include people with knowledge

and experience in the subject being evaluated.

The Delphi technique has been used for risk

management in IT projects, mainly to prioritize the

risk factors involved in these projects (Huang et al.,

2004; Schmidt et al., 2001; Nakatsu & Iacovou,

2009). This technique is especially suitable for

studies in which the objective is to improve

understanding of problems, opportunities or solutions

(Skulmoski et al., 2007).

The need or not for a new Delphi panel round is

assessed by the Kendall coefficient of concordance (W)

and the statistical significance of this coefficient and

defined by the following formula (Siegel et al., 2006):

𝑊

12𝑆

𝑚²𝑛



𝑛²  1



𝑚

∑

𝑇𝑗





Variable S represents the sum of the standard

deviations of all elements. Variable m represents the

number of panel members. Variable n represents the

number of elements evaluated in the panel. Thus,

variable Tj =

∑

𝑡𝑖³  𝑡𝑖







, where ti is the number of

ranks in the ith grouping and gj is the number of draw

groups in the jth ordering set.

4 DATA COLLECTION

This section describes how both SLR (phase 1) and

Delphi (phase 2) data were collected and aggregated.

The results of this section are presented in the next

section (Results and Discussion).

4.1 SLR Data Collection

The SLR data was collected from the select articles

according to the four steps highlighted in dark-gray in

Risk Analysis Techniques for ERP Projects based on Seasonal Uncertainty Events

175

Figure 1. In summary, the four steps are detailed as

follows: First: we read the titles and abstracts to

verify if they met the defined criteria; Second: we

read the full paper text in order to verify the adequacy

of the article to the research objectives; Third: we

applied the quality criteria; Fourth: we collect all the

relevant information from the remaining 42 articles.

Figure 2: SLR Process.

4.2 Delphi Data Collection

This subsection first describes the participants'

profile, followed by the Delphi technique details. This

section also details the Delphi first and second

rounds.

4.2.1 Panelists Profile

The selection of Project Management Professionals

and Enterprise Directors to compose the Delphi panel

was based on two criteria: professional knowledge of

the topic of research at the academic or the

professional level. All participants were from the

same country. Below are the minimum criteria

considered.

• Degree in the areas of Engineering or

Information Systems. In addition to undergraduate

Information Systems, all undergraduate engineering

was accepted, as the complexity of IT techniques and

methods exhibits typical characteristics of

engineering activities.

• Experience in project management. Selected

professionals must have at least five years of

experience in public or private organizations; or at

least three years of experience in public or private

organizations and with postgraduate in project

management.

The reason for adopting these criteria is because

professionals with technical background and project

management experience have the appropriate profile

to address project risk management issues. After

selecting the professionals who met the minimum

criteria of knowledge and experience, two groups of

experts were defined:

• System analysts, project managers, and project

leaders, hereinafter referred to as Project

Professionals (PP).

• Project professionals who work or have worked

on projects, but held an executive position,

hereinafter referred to as the board of directors (BD).

The selected experts were generically referred to

as “panelists” in subsequent steps of the Delphi

technique.

4.2.2 Delphi Technique Details

The questions sent to the experts were grouped into

the four categories identified in the SLR (RC, DS,

DP, and IP). We use online surveys through the

eSurveysPro platform. Some parts of the

questionnaires were customized for each participant

to facilitate the collection of personal data and to

provide participants with their choices in previous

rounds.

Panelists' opinions were collected through

questions about the level of importance of risk

analysis techniques in SUE. The questions used a

five-point Likert scale: 1-very low; 2-low; 3-medium;

4-high; and 5-very high. The following criteria were

used to finalize the panel rounds:

• The value of the coefficient of concordance W

equal to or greater than 0.5, determining a high

convergence between opinions (Schmidt, 1997).

• The chi-square value (χ2) greater than 43.82.

According to Siegel et al. (2006), chi-square values

for samples with 19 degrees of freedom, have p-value

of less than 0.001. In Fisher's significance scale

(Morettin & Bussab, 2017) p-value less than 0.001

represents a very strong statistical significance of W.

In summary, there were two rounds until the

criteria for completing the research rounds were met.

The first round of the Delphi panel allowed

participants to interact and suggest adjustments to

instrument questions.

The suggestions were analyzed by the researchers

before making any changes. In the first round,

researchers were also allowed to clarify questions for

participants about the research or the applied

questionnaire.

The second round used the outcome of the first

one, providing participants with the first choices and

allowing them to review their previous responses to

achieve convergence between the group. In this way,

participants received feedback from the first round

keeping their anonymity.

Start

Query the

databases

Read titles

and

abstracts

Read the

papers

Review

quality

criteria

385 articles

extracted

122 papers

selected

72 papers

selected

42 papers

selected

End

ICEIS 2020 - 22nd International Conference on Enterprise Information Systems

176

4.2.3 Delphi First Round

The panelists were contacted via email or Linkedin

platform. We explained the purpose of the survey and

provided the access link to the questionnaire.

The questionnaire was custom made, changing

only the names of the participants. It allowed

panelists' contributions to refining the questions of

the questionnaire. All contributions received were

reviewed and necessary adjustments made.

The first round was applied in March 2019 to 34

professionals. After a period of one month, it was

decided to close the first round of the panel with the

participation of 18 panelists, 14 from the PP group

and four panelists from the BD group.

The opinions collected resulted in values of W =

0.29 and chi-square = 99.18, which indicates a low

degree of convergence of opinions among the

panelists. Thus, it was necessary to carry out a new

round with the objective of increasing the degree of

convergence.

4.2.4 Delphi Second Round

The second round of the Delphi panel consisted of the

same questions evaluated in the first round and sent

to the 18 participants in the first round, allowing them

to review their answers. The responses were collected

in May 2019 and were attended by 12 panelists from

the PP group and four from the BD group. Opinions

collected in the second round resulted in values of W

= 0.52, which indicates a high degree of convergence

of opinions among the participating panelists in

(Schmidt, 1997).

The calculated value for chi-square was 158.08,

which indicates that W has high significance. Based

on these values, the results of the second round met

the panel's completion criteria, with W value having

high convergence and high significance.

5 RESULTS AND DISCUSSION

This section presents the SLR results, the Experts

Group Analysis, and the Delphi results. It also

presents the consolidated result. Then we discuss the

results and their limitations.

5.1 SLR Results

As SLR result, we identified 16 different risk

management techniques in project management,

according to the categories defined at the research

method section: Risk Category (RC), Data Source

(DS), Data Processing (DP) and Information

Presentation (IP). The median rank considers the

most/least cited risk analysis techniques in the

literature as detailed in Table 1 as follows.

Table 1: SLR risk analysis techniques.

Technique

Category

X Technique short name Technique Description

Risk Identification: risks are identified by…

Risk category



T01 - Risk source The sources where risks might come from (their sources) 13



T02 - Project area The project area where risks might come from 5

T03 - Project phase Each project phase where risks might come from 3

Data Source



T04 - Experts opinion Taking into account subject matter (experts) opinion or vision 31



T05 - Historical database Investigating and utilizing risk historical data of past projects

or company knowledge

T06 - Stakeholders opinion Taking into account stokeholds opinion or vision 1

Risk Calculation: risks are calculated via…

Data

Processing



T07 - Risk probability and impact

analysis

The probability of a certain risk might occur times the impact

it might cause if it happens



T08 - Modeling and Simulation Computer to model and simulate risk and their impacts 15



T09 - Multicriteria decision making

analysis

Merging several criteria to analyze and decide 12

T10 - Fuzzy Logic Using the fuzzy logic to predict risks and their consequences 10

T11 - Risk interdependence analysis Analyzing the interdependence among risks 7

Risk Presentation: risks are presented as…

Information

Presentation



T12 - Prioritized list of risks A simple list of risks group by their prioritization 18



T13 - Tables A table showing the risks 13



T14 - Charts A chart, like pareto or pie charts 10

T15 - Project relationship managemen

A visual representation of risks and the project management 8

T16 - Impact and probability matrix A matrix that relates each impact with their probability 6

Legend: X= Comparing with the median;  = Above;  = below; f. = Citation frequency;

Risk Analysis Techniques for ERP Projects based on Seasonal Uncertainty Events

177

Table 2: Risk analysis techniques ranking after Delphi panels.

5.2 Experts Group Analysis

The most suitable techniques for risk analysis in SUE

are those classified as very high or high importance

and they are presented in table 2 (T07, T12, T16, T03,

T05, T11 and T02). There was a very strong similarity

between the results of the PP and BD groups, and with

the total number of panelists. Techniques ranked with

importance high and very high are the same for all

groups with only one exception in the BD group. This

group assigns more importance to T01 than to T02

Only five classifications, out of 16 made by the PP

and BD groups, had a gap greater than two. Most of

these techniques (T01, T02 and T03) belong to the

RC category. This shows that the biggest difference

in the assessment made by these two groups is related

to risk categorization techniques.

Professionals that held an executive position (BD)

ranked the techniques that categorize risk by source

and project phase more importantly than techniques

that categorize risk by project area. It is plausible to

conclude that the major importance assigned by these

professionals to phase risk categorization techniques

is that they do not perform activities directly related

to risk mitigation. They manage the project outcomes

and therefore have a different view of the activities.

5.3 Delphi Results

The outcome of the second round of the Delphi panel

consisted of 16 panelists, resulted in the scores shown

in table 2. The scores are presented by panelists group

(PP and BD) in order to compare the answers of these

two groups. The techniques ranking was based on the

five-point Likert scale used by panelists to answer the

questions.

5.4 Consolidated Results

Table 3 presents the techniques classified by citation

frequency in the literature and by the importance

given by the experts. The groups' analysis is presented

below. Risk Categorization (RC).

Table 3: Consolidate literature review vs. experts’ review.

TC Techniques LR* ER*

RC T01 - Risk categorization by source #6 #12

T02 - Risk categorization by project

area

#14 #7

T03 - Risk categorization by project

phase

#15 #4

DS T04 - Experts opinion #1 #10

T05 - Historical database #8 #5

T06 - Stakeholders opinion #16 #11

DP T07 - Risk probability and impact

analysis

#2 #1

T08 - Modeling and Simulation #4 #14

T09 - Multicriteria decision making

analysis

#7 #15

T10 - Fuzzy Logic #10 #16

T11 - Risk interdependence analysis #12 #6

IP T12 - Prioritized list of risks #3 #2

T13 - Tables #5 #9

T14 - Charts #9 #8

T15 - Project relationship

management

#11 #13

T16 - Impact and probability matrix #13 #3

Legend: * = underlined scores are above the median; TC = Technique

category; LR = Literature Ranking; ER = Experts Ranking

Technique T03 was rated by experts as the most

important among risk categorization techniques,

although it is the least mentioned in the literature.

Techniques Level of Ranking Ranking differences

importance All groups PP BD between groups

(A) (B) (C) (AxB) (AxC) (BxC)

T07 - Risk probability and impact analysis Very high 1 1 3 0 2 2

T12 - Prioritized list of risks 2 2 2 0 0 0

T16 - Impact and probability matrix 3350 2 2

T03 - Risk categorization by project phase High 4 4 1 0 3 3

T05 - Historical database 5742 1 3

T11 - Risk interdependence analysis 6660 0 0

T02 - Risk categorization by project area 7592 2 4

T14 - Charts Medium 8 8 8 0 0 0

T13 - Tables 910101 1 0

T04 - Experts opinion 10 9 11 1 1 2

T06 - Stakeholders opinion Low 11 11 12 0 1 1

T01 - Risk categorization by source 12 14 7 2 5 7

T15 - Project relationship managemen

13 12 15 1 2 3

T08 - Modeling and Simulation Very low 14 13 13 1 1 0

T09 - Multicriteria decision making analysis 15 16 14 1 1 2

T10 - Fuzzy Logic 16 15 16 1 0 1

ICEIS 2020 - 22nd International Conference on Enterprise Information Systems

178

However, considering risk management in SUE,

categorization by phase is important, probably due to

it allows better management of seasonal project

uncertainties.

Data source (DS). Technique T05 (historical

database) was rated above the median both in the

frequency of literature citation and in the degree of

importance given by the experts. This result is in line

with the work of Acebes et al. (2014) who used this

technique for risk analysis in SUE. Another finding

was a divergence in classification between techniques

T04 (Expert Opinion) and T06 (Stakeholder

Opinion). While the literature cites more the former,

the experts assigned more importance to the latter. In

addition, for Delphi panel experts the two techniques

have very close importance rankings. Therefore, it is

advisable to evaluate the cost-benefit ratio of

technique T04, because it involves the cost of an

expert opinion whereas technique T06 has a similar

level of importance, with a lower cost.

Data process (DP). Technique T07 (Risk

probability and impact analysis) was the most cited

data processing technique in the literature and

considered the most important by experts. This shows

that the risk exposure measure by probability and

impact is widely accepted by experts and the

literature. On the other hand, the T10 technique

(Fuzzy Logic) is not among the most important

according to experts. We identified that this result

was due to the lack of knowledge of six panelists

about the T10 technique applied to project risk

analysis.

Information presented (IP). Reports showing a list

of prioritized risks were highlighted by experts as the

most important and were the most cited in the

literature. This is evidenced by the T12 technique

(Prioritized list of risks) that was above the median in

experts’ opinion and literature citations. On the other

hand, there is a difference between experts and the

literature: the former prefers information in charts and

matrices and the latter on tables.

In summary, the consolidate results point out the

most relevant techniques to deal with SUE among

those techniques found in the literature and also

among those ones pointed out by experts.

5.5 Research Limitations

The most relevant limitations of this research are

related to the data collection and analysis process.

Qualitative data collection might be subject to human

bias, given they involve human judgment on

collecting and classifying them (Kitchenham, 2009).

In order to avoid bias, we carefully planned and

followed the research methodology according to the

scientific procedures as much as possible.

Additionally, the results might be limited to the

Delphi participants’ vision and contexts only.

6 CONCLUSIONS

ERP systems are a fundamental part of present-day

Enterprise technological infrastructure. Anyhow,

implementing and maintaining these information

systems is not an easy deal, which, most of the time,

is delivered by ERP projects.

Seasonality brings several additional risks that are

misconceived in ERP such as the seasonal uncertainty

events (SUE) (Schmidt et al., 2001). Given this gap,

the main objective of this research was to identify and

analyse the most appropriate techniques to analyze

risks in ERP projects influenced by SUE. We defined

two goals to achieve this objective: (1) identify the

techniques used for risk analysis in ERP projects via

a Systematic Literature Review; and (2) analyze the

importance of identified techniques to SUE via the

Delphi technique.

We identified that, according to the literature,

from 2012 to 2019, there were cited 16 techniques

that have been used to address SUE on ERP projects.

The most relevant ones were, for risk identification:

risk identification by source (T01) and Experts

opinion (T04). For risk calculation: risk probability

and impact analysis (T07) and modeling and

simulation (T08). And finally, in order to present the

risks to the team and stakeholders, prioritized list of

risks (T12) and Tables (T13).

We then presented these finds to participants into

two sessions of Delphi. After their perspective, the

results were for instance that the most used by ERP

project techniques are: Risk probability and impact

analysis (T07), Prioritized list of risks (T12), and

impact and probability matrix (T16). On the other

side, the rare techniques were: Modeling and

Simulation (T08), Multicriteria decision making

analysis (T09) and Fuzzy Logic (T10).

The main research question was: what are the

most appropriate techniques to analyze risks in ERP

projects influenced by SUE? Based on these two

finds, we might state that at least 16 techniques are

actual reported regarding ERP projects. On the other

hand, there were a few techniques such as T07 and

T12 in which both literature review and industry

converged. Given these finds, we can conclude that

there is space in order to match the scientific

researches and academic world with the enterprises’

reality regarding SUE and ERP projects.

Risk Analysis Techniques for ERP Projects based on Seasonal Uncertainty Events

179

The contribution of this work to identify and

summarize all the techniques that related SUE and

ERP projects, thus helping both Industry and

Academic fields to identify, apply and training their

staff and stakeholders on these techniques. As future

researches, to expand to order contexts and countries.

Other identified gap is related to the low usage of

certain techniques, whenever we consider the

Academic or the Industry areas.

REFERENCES

Acebes, F., Pajares, J., Galán, J. M., and López-Paredes, A. 2014.

Exploring the influence of seasonal uncertainty in project

risk management. Procedia - Social and Behavioral

Sciences, 119, 329-338.

Aloini, D., Dulmin, R., and Mininno, V. 2007. Risk management

in ERP project introduction: Review of the literature.

Information & Management., 44(6), 547-567.

Aloini, D., Dulmin, R., and Mininno, V. 2012. Risk assessment

in ERP projects. Information Systems, 37(3), 183-199.

Amid, A., Moalagh, M., and Ravasan, A. Z. 2012. Identification

and classification of ERP critical failure factors in Iranian

industries. Information Systems, 37(3), 227-237.

Creswell, J. W. 2009. Qualitative Methods. In Research Design:

Qualitative, Quantitative, and Mixed Methods Approaches.

Thousand Oaks: SAGE Publications, 183–214.

Dalkey, N., & Helmer, O. 1963. An experimental application of

the Delphi method to the use of experts. Management

science, 9(3), 458-467.

Globerson, S., Zwikael, O. 2002. The impact of the project

manager on project management planning processes.

Project management journal, 33(3), 58-64.

Huang, S.M., Chang, I.C., Li, S.H., & Lin, M.T. 2004. Assessing

risk in ERP projects: identify and prioritize the factors.

Industrial management & data systems, 104(8), 681-688.

IPMA. 2015. Individual competence baseline 4th version.

International Project Management Association.

ISO. 2016. ISO Guide 73:2009. Risk management-Vocabulary.

Retrieved January 15, 2019 from <

https://www.iso.org/standard/44651.html>

Kitchenham, B. et al. 2009. Systematic literature reviews in

software engineering-a systematic literature review.

Information and software technology, 51(1), 7-15.

Mattsson, Stig-Arne. 2010. Inventory control in environments

with seasonal demand in Operations Management Research

3 (3): 138–45. https://doi.org/10.1007/s12063-010-0035-1.

Motwani, J., Subramanian, R., & Gopalakrishna, P. 2005.

Critical factors for successful ERP implementation:

Exploratory findings from four case studies. Computers in

Industry, 56(6), 529-544.

Morettin, P. A., and Bussab, W. O. 2017. Estatística Básica,

Saraiva Educação. Sao Paulo: 8

edition.

Nakatsu, R. T. & Iacovou, C. L. 2009. A comparative study of

important risk factors involved in offshore and domestic

outsourcing of software development projects: A two-panel

delphi study. Information & Management, 46(1), 57-68.

Neubarth, R., Guedes, E., de Araújo, E., & Rosini, A. 2016. IT

Governance and Change Management: Impacts on

Increasing Availability on a Financial Institution's Business

Platforms. In: 13th CONTECSI International Conference

on Information Systems and Technology Management.

OGC. 2009. Managing successful projects with PRINCE2. The

Stationery Office.

Passari, A. F. L. 2003. Exploration of atomized data for retail

sales forecasting using neural networks. Dissertation

Thesis, School of Economics, Business and Accounting,

University of São Paulo, São Paulo, Brazil.

PMI. 2017. A guide to the project management body of

knowledge (PMBOK Guide). Project Management Institute,

Newtown Square, Pennsylvania, 6th edition.

Prado, E., Mannini, P., and Barata, A. 2017. Risk Management

Analysis carried out by Third Parties in Enterprise Resource

Planning Systems. In: 14th CONTECSI International

Conference on Information Systems and Technology

Management, p. 2242-2266.

Qi, D., & Zhu, Z. 2012. The knowledge-transfer risk evaluation

model of the execution of ERP project based on AHP. In:

Fifth International Conference on Business Intelligence and

Financial Engineering, p. 636-640.

Rainer, R. K., & Cegielski, C. G. 2012. Introduction to

Information Systems: Enabling and Transforming Business,

4th edition. USA: John Wiley & Sons.

Rajagopal, P. (2002). An innovation - diffusion view of

implementation of enterprise resource planning (ERP)

systems and development of a research model. Information

and Management, 40(2), 87-114.

Schmidt, R C., Lyytinen, K., Keil, M., & Cule, P. E. 2001.

Identifying software project risks: An international Delphi

study. Journal of management information systems, 17(4),

5-36.

Schmidt, R. C. (1997). Managing Delphi surveys using

nonparametric statistical techniques. Decision Sciences,

28(3), 763-774.

Siegel, S., Castellan, N. J., Carmona, S. I. C., Cláudio, D., Silva,

D., and Accorsi, A. (2006). Estatística Não-Paramétrica para

Ciências do Comportamento. Artmed, Porto Alegre, 3th

edition.

Skinner, R., Nelson, R. R., Chin, W. W., & Land, L. 2015. The

Delphi method research strategy in studies of information

systems. Communications of the Association for Information

Systems, 37(1), 31-63.

Skulmoski, G. J., Hartman, F. T., & Krahn, J. (2007). The Delphi

method for graduate research. Journal of information

technology education, 6:1.

Stair, R., & G. Reynolds. 2017. Fundamentals of Information

Systems. Boston: Cengage Learning.

Tsai, W. H., Lin, S. J., Lin, W. R., and Liu, J. Y. 2009. The

relationship between planning & control risk and ERP

project success. In: IEEE

International Conference on

Industrial Engineering and Engineering Management,

1835-1839.

ICEIS 2020 - 22nd International Conference on Enterprise Information Systems

180