Multicriteria Decision Method for Project Ranking Considering

Uncertainty

Guilherme Augusto Barucke Marcondes

National Institute of Telecommunications, Inatel, Av. Joao de Camargo, 510, Santa Rita do Sapucai, Brazil

Keywords:

Project Selection, Multicriteria Decision Methods, Uncertainty, ELECTRE II.

Abstract:

Frequently, decision makers face the challenge of selecting projects to be executed. Resources are not enough

for funding all of them. In this challenge, they need the support of a tool or method, due to several criteria to be

considered simultaneously. Multicriteria decision methods can provide a good support. However, as inherent

to all estimation in projects, uncertainty must be addressed. This works proposes a method for incorporating

uncertainty in project selection using ELECTRE II method and Monte Carlo simulation.

1 INTRODUCTION

In general, companies face a challenge when need to

decide about projects to be executed. Available re-

source isn’t enough for executing all of them simulta-

neously Dutra et al. (2014); Agapito et al. (2019). It

leads decision makers to select a subset of projects to

be included in company’s portfolio, among the can-

didates Abbassi et al. (2014). For selection, a rank-

ing prioritizing projects is helpful, considering those

which are more aligned to the strategies and market-

ing demand, aiming to execute the best set of projects

Perez and Gomez (2014).

The correct selection of projects is essential for

companies, avoiding waste of resources Urli and Ter-

rien (2010). Applying formal project selection meth-

ods increases the chances of success Dutra et al.

(2014).

When selecting projects, decision maker needs

to compare several criteria. Multicriteria Decision

Methods (MCDM) help it, allowing elaborate a rank-

ing of options, enumerating from the best to the worst

Wallenius et al. (2008). MCDM applications have

grown in academic work publications Sadi-Nezhad

(2017).

Preference Ranking Organization Method for En-

richment Evaluation II (PROMETHEE II), VIseKri-

terijumska Optimizacija I Kompromisno Resenje

(VIKOR), Technique for Order of Preference by Sim-

ilarity to Ideal Solution (TOPSIS) and Elimination Et

Choix Traduisant la R

ealit

e II (ELECTRE II) are ex-

https://orcid.org/0000-0001-8062-4347

amples of MCDM Brans and Vincke (1985); Opri-

covic (2012); Hwang (1981); Roy and Bertier (1973).

All of them can offer, at the end, a ranking from

the best to the worst options Martins and Marcondes

(2020).

For using MCDM, specialists must evaluate the

criteria, estimating objective values for each project.

It includes uncertainty in selection, once it is inherent

in the estimation process, and a normal and inevitable

phenomenon in projects Bohle et al. (2015). There-

fore, uncertainty should be considered when applying

the MCDM.

Three-point estimation method allows to incorpo-

rate the variation caused in the values due to uncer-

tainty, given that, instead of estimating by a single

value, three are used: most likely, optimistic and pes-

simistic PMI (2017). These three values can be used

to deﬁne a triangular probability distribution for pa-

rameters under analysis, and dealing with uncertainty

in evaluations Stein and Keblis (2009).

Monte Carlo simulation is a tool for considering

uncertainty in evaluations PMI (2017); Marcondes

et al. (2017). Each simulation round draws a random

value for the parameters, based on the deﬁned proba-

bility distribution. After several rounds, you can ob-

serve the variation in the results.

This work proposes a way of considering un-

certainty in ranking projects by MCDM. For each

project, three-point estimation is proceeded, triangu-

lar distributions are set and simulation done. The

proposal is exempliﬁed by applying ELECTRE II

method over a set of eleven real software development

projects (the same procedure could be applied to any

Marcondes, G.

Multicriteria Decision Method for Project Ranking Considering Uncertainty.

DOI: 10.5220/0010183601230128

In Proceedings of the 10th International Conference on Operations Research and Enterprise Systems (ICORES 2021), pages 123-128

ISBN: 978-989-758-485-5

123

other MCDM listed above).

The remaining of this paper is organized as fol-

low: Section 2 presents the principles of multicriteria

decision methods, detailing ELECTRE II; the impor-

tance of uncertainty in project selection problems is

presented in Section 3; Section 4 proposes a method

for selecting projects considering uncertainty; which

is exempliﬁed by a real problem in Section 5; Section

6 concludes the work.

2 MULTICRITERIA DECISION

MAKING

Single criterion decisions are very intuitive, once one

must choose the alternative with higher preference

score. However, when decision depends on more

than one criterion, the choice must consider, for in-

stance, weights and conﬂicts among criteria. It de-

mands more sophisticated methods Tzeng and Huang

(2011).

For project selection, it is important to use MCDM

to ﬁnd an appropriate assessment, once it often is in-

volved with multiple criteria. These methods allow

to rank different alternatives subject to qualitative cri-

teria. Its application has been rising in the past few

years Sadi-Nezhad (2017).

There are a lot of MCDM in the literature. Four of

these methods are: Preference Ranking Organization

Method for Enrichment Evaluation II (PROMETHEE

II), VIseKriterijumska Optimizacija I Kompromisno

Resenje (VIKOR), Technique for Order of Preference

by Similarity to Ideal Solution (TOPSIS) and ELimi-

nation Et Choix Traduisant la R

ealit

e II (ELECTRE

II). PROMETHEE II can be used when a complete

classiﬁcation is necessary in the presence of a ﬁnite

set of alternatives. VIKOR solves decision prob-

lems with criteria of the same priority, identifying

the alternative closest to the ideal (allows the def-

inition of rankings). TOPSIS evaluates the perfor-

mance of alternatives with several comparison crite-

ria. The closer to the ideal solution and far from the

non-ideal, the better the alternative Martins and Mar-

condes (2020). ELECTRE II allows the calculation

of the agreement and disagreement indices for each

alternative, allowing, with these values, the construc-

tion of a preference classiﬁcation Tzeng and Huang

(2011).

For project selection, MCDM must follow the

steps (adapted from Opricovic and Tzeng (2004)):

• Establishing evaluation criteria that relate to

goals;

• Evaluating projects in terms of criteria;

• Applying an MCDM;

• Accepting one alternative (or some of them) as se-

lected.

An MCDM that is able to produce a ﬁnal ranking is

useful. This ranking can indicate a prioritizing list of

projects, allowing the selection of the ﬁrst one, two,

three, four, and so on, options, depending on the num-

ber of projects to be executed. Any method presented

in this section allows decision makers to achieve this

requirement. For this work, the example presented in

Section 5 uses ELECTRE II.

2.1 Elimination Et Choix Traduisant la

ealit

e II (ELECTRE II)

There is four ELECTRE (in English Elimination and

Choice Translating Reality) methods (I, II, III and

IV). They have different applications Opricovic and

Tzeng (2006):

• ELECTRE I for selection, but without a ranking;

• ELECTRE II, III and IV for ranking problems;

• ELECTRE II and III when it is possible and desir-

able to quantify the relative importance;

• ELECTRE III incorporates the fuzzy nature of de-

cision making;

• ELECTRE IV when quantiﬁcation is not possible.

The ELECTRE II method was chosen due to the pos-

sibility of ranking the alternatives and because, in

project selection problems, it is possible quantifying

relative importance. It is an approach for multicri-

teria decision, based on the outranking relation. It

works with the concepts of concordance and discor-

dance. For each alternative (projects, in this paper),

these two indexes are calculated, considering all cri-

teria Opricovic and Tzeng (2006); Tzeng and Huang

(2011).

For identifying how much alternative a is, at least,

as good as alternative b, one must calculate concor-

dance index C(a, b). Discordance index D(a, b) is a

measure of how much strictly preferable alternative

b is in comparison to alternative a Tzeng and Huang

(2011).

Some quantities are needed for calculating these

indexes Tzeng and Huang (2011).

(a, b) =

{

| g

(a) > g

(b)

}

(1)

(a, b) =

{

| g

(a) = g

(b)

}

(2)

−

(a, b) =

{

| g

(a) < g

(b)

}

(3)

ICORES 2021 - 10th International Conference on Operations Research and Enterprise Systems

124

(a, b) =

∑

j∈I

(a,b)

(4)

(a, b) =

∑

j∈I

(a,b)

(5)

−

(a, b) =

∑

j∈I

−

(a,b)

(6)

where:

• i represents the i

selection criterion (i = 1, ..., n);

• g

( j) indicates the preference value of the i

se-

lection criterion for alternative j ( j = 1, ..., J);

• w

is the weight of the i

selection criterion;

• C

∈ [0, 1] indicates if, for the i

selection crite-

rion, alternative b is strictly preferable in compar-

ison to alternative a, or the opposite, respectively.

The concordance index C(a, b) of alternative a with

respect to alternative b is Tzeng and Huang (2011):

C(a, b) =

(a, b) +W

(a, b)

(a, b) +W

−

(a, b)

(7)

The discordance index D(a, b) of alternative a

with respect to alternative b is Tzeng and Huang

(2011):

D(a, b) =

max

i∈I

−

(a,b)

(a) − g

(b)|

max

i∈I

∗

− g

∗∗

)

(8)

where:

• g

∗

is the highest preference value for the i

selec-

tion criterion;

• g

∗∗

is the lowest preference value for the i

selec-

tion criterion.

For ranking the alternatives, decision maker must

compare the lists of concordance index, in descend-

ing order, discordance index, in ascending order.

Alternatively, a ﬁnal ELECTRE II index could be

calculated by:

e = C(a, b) − D(a, b) (9)

The ranking is constructed ordering e in descending

order, from the highest (best option) to the lowest

(worst option) values.

3 UNCERTAINTY

For applying MCDM in projects, it is necessary that

decision maker evaluates and estimates the values of

criteria for each project. Uncertainty is an inherent

effect of estimating and forecasting Marcondes et al.

(2017).

When deﬁning the value of any parameter, the de-

cision maker (or project specialist) chooses the one

that best represents his evaluation. However, this es-

timation may not be accurate. Or even, if more than

one person evaluates and estimates such parameters,

the estimated values may differ.

For instance, if three specialist in the projects de-

ﬁne their values for some selection criteria, instead of

a single value for each one, they are likely to have

a range of values. It represents the uncertainty in

parameter deﬁnition, impacting in ﬁnal decision of

projects.

For minimizing this impact, instead of working

with a single value, three-point estimation can be use-

ful. It is done by estimating the values most likely,

optimistic and pessimistic. The most likely value is

the estimation of the parameter best understanding by

evaluator (probably, this the value estimated if a sin-

gle point estimation is proceeded). Optimistic and

pessimistic values must reﬂect the best and worst sce-

narios, respectively PMI (2017).

These three values can be used to construct a tri-

angular probability distribution PMI (2017), as pre-

sented in Figure 1:

• parameter a is equal to pessimistic estimation;

• parameter b is equal to optimistic estimation;

• parameter c is equal to most likely estimation.

Probability Density Function

Random Variable

a bc

Figure 1: Triangular distribution based on three point esti-

mation.

4 PROPOSED METHOD

The method proposed in this work is for supporting

decision makers in project selection. Its ﬁnal objec-

tive is a deﬁnition of a ranked list of projects, from

the best option to the worst, for supporting decision

of which ones to execute. It is built based on ELEC-

TRE II method, considering uncertainty.

For each project, three parameters are estimated

for each criterion. Triangular distributions are set, as

Multicriteria Decision Method for Project Ranking Considering Uncertainty

125

described in Section 3. Also the weights of criteria

must be established.

Monte Carlo simulation allows incorporating un-

certainty in evaluation. For each round (m rounds),

project parameters are randomly chosen based on the

triangular distributions. The values are normalized in

a common scale, for avoiding distortion due to dif-

ferent value ranges. Then ELECTRE II evaluation is

executed. At the end of rounds, there are m sets of e

indexes calculated, and a ﬁnal e index set is achieved

deﬁning the mean of them.

The algorithm for ranking projects, with a

stochastic approach to address uncertainty in the def-

inition of parameters and using Monte Carlo simula-

tion is the following:

Begin

% m is the number of Monte Carlo rounds.

Define m

Read pessimistic_project_estimation

Read most_likely_project_estimation

Read optimistic_project_estimation

Read criteria_weigths

Repeat m times

{

Define ramdomly project_parameters

Normalize values

Calculate e_index

Store e_index

}

Calculate mean of m sets of e_index values

Define final ranking

End

5 NUMERICAL EXAMPLE

For exemplify, the method proposed in Section 4

was applied in a set of eleven real software develop-

ment projects, from a software R&D service provider.

Three specialists on software projects and market

from the company estimated the values for four cri-

teria:

• C1 - Return/risk rate (weight - 0,4): a ratio be-

tween the estimated return and the associated risk

(from 1 - the lowest to 10 - the highest);

• C2 - Competitiveness improvement (weight -

0,3): the capacity of project for improving com-

pany competitiveness (from 1 - the lowest to 10 -

the highest);

• C3 - Market potential (weight - 0,2): the capac-

ity of project for improving market share or mar-

ket insertion (from 1 - the lowest to 10 - the high-

est);

• C4 - Degree of innovation (weight - 0,1): how

innovative the project is (from 1 - the lowest to 10

- the highest).

The objective was to select three projects for exe-

cution. They should be chosen in order to be more

aligned with the company’s strategies and met the de-

ﬁned criteria in the best way. Preparing a ranking of

projects (based on the criteria and using ELECTRE

II method), the three best ranked ones should be exe-

cuted.

The ﬁrst step was the deﬁnition, by the specialists

consulted, of the values of project estimations. For

each of these, they deﬁned the most likely, pessimistic

and optimistic values, of the criteria deﬁned for se-

lection. Table 1 presents most likely values, which

is generally used when single estimation is applied

in selection. Tables 2 and 3 present pessimistic and

optimistic values, respectively, to allow applying the

selection considering the uncertainty.

Table 1: Projects Characteristics - Most Likely.

Criteria

Project C1 C2 C3 C4

A 10 3 2 3

B 8 5 8 7

C 2 6 5 4

D 1 2 9 10

E 5 9 10 6

F 6 3 2 2

G 7 7 7 9

H 3 5 3 3

I 8 1 6 7

J 9 9 2 4

K 3 8 1 5

Before simulation using the proposed method, one

was made using the single estimation (values pre-

sented in Table 1) as a reference for comparing the

results. In this case, the projects that would be se-

lected are J, E and G. They were the best ranked by

ELECTRE II e index, as can be seen in graphic of

Figure 2, presenting the best option on the left, and

ICORES 2021 - 10th International Conference on Operations Research and Enterprise Systems

126

Table 2: Projects Characteristics - Pessimistic.

Criteria

Project C1 C2 C3 C4

A 8 2 1 2

B 7 4 7 6

C 1 4 3 3

D 1 1 7 9

E 3 6 8 5

F 5 2 1 1

G 5 5 5 8

H 2 4 2 2

I 7 1 4 5

J 6 8 1 3

K 2 7 1 3

Table 3: Projects Characteristics - Optimistic.

Criteria

Project C1 C2 C3 C4

A 10 4 4 5

B 9 6 9 8

C 4 7 6 5

D 2 3 10 10

E 6 10 10 7

F 7 6 3 4

G 9 8 9 10

H 4 6 5 4

I 9 3 7 8

J 10 10 3 5

K 5 10 2 7

the worst on the right.

Figure 2: Projects Ranking - Without Uncertainty.

Applying the selection with uncertainty, as pre-

sented in Section 4 (Monte Carlo simulation with

10,000 rounds), the ranking of projects changed to the

one shown in the Figure 3, indicating projects G, J and

B for execution.

Comparing these two results (without and with

uncertainty), some differences are identiﬁed. The ﬁrst

four positions of ranking changed from J, E, G and B

Figure 3: Projects Ranking - With Uncertainty.

to G, J, B and E. And the best ranked project changed

from J to G. Another difference was the inversion ob-

served in 8

and 9

ranking positions (from C / F to

F / C). The best three options, in simulation with un-

certainty, were projects G, J and B (comparing to re-

sult without uncertainty, project B was included, and

project E excluded). These changes showed the im-

portance of considering uncertainty in project selec-

tion.

Finally, considering the uncertainty in selection,

the chosen projects were G, J and B. They were the

best ones in ELECTRE II e index ranking.

6 CONCLUSIONS

Project selection is a challenge in portfolio manage-

ment in companies. Generally, there is no enough re-

sources for funding all listed projects. Due to this,

decision makers need to choose those which will be

executed.

The selection is often not direct, as it depends on

several criteria, which must be evaluated simultane-

ously. Some of them can also be in conﬂict with each

other.

Multicriteria decision methods can help decision

makers in selection. They are a good solution for

handling decisions that involve multiple criteria, as:

PROMETHEE II, TOPSIS, VIKOR and ELECTRE

II.

However, these methods based the decision on

values estimated for the criteria. As an estimation,

these values can bring uncertainty for selection.

The work presented in this paper proposes a

method to incorporate uncertainty in decision, sup-

porting decision makers. Instead of estimation with a

single value, it is done using three: most likely, pes-

simistic and optimistic. Based on them, one proceeds

a Mote Carlo simulation, deﬁning the parameters ran-

domly in each round (in this case, applying triangular

Multicriteria Decision Method for Project Ranking Considering Uncertainty

127

distribution).

In the numerical example presented, the target was

choosing three of eleven projects. The results indi-

cated an important change. Without uncertainty, the

selected projects would be J, E and G. However, the

ELECTRE II e index ranking changed when select-

ing with uncertainty. After Monte Carlo simulation

(10,000 rounds), projects indicated for execution were

G, J and B. The best option changed from J to G. And

the project E, presented in the list when no uncertainty

was considered, was excluded of the ﬁnal list, includ-

ing project B. It highlighted the importance of con-

sidering uncertainty in selection, due to its impact on

ﬁnal results.

For future works, some issues must be considered

in selection:

• Apply a fuzzy approach to address uncertainty

rather than Monte Carlo simulation;

• Constraints as developers and equipment avail-

able;

• Evaluate projects with more than one MCDM;

• Time needed for each project execution.

REFERENCES

Abbassi, M., Ashraﬁ, M., and Tashnizi, E. S. (2014). Se-

lecting balanced portfolios of R&D projects with in-

terdependencies: A cross-entropy based methodology.

Technovation, 34(1):54–63.

Agapito, A. O., Vianna, M. F. D., Moratori, P. B., Vianna,

D. S., Meza, E. B. M., and Matias, I. O. (2019). Using

multicriteria analysis and fuzzy logic for project port-

folio management. Brazilian Journal of Operations &

Production Management, 16(2):347–357.

Bohle, F., Heidling, E., and Schoper, Y. (2015). A new

orientation to deal with uncertainty in projects. Inter-

national Journal of Project Management, 34(7):1384–

1392.

Brans, J. P. and Vincke, P. (1985). A preference ranking

organisation method: (the promethee method for mul-

tiple criteria decision-making). Management Science,

31(6):647–656.

Dutra, C. C., Ribeiro, J. L. D., and de Carvalho, M. M.

(2014). An economic-probabilistic model for project

selection and prioritization. International Journal of

Project Management, 32(6):1042–1055.

Hwang, C.L.; Yoon, K. (1981). Multiple Attribute Decision

Making: Methods and Applications. Springer-Verlag,

Nova York, EUA.

Marcondes, G. A. B., Leme, R. C., Leme, M. S., and

da Silva, C. E. S. (2017). Using mean-Gini and

stochastic dominance to choose project portfolios with

parameter uncertainty. The Engineering Economist,

62(1):33–53.

Martins, D. T. and Marcondes, G. A. B. (2020). Project

portfolio selection using multi-criteria decision meth-

ods. In IEEE International Conference on Technology

and Entrepreneurship – ICTE.

Opricovic, S. (2012). Multi-criteria optimization of civil en-

gineering systems (in Serbian, Visekriterijumska opti-

mizacija sistema u gradjevinarstvu). PhD thesis, Fac-

ulty of Civil Engineering, Belgrade.

Opricovic, S. and Tzeng, G.-H. (2004). Compromise so-

lution by mcdm methods: A comparative analysis of

vikor and topsis. European Journal of Operational

Research, 156(2):445–455.

Opricovic, S. and Tzeng, G.-H. (2006). Extended vikor

method in comparison with outranking methods. Eu-

ropean Journal of Operational Research, 178(2):514–

529.

Perez, F. and Gomez, T. (2014). Multiobjective project port-

folio selection with fuzzy constraints. Annals of Op-

erations Research, 236:1–23.

PMI (2017). A Guide to the Project Management Body of

Knowledge. Project Management Institute, Atlanta,

EUA, 6 edition.

Roy, B. and Bertier, P. (1973). La methode ELECTRE II:

Une application au media-planning. Operational Re-

search, page 291–302.

Sadi-Nezhad, S. (2017). A state-of-art survey on project

selection using mcdm techniques. Journal of Project

Management, 2(1):1–10.

Stein, W. E. and Keblis, M. F. (2009). A new method to

simulate the triangular distribution. Mathematical and

Computer Modelling, 49(5-6):1143–1147.

Tzeng, G. H. and Huang, J. J. (2011). Multiple attribute de-

cision making: methods and applications. Chapman

and Hall/CRC.

Urli, B. and Terrien, F. (2010). Project portfolio selec-

tion model, a realistic approach. INTERNATIONAL

TRANSACTIONS IN OPERATIONAL RESEARCH,

17(6):809–826.

Wallenius, J., Dyer, J. S., Fishburn, P. C., Steuer, R. E.,

Zionts, S., and Deb, K. (2008). Multiple criteria deci-

sion making, multiattribute utility theory: Recent ac-

complishments and what lies ahead. Management Sci-

ence, 54(7):1336–1349.

ICORES 2021 - 10th International Conference on Operations Research and Enterprise Systems

128