Modified, Stakeholders Perspective based DEA Approach

in IT and R&D Project Ranking

Dorota Kuchta

, Dariusz Skorupka

, Artur Duchaczek

and Magdalena Kowacka

Faculty of Computer Science and Management, Wrocław University of Technology,

Smoluchowskiego Street 25, 50-372 Wrocław, Poland

Faculty of Management, The General Tadeusz Kosciuszko Military Academy of Land Forces,

Czajkowskiego Street 109, 51-150 Wroclaw, Poland

Key

words: Data Envelopment Analysis, Project, Project Stakeholder, Project Input, Project Output.

Abstract: The state of art of Data Envelopment Analysis applied to IT and R&D projects evaluation and ranking is

presented. Then the role and importance of project stakeholders is discussed. It is shown that this role is not

taken into account in the original DEA method applied to project evaluation and ranking. Thus a modification

of the DEA method is proposed, in which the stakeholders perspective plays a crucial role. An example

illustrates the modified method itself and its advantages.

1 INTRODUCTION

The Data Envelopment Analysis (DEA) has been

used for years in the evaluation and comparison of

production units, where outputs (produced goods) are

generated from inputs (raw resources and labour)

(Charnes et al., 1978). The term of production units

used in the DEA method has been generalised, so that

bank divisions and hospital units can be evaluated,

ranked and compared too. The inputs in such cases

are labour hours and budgets, the outputs - the

services rendered, the number of patients or clients

served etc. However, in recent years the method has

been used among others to evaluate, compare and

rank projects, where inputs and outputs may be of a

different nature than in case of production units, even

in the generalized sense. In case of projects,

especially R&D or IT projects, the inputs and above

all outputs may be of a qualitative nature - for

example, an important output of a project may be the

customer satisfaction. Still, even in such a case the

DEA method has turned out to useful in project

evaluation and ranking - the relevant literature will be

discussed in Section 2.

However, in the DEA method applied to projects

there is an inherent mistake. The DEA in its original

form (see Section 2) maximises the ratio “weighted

outputs sum/weighted inputs sum” for each project in

turn, while the decision variables are the weights of

inputs and outputs. The idea is that each project has

the right to “decide” which weights to use with

respect to the inputs and outputs in order to show

itself in the most positive light. Thus, the weights may

take any nonnegative value, the only condition is that

the ratio “weighted output/weighted input” is

maximal. However, it has been overlooked that

projects are seen differently by different stakeholders

(see Section 3). Thus, if the idea is to give to each

project the possibility to show itself in a positive light,

it should also be possible for each project to choose

the best perspective among those of different

stakeholders – and in the eyes of individual

stakeholders certain sets of inputs and outputs do not

count. As it will be shown in Section 4, the possibility

to consider various views of various decision makers

is not present in the original DEA method. We will

introduce it in Section 4. In Section 5 it will be

illustrated by means of a computational example.

2 DEA METHOD APPLIED TO

PROJECT RANKING

As mentioned above, the DEA method has been

recently used to evaluate and compare various units,

which may be production units, banks, hospital units,

and, last but not least, projects. The basis for the

comparison are inputs used and outputs generated by

158

Kuchta, D., Skorupka, D., Duchaczek, A. and Kowacka, M.

Modiﬁed, Stakeholders Perspective based DEA Approach in IT and R&D Project Ranking.

In Proceedings of the 18th International Conference on Enterprise Information Systems (ICEIS 2016) - Volume 2, pages 158-165

ISBN: 978-989-758-187-8

the units in question. If there are 1 units, each of

them is evaluated by a separated fractional

programming model (which may be reduced to a

linear one), where the unit being evaluated is

numbered as the 0-th unit. The model is formulated

as follows (Charnes et al., 1978).



∑

















∑

















→

(1)

∑

















∑



















1,1,…,

(2)







0, 





0

(3)

where 





  1, … ,  are inputs of the k-th object,

0,…, and 





  1, … , are its outputs,







1,…, and 





  1, … ,  are decision

variables and at the same time weights of the,

respectively, inputs and outputs, chosen from the

point of view of the 0-the unit - in order to maximise

its performance, defined in (1).

Each unit becomes in turn the 0-th unit. The value

of the objective function of the fractional

programming problem (1)-(3) formulated when the k-

th unit becomes the 0-th unit will be denoted as 



and will be considered to be the performance of the k-

th unit. The units are than ranked in the decreasing

order of 



  0, … . ,.

The ratios 







0,….,



correspond to the

situation when each unit has the right to choose the

inputs and outputs weights is such a way that the ratio

“weighted sum of outputs/weighted sum of inputs” as

for it as high as possible, and the only constraints are

(2) and (3).

Recently the DEA method has been often applied

to projects, especially to IT and R&D projects. In the

following we will show which inputs and outputs are

chosen for projects when they are compared by means

of the DEA method.

For IT projects (understood as projects

concerning the development, installation and use of

computer systems and applications (American

Heritage Dictionaries, 2014) the authors use the

following inputs:

 cost of the project, duration of the project,

number of employees needed - the actual

and the planned one (Gusmao and Costa,

2012; Asosheh et al., 2010);

 green dollar cost (expenses paid to entities

outside the organisation), brown dollar cost

(internal expenses, the cost of personnel

used), project actual duration, potential risk

(potential loss to the corporation) (Sowlati et

al., 2005).

 labour, other expenses, duration (Wray and

Mathieu, 2008)

The outputs for IT projects have been grouped by

various authors in several categories (Sowlati et al.,

2005; Gusmao and Costa, 2012; Wray and Mathieu,

2008):

 financial perspective (cost reduction of the

organisation in which the product of the IT

project has been implemented, profit

increase of the organisation, green dollar

benefits (profit to the organisation or

reduction of expenses paid outside the

organisation), brown dollar benefits

(reduction of internal expenses, reduction of

the cost of personnel used);

 internal business perspective (a better

control of internal processes, their increased

security achieved thanks to the product of

the project in question);

 customer/stakeholder perspective (increased

customer satisfaction, a higher compliance

with needs of stakeholders) - the role of

various stakeholders is here underlined;

 learning perspective (benefits consisting in

the fact that thanks to the project the

organisation has achieved new skills);

 uncertainty perspective (here the risks linked

to the use of the project product are meant:

the higher the risks, the lower the output of

the project);

 the complexity of the product of the project;

 intangible benefits (to individual members

of the organisation or to organisational units

of the organisation or to the whole

organisation realising the project);

 software production related outputs:

number of function points realised, number

of lines of code.

R&D projects (R&D, i.e. Research and

Development, comprises creative work undertaken

on a systematic basis in order to increase the stock of

knowledge, including knowledge of man, culture and

society and the use of this stock of knowledge to

devise new applications (Frascati Manual, 2002))

have also been subject to evaluation and ranking by

means of DEA method, The following inputs have

been used in the literature (Eilat et al., 2008; Revilla

et al., 2003; Yuan and Huang, 2002):



project cost (budgeted and actual):

 the full time equivalent of highly trained

personnel (managers, engineers and

scientists, holding PHD, master, bachelor

degree) used for the realisation of the

project;

Modiﬁed, Stakeholders Perspective based DEA Approach in IT and R&D Project Ranking

159

 total organisation revenues;

 total organisation R&D Budget, Total

number of corporate employees

 total number of organisation employees;

It has to be underlined that the last three inputs

listed above refer not to the project being evaluated,

but to the whole organisation implementing the

project.

As for the outputs, the following ones have been

used for R&D projects (Eilat et al., 2008; Revilla et

al., 2003; Yuan and Huang, 2002):

 discounted cash flow generated by the

project;

 performance improvement achieved thanks

to the project;

 customer satisfaction with the product of the

project;

 congruence with the strategy of the

organisation realising the project;

 synergy with other projects realised by the

organisation;

 project team satisfaction;

 the number of team members trained in

project management thanks to the project

realisation;

 probability of technological and commercial

success of the project product;

 technical gap size covered by the project

product;

 the newness of the technology used;

 the complexity of market activities needed

to commercialize the project product;

 new scientists gained by the organisation

thanks to the project;

 total income generated by the project;

 number of patents and copyrights gained

thanks to the project;

 number of dissertations worked out thanks

to the project;

 number of reports issued thanks to the

project;

 number of technology innovations worked

out thanks to the project;

 number of seminars organised thanks to the

project;

 number of technology transfers resulting

from the project.

To sum up the state of the art of the DEA

application to IT and R&D projects evaluation and

ranking, it has to be said that the inputs and outputs

used are often of a qualitative nature and their value

has to be given by an expert. Another important thing

is that the inputs and outputs used are not of the same

importance to each project stakeholder. For example,

the team satisfaction, new dissertations and new skills

of the project team may be of a high importance to the

persons responsible of the scientific development of

the organisation members, but it will be of no

importance to the financial manager of the

organisation. For the latter the project cost and the

revenues generated by it will be much more

important.

The problem of the diversity of views of different

project stakeholders and of their importance is treated

in the following section.

3 PROJECT STAKEHOLDERS

AND THEIR ROLE

Project stakeholders (project stakeholder is an

individual, group, or organization who may affect, be

affected by, or perceive itself to be affected by a the

project (PMI, 2013)) are often very diversified, and

often have contradictory expectations with respect to

the project.

This statement can be justified by the list of

stakeholders of a certain R&D project, financed by an

external institution, which was realised be a team at a

university with one of the authors of the present paper

as project manager. It has to be underlined that the list

was not completely identified before the project start.

It is only once the project was closed that the

complete list of the stakeholders was known. This

complete list is as follows:

 the members of the project team;

 the project manager;

 the accounting department;

 the project management department;

 the financial manager of the university;

 the scientific manager of the department.

The members of the project team and the project

manager wanted to attain the project goal, which

consisted in elaborating a new accounting system, but

its achievement depended on the information given

by the accounting department. The accounting

department did not want a new accounting system, so

they did not cooperate, as for them the fulfilment of

the project goal meant more work and more

challenges. In this situation the financial manager of

the university was interested only in one thing: that

the project is not found to be a complete failure by the

financing institution, so that the university does not

have to pay back the project budget. The aim of the

project manager was then to find a substitute goal

which would be accepted by the financing institution.

ICEIS 2016 - 18th International Conference on Enterprise Information Systems

160

The team members were not very motivated to fulfil

the substitute goal, but they did it, in order to help the

project manager. The scientific manager of the

department wanted to have new publications in good

journals and he did not care the realisation of which

goal these publications will present. The project

management department wanted to have all the

reports in due times, so that they could register them

and send to the financing institution. They did not

care much about all the problems which caused that

the project was in danger.

Finally the project was accepted with the substitute,

much more moderate goal, so that the university did

not have to pay any money back and in all official

documents the project is qualified as a success,

although in the eyes of the project team and the

project manager it was a failure, partially due to the

wrongful identification of project stakeholders and

their views. Above all the role of the accounting

department was not identified properly.

The project stakeholders are important. The have

been becoming more and more present in the project

success perception. In the past it was considered that

a project is successful if it meets the specification

(scope), cost (budget) and time (deadline). Today

most authors expand this definition substantially,

introducing other project success measures. The

nature of these extensions can be summarised by the

following statement: “There have to be two groups of

project success measures: objective measures (such

as time or cost) and subjective measures (such as the

satisfaction of different project stakeholders)” (Chan

et al., 2002; Chan et al., 2004). Subjective measures

are necessary, because the perception of project

success depends strongly on the assessor (Davis,

2014).

That is why project stakeholders management is

considered as very important. There are many

proposals of methods to identify and manage

stakeholders, both for projects in general (Hartono et

al., 2014; Missonier and Loufrani-Fedida, 2014) and

for IT (Sudevan et al., 2014) and R&D projects

(Broom et al., 2013; Geeson et al., 2015).

In the existing DEA approaches to project

evaluation and ranking, the stakeholders and their

varying views have not been taken into account. All

the possible inputs and outputs are allowed in model

(1)-(3). Each project tends to maximize its

performance by choosing the weights which

maximize function (1) and are selected among all

non-negative values of the decision variables (3).

However, not all the inputs and outputs are important

for each stakeholder. In many cases the situation will

be as follows: for each stakeholder only a certain

subset of inputs and a certain subset of outputs will

matter, the other outputs and inputs will have weights

equal to zero from the outset. If the philosophy of the

DEA approach consists in letting each project present

itself in the best light, each project should also have

the right to choose the stakeholder for which it has the

best performance. And in some cases the decision

maker will want to asses and rank projects taking

various stakeholders into account. The original DEA

method does not make it possible. The proposal

formulated in Section 4 covers this gap.

Our proposal will allow to asses and rank projects

taking into account for example the following

stakeholders and their diversified views:

1) for IT projects:

a) a customer representative, representing the

upper management level, interested in one

input: the project price (linked to the project

budget), and in three outputs: customer

satisfaction, the risk linked to the project

product and the product complexity;

b) the software producer representative,

representing the upper management level,

interested in one input: the green dollar cost,

and in on one output: the green dollar

benefits;

c) another representative of the same software

producer, representing a lower management

level, interested in one input - the number of

employees that were involved in the project,

and in three types of output: the skills that

the employees gained thanks to the project,

other intangible benefits the employees

gained thanks to the project, intangible

benefits gained through the project team as a

whole;

d) another representative of the same software

producer, representing an upper

management level, interested in three inputs:

the project cost, the project duration, the

potential risk linked to the project

realisation, and in two outputs: the customer

satisfaction and the compliance of the

project with customer strategy - which will

mean chances for more contracts in the

future.

2) For R&D projects:

a) the financing institution, interested in one

output: the budget, and in two types of

outputs: the number of publication in high

quality journals and the commercialisation

chances of the product;

b) the dean of a university faculty at which the

project is realised, interested in two types of

Modiﬁed, Stakeholders Perspective based DEA Approach in IT and R&D Project Ranking

161

input: the full time equivalent of personnel

with various degrees used in the project and

the project duration, and in three types of

output: the numbers of publications, the

number if dissertations worked out and

published, the number of high quality

personnel members employed thanks to the

projects;

c) The financial manager of the institution at

which the project is realised, interested in

one type of input - the time his/her

employees had to spend to help to prepare

the project application and in one output: the

part of the project budget that will be left at

the university disposal.

Of course, the above examples are theoretic,

although partially based on the authors’ experience.

They are not meant to show what the corresponding

stakeholders should be interested in, but what they are

often interested in, even if it is not correct. The main

goal of the above examples is to show that various

stakeholders, also those who are important in the

organisation which realises the project or for which

the project is realised, may have substantially

different views about what the significant inputs and

outputs of a project are. The modified DEA method

will make it possible to take this into account.

4 MODIFIED DEA METHOD

Let  



1,2,…,



be the set of the indices of all

possible inputs that may be considered for the set of

projects which are to be evaluated and ranked, and

 



  1,2, … , 



the set of the indices of all

outputs. There are K+1 projects to be evaluated and

ranked. Let 



  1,2, … ,



be the set of the

indices of all the stakeholders that have been

identified for the projects in question.

For the l-th stakeholder (∈) there are given two

sets: 



⊂ and 



⊂. These sets represent the

indices of the inputs and outputs taken by the l-th

stakeholder into consideration. It follows that from

the point of view of the l-th stakeholder it holds:







0 and 





0

(4)

for all ∈∖



and ∈  ∖ 



where 





and 





are 





and 





from (3) when the k-

the project becomes the 0-th project for problem (1)-

(3), k=0,…K.

We can thus for each project solve L+1 problems:

one identical to problem (1)-(3), where the individual

stakeholders are not taken into account, one for each

stakeholder (∈). The stakeholders linked

problems will be of the following form (for each ∈

):

∑

















∑

















→

∑

















∑

















1, 1,…,

(5)







0, 





0







0 and 





 for all ∈∖



and ∈  ∖ 



where each project in turns becomes the 0

project.

For each project k=0,1,…K we would then have

L+1 evaluations: 



, being the value of the objective

function of the problem (1)-(3), representing the best

possible project evaluation when all inputs and

outputs are put in the same box and treated in the

same way, and 





, ∈, being the objective function

of problem (5), when the k-th becomes the 0

project,

and representing the best possible project evaluation

when only the inputs and outputs important to the l-

th stakeholder are taken into account.

These values can be then interpreted in several

ways. We can aggregate them to a final project

ranking for example as follows (



, k=0,1,…K

stands for the final ranking of the k-th project):









,





,  ∈ 

(6)

or we can define 



as a weighted sum of





and





,∈.

We adopt here the formula (6), where the basic

idea of the DEA method is retained: each project can

present itself in the best possible way, by choosing

only the weights of the inputs and outputs, but also

the stakeholder who would put it in a good position.

But any method of aggregating values 



,





, ∈ is

better than the original approach, in which only

values 



are calculated., because the original method

does not allow to take into account the view of even

the key stakeholders. Using the original DEA method

we may rank lowly some projects which are in fact

good, because they would be highly appreciated by

the key stakeholders whose opinion is crucial for our

organisation. In the next section we will illustrate the

proposed approach by an example.

5 COMPUTATIONAL EXAMPLE

Let is consider 10 R&D projects, whose all possible

inputs and outputs (for all possible stakeholders) are

ICEIS 2016 - 18th International Conference on Enterprise Information Systems

162

given in Table 1. All the inputs names in the Table 1

should be accompanied by words “used in the

project” and all the output names by words

“generated by the project”. The values of the inputs

and outputs have been given by experts and it has

been made sure that they are commeasurable.

Problem (1)-(3), thus the original DEA method,

gives the results shown in Table 2. The original DEA

method would thus give us the following ranking of

the projects: P5, P7, P8, P4, P6, P1, P2, P9, P3, P10.

Let us now consider two key stakeholders (L=2).

We have the following information:

 



 4,5,







1,2,3,4,5



;

 



={1,2,3,4,5},







1,2,3



This means that the first stakeholder disregards

inputs other than the number of assistants and

associate professors engaged in the project and the

second stakeholder disregards among outputs the

number of patents and the number of scientific

degrees generated be the project.

Table 1: All possible inputs and outputs for 10 example projects.

Inputs and outputs names

Inputs and outputs values

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10

Inputs

budget [mln. EURO] 0.2 0.3 0.5 0.9 0.1 0.6 0.2 0.8 0.8 0.1

duration [months] 12 12 24 36 36 12 24 36 24 24

b of full professors 2 2 2 1 3 2 2 3 1 1

b of associate professors 5 5 6 8 2 3 3 6 4 5

b of assistant professors 3 8 2 4 6 9 2 4 2 3

Outputs

b of high quality publications 2 1 1 2 5 2 2 3 1 0

b of international conferences presentations 10 12 10 11 13 9 8 8 5 15

b of monographs 1 2 1 1 0 2 2 1 0 1

b of patents 0 0 1 1 0 1 1 2 0 0

b of scientific degrees and titles 2 1 2 3 1 1 2 2 1 3

Table 2: The results of the original DEA for the example projects.

Input and output weights

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10

Inputs

a1 0.606 0.909 0.000 0.000 0.145 0.000 0.545 0.000 0.000 1.173

a2 0.066 0.029 0.000 0.000 0.000 0.084 0.000 0.026 0.000 0.000

a3 0.000 0.000 0.000 1.003 0.000 0.000 0.000 0.397 1.356 0.575

a4 0.000 0.000 0.000 0.000 0.000 0.000 0.012 0.000 0.000 0.000

a5 0.000 0.000 2.030 0.000 0.000 0.000 0.031 0.044 1.162 0.000

Ouputs

b1 0.303 0.000 0.000 0.535 0.014 0.000 0.027 0.000 1.227 0.000

b2 0.000 0.015 0.406 0.000 0.000 0.050 0.000 0.000 0.323 0.116

b3 0.000 0.379 0.000 0.000 0.000 0.201 0.210 0.000 0.000 0.000

b4 0.000 0.000 0.812 0.267 0.000 1.205 0.172 1.499 0.000 0.016

b5 0.545 0.000 0.000 0.334 0.000 0.000 0.000 0.000 0.000 0.114

Objective function value 1.867 1.512 1.200 2.333 5.000 2.050 3.098 1.308 0.772 3.000

Table 3: The results of problem (5) for l=1.

Input and output weights

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10

Inputs

a1 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

a2 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

a3 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

a4 0.070 0.425 0.010 0.010 0.660 1.000 0.029 0.220 0.010 0.082

a5 0.871 0.010 1.000 0.985 0.010 0.010 0.053 0.169 1.000 0.368

Outputs

b1 0.186 0.000 0.000 0.000 1.000 0.000 0.023 0.000 0.318 0.000

b2 0.197 0.070 0.203 0.000 0.000 0.158 0.000 0.000 0.174 0.123

b3 0.000 0.368 0.000 0.000 0.000 0.499 0.257 0.000 0.000 0.000

b4 0.000 0.000 0.403 0.000 0.000 0.755 0.000 1.000 0.000 0.000

b5 0.000 0.000 0.000 1.000 0.000 0.000 0.000 0.000 0.000 0.000

Objective function value 0.792 0.714 1.183 0.746 3.623 1.029 2.909 1.000 0.583 1.216

Modiﬁed, Stakeholders Perspective based DEA Approach in IT and R&D Project Ranking

163

Table 4: The results of problem (5) for l=2.

Input and output weights

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10

Inputs

a1 0.000 1.000 0.000 0.000 1.000 0.000 0.168 0.000 0.000 1.000

a2 0.134 0.032 0.000 0.000 0.000 0.130 0.000 0.000 0.000 0.000

a3 0.000 0.000 0.000 1.000 0.000 0.000 0.000 0.390 1.000 0.343

a4 0.000 0.000 0.000 0.000 0.012 0.303 0.110 0.000 0.000 0.000

a5 0.336 0.000 1.000 0.000 0.000 0.000 0.194 0.660 0.849 0.016

Ouputs

b1 0.672 0.010 0.193 0.481 0.068 1.000 0.033 1.000 0.896 0.010

b2 0.269 0.015 0.199 0.046 0.010 0.010 0.010 0.010 0.236 0.090

b3 0.201 0.415 0.010 0.313 0.010 0.980 1.000 0.010 0.010 0.010

b4 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

b5 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

Objective function value 1.615 1.489 1.098 1.779 3.790 1.637 2.854 0.811 0.769 2.781

Table 5: Position of each project in the three rankings.

ranking P1 P2 P3 P4 P5 P6 P7 P8 P9 P10

Original DEA 6 7 9 4 1 5 2 3 8 10

Stakeholder 1 7 9 4 8 1 5 2 6 10 3

Stakeholder 2 5 7 8 6 1 4 2 9 10 3

Table 6: Position of each project in the final ranking on the basis of (6).

Ranking P1 P2 P3 P4 P5 P6 P7 P8 P9 P10

Value (6) 1,867 1,512 1,200 2,333 5,000 2,050 3,098 1,308 0,772 3,000

Final ranking 6 7 8 4 1 5 2 9 10 3

Table 3 and 4 contain the results of the application

of problem (5) for l=1 and 2.

If we considered just the view of the first

stakeholder (Table 3), we would have the following

ranking of the projects: P5, P7, P10, P3, P6, P8, P1,

P4, P2, P9, which is different than the one obtained

by means of the original DEA method (especially if

we consider e.g. project P10).

The view of the second stakeholder alone would

give still another ranking - P5, P7, P10, P4, P6, P1,

P2, P3, P8, P9 - which is shown in Table 4

In Table 5 the positions of each project in the three

rankings is shown. This may be a valuable

information, for example, we can see that some

projects have a very stable position (P5, P7, P6), but

in case of some projects the evaluation changes

considerably (P10, P8). The influence of the

stakeholders may be high.

In Table 6 the aggregated values (6) are shown.

This ranking shows the best situation of each project

according to the choice of weights and the

stakeholders and it might be a useful tool for the

project evaluation. The projects at the top of the final

ranking are certainly good in the eyes of some

stakeholders. For example, if we had used the original

DEA method, project P10 would have been rejected.

But this project seems to satisfy two key stakeholders.

Rejecting it would mean disregarding the key

stakeholders of our organisation, which is decisively

wrong.

6 CONLUSIONS

In this paper we propose a modification of the DEA

method which can be used to evaluate projects. The

core of our proposal is the possibility to include in the

DEA model the perspectives of various project

stakeholders. Project stakeholders, especially the key

ones, cannot be neglected in project evaluation and

selection. The original DEA method used for project

evaluation disregarded them.

The proposal, combined with project stakeholders

management methods and with careful project inputs

and outputs identification and evaluation, may be

useful in project selection problems. But of course

real world cases are needed to prove it.

REFERENCES

American Heritage Dictionaries, 2014. The American

Heritage Student Science Dictionary. Boston, MA:

Houghton Mifflin Harcourt Publishing Company.

Asosheh, A., Nalchigar, S., Jamporazmey, M., 2010.

Information technology project evaluation: An

ICEIS 2016 - 18th International Conference on Enterprise Information Systems

164

integrated data envelopment analysis and balanced

scorecard approach, Expert Systems with

Applications, vol. 37, no. 8, pp. 5931-5938.

Broom, M., Brady, B., Kecskes, Z., Kildea, S., 2013. World

Café Methodology engages stakeholders in designing a

Neonatal Intensive Care Unit, Journal of Neonatal

Nursing, vol. 19, no. 5, pp. 253-258.

Chan, A.P.C., Scott, D., Chan, A.P.L, 2004. Factors

affecting the success of a construction project. “Journal

of Construction Engineering and Management”, vol.

130 no. 1, pp. 153–155.

Chan, A.P.C., Scott, D., Lam, E.W.M., 2002. Framework

of success criteria for design/build projects, Journal of

Management in Engineering, vol. 18, no. 3, pp. 120-

128.

Charnes, A., Cooper, W.W., Rhodes, E., 1978. Measuring

efficiency of decision making units. Eur. J. Opl. Res. 2,

429444.

Davis, K., 2014. Different stakeholder groups and their

perceptions of project success, International Journal of

Project Management, vol. 32, no. 2, pp. 189-201.

Eilat, H., Golany, B., Shtub, A., 2008. R&D project

evaluation: An integrated DEA and balanced scorecard

approach, Omega, vol. 36, no. 5, pp. 895-912.

Frascati Manual: Proposed Standard Practice for Surveys

on Research and Experimental Development, 6th

edition, www.oecd.org/sti/frascatimanual, 6th edition,

2002, Pages: 266, ISBN 978-92-64-19903-9.

Geeson, N., Quaranta, G., Salvia, R., Brandt, J., 2015.

Long-term involvement of stakeholders in research

projects on desertification and land degradation: How

has their perception of the issues changed and what

strategies have emerged for combating desertification?,

Journal of Arid Environments, vol. 114, pp. 124-133.

Gusmao, A.P.H., Costa, A.P., 2012. Evaluation of it/is

outsourcing projects using the dea methodology. MCIS

2012 Proceedings. Paper 24.

http://aisel.aisnet.org/mcis2012/24.

Hartono, B., Sulistyo, S.R., Praftiwi, P.P. , Hasmoro, D.,

2014. Project risk: Theoretical concepts and

stakeholders' perspectives, International Journal of

Project Management, vol. 32, no. 3, pp. 400-411.

Missonier, S., Loufrani-Fedida, S., 2014. Stakeholder

analysis and engagement in projects: From stakeholder

relational perspective to stakeholder relational

ontology, International Journal of Project Management.

Project Management Institute (PMI), 2013. Managing

change in organizations: A practice guide. Newtown

Square, PA: Author.

Revilla, E., Sarkis, J., Modrego, A., 2003. Evaluating

performance of public - private research

collaborations: A DEA analysis, Journal of the

Operational Research Society, vol. 54, no. 2, pp. 165-

174.

Sowlati, T., Paradi, J.C., Suld, C., 2005. Information

systems project prioritization using data envelopment

analysis, Mathematical and Computer Modelling, vol.

41, no. 11-12, pp. 1279-1298.

Sudevan, S., Bhas, M., Pramod, K.V., 2014. A typology of

stakeholder identification methods for projects in

software industry, 14th Middle Eastern Simulation and

Modelling Multiconference, MESM 2014 - 4th

GAMEON-ARABIA Conference, GAMEON-

ARABIA 2014, pp. 5.

Wray, B., Mathieu, R., 2008. Evaluating the performance

of open source software projects using data

envelopment analysis, Information Management and

Computer Security, vol. 16, no. 5, pp. 449-462.

Yuan, B., Huang J.-N., 2002. Applying Data Envelopment

Analysis to Evaluate the Efficiency of R&D Projects —

A Case Study of R&D in Energy Technology. in:

Technology Commercialization. pp 111-134.

APPENDIX

The research was partially supported by the National

Science Center in Poland, grant no. 260084

"Research projects success and failure factors".

Modiﬁed, Stakeholders Perspective based DEA Approach in IT and R&D Project Ranking

165