Structuring Multicriteria Resource Allocation Models

A Framework to Assist Auditing Organizations

Vivian Vivas

1,2

and Mónica Duarte Oliveira

Centre for Management Studies of Instituto Superior Técnico (CEG-IST), Universidade de Lisboa, Lisbon, Portugal

Department of Planning, Budget and Finance, Comptroller General of the Union, Brasilia, Brazil

Keywords: Audit Planning, Multicriteria Evaluation, Portfolio Decision Analysis, Problem Structuring, Resource

Allocation.

Abstract: Multicriteria resource allocation models have been reported in the literature to support decision makers in

selecting options/projects/programmes. These models are particularly important in public contexts in which

resources are limited and there is an increasing demand for transparency and accountability in spending.

Despite the potential of these models to promote an effective use of scarce resources, there is little organized

and integrated research on how to structure them. In this paper we propose a framework with techniques and

tools to support the structuring of multicriteria resource allocation models, so that these models have a

potential to assist organizations in evaluating and selecting audit and control actions; and we provide

illustrative examples on to apply these techniques and tools in the context of the Comptroller General of the

Union, the Ministry of the Brazilian federal government responsible for helping the Brazilian president

regarding the treasury, federal public assets application and the government's transparency policies.

1 INTRODUCTION

Brazil is a large country that has in place

governmental programs that reach all its territory, and

in which the public spending of federal funds is

audited by the Ministry of Transparency, Supervision

and Comptroller General of the Union (CGU).

Similar to public auditing organizations in other

countries, the activities of the CGU integrate actions

of corruption prevention, fraud deterrence, public

accounting, comptroller, ombudsman activities and

increased transparency in management. In a time in

which the country is going through a severe economic

crisis, CGU has a key role in promoting transparency

and accountability in public spending.

Since resources are scarce, CGU public managers

must choose the set of projects to be executed with

the available budget, considering costs and expected

returns. This is a resource allocation situation well

recognized in literature and, in this context, the use of

multicriteria decision analysis concepts and tools can

become useful and necessary.

Several multicriteria models for resource

allocation have been reported in literature to support

decision-makers in managing portfolios, taking into

account of costs, benefits and risks (Liesiö et al.,

2007; Phillips and Bana e Costa, 2007; Lourenço et

al., 2012; Oliveira et al., 2012). However, there is

little indication in the decision sciences and

operational research literature on how to structure

such type of problems in an integrated and organized

manner (Montibeller et al., 2009). Proper structuring

is required for building models that can effectively

assist decision-makers.

This paper aims to fill this gap by proposing a

framework to structure multicriteria resource

allocation models (MRAM) in the context of auditing

organizations. Specifically, the framework defines

procedures and methods that can help to structure

MRAM with a potential to improve the internal

processes of organizations that have budget

constraints and perform audit and inspection actions,

such as in the CGU. The remainder of the paper is

structured as follows. The next section outlines

broadly the multicriteria resource allocation problem

and key approaches set out in the literature to address

those problems. Then we suggest a set of techniques

and tools for the structuring MRAM and provide

examples of its application for the auditing context.

The paper ends with discussion of some relevant

issues and directions for future research.

Vivas V. and Duarte Oliveira M.

Structuring Multicriteria Resource Allocation Models - A Framework to Assist Auditing Organizations.

DOI: 10.5220/0006189503210328

In Proceedings of the 6th International Conference on Operations Research and Enterprise Systems (ICORES 2017), pages 321-328

ISBN: 978-989-758-218-9

321

2 THE (CLASSICAL) RESOURCE

ALLOCATION PROBLEM

2.1 General Definition

The multicriteria resource allocation problem is

characterized by the selection of attractive projects

(portfolio) to be financed under the presence of a

limited budget and of other relevant constraints. So,

the prioritization and/or selection of options aims at

generating portfolios of projects – which entail

multiple benefits, costs and uncertainties – that offer

the best overall value for a given budget. Clearly, the

analyses of portfolios will depend on how the

organization’s decision-makers values distinct

project benefits and risks, as well as on the costs

required by those projects and by context constraints.

As these benefits are usually multi-dimensional (e.g.,

losses recovery, strategic fit, social responsibility,

safety etc.), this is a multicriteria problem.

The multicriteria resource allocation literature

suggests two main modelling approaches that can

inform the prioritization and/or the selection of

projects and that can be used by the CGU: the

optimization approach (Bana e Costa and Soares,

2004; Liesiö et al., 2007; Lourenço et al., 2012;

Oliveira et al., 2012) and the prioritization approach

(Bana e Costa et al., 2006; Phillips and Bana e Costa,

2007), which we now briefly describe.

2.2 The OPTIMIZATION Approach

Following Oliveira et al. (2012), the performance 



of each project  in the benefit criterion  can be

measured by a level in the respective descriptor, with

partial value 



(



). Under an additive structure

(which requires the respect for mutual independence

conditions), the value of the overall benefit 



of the

project , with 



represent the weight assigned to

criterion , can be determined as:









,…,



=





















=1and



> 0( = 1,…,)





(1)

Considering each project  has 



>0and cost 



 is the total of available resources, and as 



=1, if

the project  is included in the best portfolio and 

otherwise, we have:

: 











(2)

 : 











≤,





∈



0,1



, = 1,…,.

(3)

The best project portfolio will be found by solving

this optimization problem. Additional constraints can

be considered.

2.3 The PRIORITIZATION Approach

Following Bana e Costa et al. (2006), the

prioritization approach can be applied in six steps, in

which the first three steps are similar to the

optimization approach but also necessary: 1. List the

projects; 2. Use a multicriteria value model, as

Equation (1), for instance, to determine the added

expected benefit 



, if the project  is financed; 3.

Define the cost 



of each project, equal to the amount

of financial support funding; 4. Calculate the benefit-

to-cost ratio (



=



/



) of each project; 5. Rank the

projects from the highest to the lowest benefit-to-cost

ratio; and 6. Go down the list, choosing projects until

the available budget is depleted.

A variant of this prioritization approach is found

in Phillips and Bana e Costa (2007), that use the

Equity, a software for portfolio analysis, which

enables a classification of projects within an

organizational structure logic. Specifically, the funds

can be spent on different levels in various

organizational units or functions, called areas. In each

of the areas , the options are evaluated based on

criteria of benefits and risks , resulting in ×

scales. For a given criteria  is assigned a within

criteria weight 



. The total value of each option

and the benefit-cost ratios are:





=

∑





.

()

.



∑∑





.



(4)













(5)

The options are ranked from highest to lowest

ratio 



. The Equity structure can also be used within

an optimization approach, although requiring a more

sophisticated optimization model.

Several decision support tools assist the

implementation of both approaches, being that the

case of PROBE - Portfolio Robustness Evaluation

(Lourenço et al., 2012), RPM - Robust Portfolio

Modelling (Liesiö et al., 2007, 2008; Vilkkumaa et

al., 2014) and the resource allocation module of M-

MACBETH (Bana e Costa et al., 2012; Hummel et

al., 2017).

ICORES 2017 - 6th International Conference on Operations Research and Enterprise Systems

322

2.4 Auditing Context

Both the prioritization and optimization modelling

approaches can be useful for assisting decision-

making processes of auditing organizations, as

directly or indirectly shown by distinct studies:

Bradbury and Rouse (2002) point out that the audit

risk assessment is an essential part of the audit

planning process. As the authors explain, numerical

risk scores for each audit unit, together with

materiality, can be used as the basis for the audit

resource allocation. In turn, some studies have

presented models to allocate internal auditing time

and others auditing resources to projects (Krüger and

Hattingh, 2006; Mohamed, 2015), using the

optimization approach.

Prior to the use of these models, one needs to

structure the multicriteria resource allocation model.

I.e., to build such a model it is necessary to get all the

information pertaining on models, which means

defining the organizational areas, audit units, project

options, costs, measurement criteria of benefits, risks,

synergies and interdependencies between projects

and other necessary factors (Friend and Hickling,

2005; Keeney, 1992; Montibeller et al., 2009), as well

as to understand who should participate in model

construction and whom the model is expected to

assist. Such structuring will show whether an

optimization or a prioritization approaches should be

used, and whether these approaches need further

development (note this is not the focus of this article).

3 STRUCTURING RESOURCE

ALLOCATION DECISION

MODELS

We herein propose a framework with techniques and

tools to help defining and structuring MRAM to assist

auditing organizations. Departing from the work

presented by Belton and Stewart (2002), the proposed

framework, shown in Figure 1, is able to generate

background information to build MRAM. Note that

applying the propose framework will require the use

of technical tools and concepts, as well to involve

decision-makers into participatory processes (for

instance, to build a multicriteria value model), i.e., the

adoption of a socio-technical process (Phillips and

Bana e Costa, 2007). In this article, we focus on the

techniques, rather than on the social process.

Each stage of the framework must generate

relevant information to building the model in a

structured way. The choice of which tools to use

depends on the context of the problem being

addressed, on which tools best fit the organizational

culture, and on the user's familiarity with those tools.

Figure 1: Framework to assist the structuring resource

allocation models.

3.1 Problem Identification

The first step is to identify the type of decision

problem and understand the different perceptions of

the actors relevant for the decision. Auditing

organizations commonly need to choose the control

actions to be performed by audit teams, taking into

account the audit risks and available resources. Is this

a prioritization problem? Is this a ranking problem? Is

this about project selection with budget constraints?

Or, moreover, does project selection involve possible

conflicts of interest? The identification of the decision

problem type is a key factor for MRAM.

In this step we suggest the use of structuring tools

for problem definition, such as those cited by Franco

and Montibeller (2011): cognitive mapping, dialog

mapping, Soft Systems Methodology (SSM), group

model building.

As explained by Eden (2004), a cognitive map is

a graphical representation of thoughts in a network

shape containing nodes and arrows whose direction

implies causality. It is a powerful tool to capture

different aspects of the problem to be addressed and

is helpful to clarify people’s ideas and perceptions.

Another tool is Dialog Mapping that seeks to build

common understanding for wicked problems, which

are ill structured and complex and can lead to

different views and solutions depending on different

stakeholders’ perceptions. A diagram or map is

shown in a shared display with use of a conversational

grammar called IBIS, Issue Based Information

System, that represents the moves in a conversation

as questions, ideas (possible answers to the question),

and arguments (pros and cons to the ideas) (Conklin,

2006).

Soft systems methodology (SSM) is an approach

for dealing with problematical messy situations. Its

Structuring Multicriteria Resource Allocation Models - A Framework to Assist Auditing Organizations

323

Figure 2: Mapping key concerns for developing an inspection program cycle with a means-ends objectives network.

use is recommended when divergent views on the

problem definition exist. It is an action-oriented

process of investigation in which users learn their

way from finding out about the situation and what can

be done to improve it (Checkland and Poulter, 2010).

In turn, a Group Model Building is a data analysis

method from a group of decision makers. The

dynamic patterns and relationships between key

factors discussed by the group are portrayed to talk

and analyse, resulting in new insights and possible

new strategies or scenarios (Richardson and

Andersen, 1995).

In addition, Friend and Hickling (2005) have

presented the Strategic Choice Approach (SCA) that

is useful to support the creation and definition of the

problem in uncertain contexts.

Following Keeney's (1992) guidelines, one can

also frame a decision situation by structuring the

strategic, fundamental and mean objectives through

means-ends relationships. Giving an example on

auditing context, CGU performing an inspection

program in states and municipalities, in order to

assess the expenses incurred by these entities

involving federal funds. The scope and entities to be

inspected are chosen based on indicators divided into

four dimensions: Control, Transparency, Economic

and Social Development and Materiality.

Figure 2 illustrates the means-ends network for

the CGU problem described. The main objective of

an inspection cycle is to define the control actions

(projects) that will be performed, within the available

resources, which means defining auditing scope,

auditees and measure expected returns/impacts. The

map highlights key issues of the decision problem,

namely the value system organized in a means-ends

network. In fact, visual tools are useful to define and

clarify the problem may be relevant in this step.

Once the problem is defined, as Franco and

Montibeller (2011) well emphasized, it is necessary

identify which aspects or particular decisional

element of the decision problem will be evaluated in

the model to be built. However, before that, we need

to identify the key actors involved in the process.

3.2 Stakeholders Identification

The next step seeks to identify the key stakeholders

and analyse their power and influence on the decision

context. Bryson (2004) presents an array of

techniques useful for stakeholders’ identification and

analysis and which grouped into four categories,

which should be used in this step: organizing

participation; creating ideas for strategic

interventions; building a winning coalition around

proposal development, review and adoption; and

implementing, monitoring and evaluating strategic

interventions. The author highlights five stakeholder

identification and analysis techniques to helping

organize participation: a process for choosing

stakeholder analysis participants; the basic

stakeholder analysis technique; power versus interest

grids; stakeholder influence diagrams; and the

participation planning matrix. He lists six additional

techniques to creating ideas for strategic

interventions: bases of power and directions of

interest diagrams; finding the common good and the

structure of a winning argument; tapping individual

stakeholder interests to pursue the common good;

stakeholder-issue interrelationship diagrams;

problem-frame stakeholder maps; and ethical

ICORES 2017 - 6th International Conference on Operations Research and Enterprise Systems

324

analysis grids. The author also considers three

techniques for proposal development review and

adoption: stakeholder support versus opposition

grids, stakeholder role plays and policy attractiveness

versus stakeholder capability grids. And, finally,

presents policy implementation strategy development

grid for the last category.

From these techniques, we can highlight grouping

the stakeholders in the matrix power/interest,

proposed by Mendelow (1981), in which is possible

to perceive how communication and relationships

between stakeholders can affect the model structure

and its implementation. Figure 3, for instance, helps

to understand differences in power and influence of

key stakeholders in the CGU inspection program.

Figure 3: Power-interest matrix applied to an inspection

action.

Ferretti (2016) shows that, under the existence of

a plurality point of views, one needs to understand

these differences, which requires the framework steps

that follow.

3.3 Goals and Values Identification

Once the problem and the stakeholders are identified,

one needs to have an understanding of the goals and

values of the stakeholder(s). We can underline the

concept of decision framing presented by Keeney

(1992) which points out that values are used for

evaluation and should reflect the decision-makers

objectives. He highlights that there are two distinct

types of objectives, the fundamental objectives and

the mean objectives. While the former features an

essential reason for the interest in the decision

situation, the mean objectives are just a way to

achieve them. As the author also emphasizes,

structure objectives give the basis for any use of

quantitative modelling and the fundamental

objectives hierarchy can indicate the set of objectives

over which attributes should be defined.

A structuring tool widely used in decision analysis

is the value tree, which displays the family of key-

concerns in a tree form and offers a useful visual

overview of the main objectives in different levels of

increasing specification (Bana e Costa, 2001; Bana e

Costa et al., 2004). In Figure 4 we present a value tree

with the fundamental objectives to be attained with an

inspection action. For instance, the “Management

Continual Improvement” objective is concerned with

the assessment of the inspection program's objective

component in terms of efficiency and technical

quality as well as the agreement on the entities and

the areas to be audited.

Figure 4: Value tree for an inspection action built with M-

MACBETH.

At this stage, it is also important to look for the

alternative’s costs and identify the measurement

criteria of alternative performances (expected

benefits). One can make use of the framework for

structuring options and areas and criteria presented in

Montibeller et al. (2009). The authors propose two

approaches to structuring criteria, based on Keeney’s

concepts: Alternative-focused thinking (AFT), which

criteria are defined from the characteristics that

distinguish options and Value-focused thinking

(VFT), where the evaluation criteria should reflect the

organization’s values and strategic objectives.

3.4 Alternatives Identification

The identification of decision alternatives, which in

auditing context means the identification of audit

projects that will be evaluated, is an important step in

the structuring process and can be performed through

different techniques / tools.

In organizations segregated by pre-defined areas,

where the initial set of project options is relatively

stable, it can be used the AFT above described, in

which, after problem definition, the projects are

identified and, then, the values (criteria) to consider

in the evaluation are specified. In turn, on the VFT,

organizational values and goals are initially set. The

options are then created thinking on how to achieve

these goals (Keeney, 1992).

Another useful tool presented by Howard (1988)

is the strategy-generation table. It shows how a total

Structuring Multicriteria Resource Allocation Models - A Framework to Assist Auditing Organizations

325

strategy can be specified by combination of options

under several dimensions, called strategy themes.

In turn, one may apply analysis of interconnected

decision areas (AIDA) technique, present in Strategic

Choice Approach (Friend and Hickling, 2005), that

allows visualization of the compatibilities and

incompatibilities of options within a problem focus.

One can still make use of cognitive map to

explore/identify decision alternatives (Eden, 2004).

Montibeller and Belton (2006) proposed the causal

map, which can also be used to identify and agree to

a set of potential strategic options. As the authors

highlight, a causal map is a network of inter-linked

concepts (ideas) which tries to represent the discourse

of a person through means-ends structure, whereby

decision options are means of achieving the decision-

makers’ goals.

In the CGU inspection case, since the projects to

be evaluated depend on the definition of the federal

state to be inspected and the audit scope, we can map

the set of options surrounding the inspection program

to gain a better understanding of the issues, their

interrelations and perceived implications to the model

to be built.

3.5 Uncertainties Identification

An analysis of which uncertainties are key for the

evaluation of options and for the allocation of

resources is required. To exemplify, uncertainties

may be related with the budget, with the measurement

of options performance and with the

importance/weight of objectives.

Vilkkumaa et al. (2014) make a Bayesian

modelling of uncertainties, to be considered in the

selection of project portfolios. There is still another

classification in Strategic Choice Approach to

identify the uncertainties relating to the working

environment, related to the guiding values and related

to the choices in related agendas (Friend and

Hickling, 2005). Thus, different uncertainty types

may require different analysed with the prioritization

and/or optimization modelling approaches.

In the auditing context, as highlighted by Krüger

and Hattingh (2006, p.62), we can mention that “risk

is seen as a measure of uncertainty and is linked to

the possible loss in an audit area — uncertainty in

achievement of business objectives. The possible loss

in an audit area will depend on specific

characteristics and these characteristics are termed

audit risk factors. Examples of well known and

frequently used risk factors include complexity of

operations, financial implications, recent changes,

time since last audit, etc.” – these issues should be

discussed for each context and have naturally an

impact on the MRAM to be developed.

In the CGU example, a relevant audit risk factor

to be considered in the model may be related to the

uncertainty in estimate the project (control actions)

values to be included in the inspection program

portfolio.

3.6 Constraints Identification

It is also necessary to identify constraints that may be

relevant for the allocation of scarce resources to

competing projects. For instance, there may be

resources/budget restrictions, synergies between

projects or interdependencies between projects.

At this stage, in a brainstorm session/focus group,

one can use VFT to elicit the main constraints

involved in the decision problem by equations

(Keeney, 1992). AIDA can also help with Option

Bars that bring the incompatibilities that can be

translated into equations to be added to the value

model used (Friend and Hickling, 2005).

In the CGU case, it is important to consider the

following constraints:

Budgetary. Identify financial cost of each control

action and prioritize projects within the available

budget, so as to be accounted for in equation (3).

Logistical. The distribution of teams available for

each control action needs to be accounted for (e.g.,

equipment, vehicles, and special displacements).

Whereas ℎ



the amount of resources  consumed by

the project  and 



the total available resources . It

has been:

ℎ







≤







(6)

Context. Projects of entities identified as

vulnerable should be positively discriminated. Be the

corresponding project to the federal entity identified

as vulnerable, one should have:





(7)

3.7 Interactions between the Stages

To complete the structuring process, one cannot apply

the framework without considering the joint analysis

of different framework stages, as these are key to

select and/or develop MRAM. Table 1 summarises

techniques and tools included in the proposed

framework. The diagonal includes techniques and

tools previously described, and the remainder cells

provide tools that can assist more complex analyses.

ICORES 2017 - 6th International Conference on Operations Research and Enterprise Systems

326

Table 1: Selection of techniques and tools that can assist structuring (crossing framework stages).

Stakeholders Goals and Values Alternatives Uncertainties Constraints

Stakeholders

Stakeholder Power-

interest Grid

Stakeholder Visualization

Influence Map

Negotiation Analysis

Drama Theory

Conflict

Dissolution

Drama Theory

Goals and

Values

Negotiation Analysis

Drama Theory

Value Tree

Decision framing

Fundamental Objectives

Hierarchy

Value Tree

Causal Map and

MCDA

DSS PROBE

RPM

Alternatives

Conflict Dissolution

Drama Theory

Value Tree

Causal Map and

Multicriteria Decision

Analysis (MCDA)

Cognitive Map

SCA

AFT, VFT

Strategy Table

AIDA in SCA

RPM

Uncertainties

DSS PROBE

RPM

Bayesian modelling

SCA

Risk Factor Analysis

Constraints

AIDA in SCA

RPM

Brainstorm

Focus group

VFT

AIDA in SCA

For instance, different stakeholders (single,

multiple, group) can lead to different goals and values

and can generate different sets of alternatives and

criteria.

In this situation, it may be useful to apply conflicts

dissolution modelling techniques to have an

understanding for possible win-win solutions, which

are often used for evaluation models but can be

adapted to the structuring context. (Bana e Costa et

al., 2001; Edwards et al., 2007)

As implications for resource allocation models,

we can cite:

 Multiply stakeholders: Preparation of a

cognitive map to every stakeholder, analysis of

common and divergent characteristics.

Conducting focus group/brainstorming sessions

for the preparation of an aggregated map (Ferretti,

2016). The use of bargain negotiation/drama

theory (Edwards et al., 2007; Rosenhead and

Mingers, 2001) can be useful.

 Group of stakeholders: The necessity for using

techniques conflict dissolution in brainstorming

session/focus group (Bana e Costa, 2001; Bana e

Costa et al., 2001; Salo, 1995).

Regarding uncertainties, it may be related to the

objectives and values, since the weights of the criteria

might influence the project consequences - in this

case robustness analysis and impact measurement can

be used.

Thus, the result to be presented will be determined

by the whole process and possibly different MRAM

may emerge. Therefore, the modelling approaches

presented in Section 2 may need to be enhanced and

developed for the context.

4 DISCUSSION

This paper combined decision making techniques and

tools to support the structuring of multicriteria

resource allocation models for the auditing context, in

an attempt to aid stakeholders involved in the auditing

decisions and which are pressured and charged for

transparency and accountability in public spending.

The application of the framework requires

thinking about which decision-makers and

stakeholders should be directly involved in each

framework stage, together with a

facilitator/consultant, an analyst, an recorder and/or

others necessary roles in the process (Richardson and

Andersen, 1995). This is necessary so that decision-

makers will have confidence in MRAM results.

For future research, it is relevant: to extend the

concepts and techniques to be used in the distinct

framework stages; to systematically apply the

framework for well-defined decisions at CGU and in

other real-world auditing contexts; and, to measure

the added value of using the framework.

Structuring Multicriteria Resource Allocation Models - A Framework to Assist Auditing Organizations

327

REFERENCES

Bana e Costa, C.A., 2001. The use of multi-criteria decision

analysis to support the search for less conflicting policy

options in a multi-actor context: Case study. Journal of

Multi-Criteria Decision Analysis. 10(2), 111–125.

Bana e Costa, C.A., Antão da Silva, P., Nunes Correia, F.,

2004. Multicriteria Evaluation of Flood Control

Measures: The Case of Ribeira do Livramento. Water

Resources Management. 18(3), 263–283.

Bana e Costa, C.A., De Corte, J.-M. and Vansnick, J.-C.,

2012. MACBETH. International Journal of

Information Tech. & Decision Making. 11(2), 359–387.

Bana e Costa, C.A., Fernandes, T.G., Correia, P.V.D., 2006.

Prioritisation of public investments in social

infrastructures using multicriteria value analysis and

decision conferencing: A case study. International

Transactions in Operational Research. 13(4), 279–297.

Bana e Costa, C.A., Nunes Da Silva, F., Vansnick, J.-C.,

2001. Conflict dissolution in the public sector: A case-

study. European Journal of Operational Research.

130(2), 388–401.

Bana e Costa, C.A., Soares, J.O., 2004. A multicriteria

model for portfolio management. The European

Journal of Finance. 10, 198–211.

Belton, V., Stewart, T., 2002. Multiple Criteria Decision

Analysis: An Integrated Approach. Kluwer: Dordrecht.

Bradbury, M.E., Rouse, P., 2002. An Application of Data

Envelopment Analysis to the Evaluation of Audit Risk.

Abacus. 38(2), 263–279.

Bryson, J.M., 2004. Stakeholder Identification and

Analysis Techniques. Public Management Review.

6(1), 21–53.

Checkland, P., Poulter, J., 2010. Soft Systems

Methodology. In: Reynolds, M., Holwell, S. (Eds.),

Systems Approaches to Managing Change: A Practical

Guide. Springer-Verlag, London, 191–242.

Conklin, J., 2006. Dialogue Mapping: Building Shared

Understanding of Wicked Problems. Wiley.

Eden, C., 2004. Analyzing cognitive maps to help structure

issues or problems. European Journal of Operational

Research. 159(3), 673–686.

Edwards, W., Miles Jr., R.F., von Winterfeldt, D., 2007.

Advances in decision analysis: From foundations to

applications. New York: Cambrigde University Press.

Ferretti, V., 2016. From stakeholders analysis to cognitive

mapping and Multi-Attribute Value Theory: An

integrated approach for policy support. European

Journal of Operational Research. 253(2), 524–541.

Franco, L.A., Montibeller, G., 2011. Problem Structuring

for Multicriteria Decision Analysis Interventions. In:

Cochran et al. (Eds.) Wiley Encyclopedia of Operations

Research and Management Science. Wiley, USA.

Friend, J., Hickling, A., 2005. Planning Under Pressure:

The Strategic Choice Approach. Third. ed. Elsevier

Butterworth-Heinemann.

Howard, R.A., 1988. Decision Analysis: Practice And

Promise. Management Science. 34(6), 679–695.

Hummel, M.J., Oliveira, M.D., Bana e Costa, C.A.,

Ijzerman, M.J., 2017. Supporting the project portfolio

selection decision of research and development

investments by means of multi-criteria resource

allocation modelling. In: Marsh, K., Goetghebeur, M.,

Thokala, P., Baltussen, R. (Eds.) Multi-Criteria

Decision Analysis to Support Healthcare Decisions.

Springer.

Keeney, R.L., 1992. Value-focused thinking: A Path to

Creative Decisionmaking. Harvard University Press.

Krüger, H.A., Hattingh, J.M., 2006. A combined AHP-GP

model to allocate internal auditing time to projects.

ORiON. 22(1), 59–76.

Liesiö, J., Mild, P., Salo, A., 2007. Preference programming

for robust portfolio modeling and project selection.

European Journal of Oper Res. 181(3), 1488–1505.

Liesiö, J., Mild, P., Salo, A., 2008. Robust portfolio

modeling with incomplete cost information and project

interdependencies. European Journal of Operational

Research. 190(3), 679–695.

Lourenço, J.C., Morton, A., Bana e Costa, C.A., 2012.

PROBE - A multicriteria decision support system for

portfolio robustness evaluation. Decision Support Sys-

tems, 54(1), 534–550.

Mendelow, A.L., 1981. Environmental Scanning - The

Impact of the Stakeholder Concept. International

Conference on Information Systems. 407–417.

Mohamed, A.M., 2015. Operations Research Applications

in Audit Planning and Scheduling. International

Journal of Social, Behavioral, Educational, Economic,

Business and Industrial Engineering. 9(6), 2026–2034.

Montibeller, G., Belton, V., 2006. Causal maps and the

evaluation of decision options - a review. Journal of the

Operational Research Society. 57(7), 779–791.

Montibeller, G., Franco, L.A., Lord, E., Iglesias, A., 2009.

Structuring resource allocation decisions: A framework

for building multi-criteria portfolio models with area-

grouped options. European Journal of Operational

Research. 199(3), 846–856.

Oliveira, M.D., Rodrigues, T.C., Bana e Costa, C.A., Brito

de Sá, A., 2012. Prioritizing health care interventions:

A multicriteria resource allocation model to inform the

choice of community care programmes. In: Tanfani, E.,

Testi, A. (Eds.), Advanced Decision Making Methods

applied to Health Care. Springer, 141-154.

Phillips, L.D., Bana e Costa, C.A., 2007. Transparent

prioritisation, budgeting and resource allocation with

multi-criteria decision analysis and decision

conferencing. Annals of Oper. Research. 154, 51–68.

Richardson, G.P., Andersen, D.F., 1995. Teamwork in

group model building. System Dynamics Review. 11(2),

113–137.

Rosenhead, J., Mingers, J.(Eds.), 2001. Rational analysis

for a problematic world revisited: problem structuring

methods for complexity, uncertanty and conflict. Wiley.

Salo, A., 1995. Interactive decision aiding for group

decision support. European Journal of Operational

Research. 84, 134–149.

Vilkkumaa, E., Liesiö, J., Salo, A., 2014. Optimal strategies

for selecting project portfolios using uncertain value

estimates. European Journal of Operational Research.

233(3), 772–783.

ICORES 2017 - 6th International Conference on Operations Research and Enterprise Systems

328