Assessment of Relative Technical Efficiency of Small Mental Health

Areas in Bizkaia (Basque Country, Spain)

Nerea Almeda

, Carlos García-Alonso

, José Alberto Salinas-Pérez

, Mencía R. Gutiérrez-Colosía

and Luis Salvador-Carulla

Universidad Loyola Andalucía, Department of Psychology, Seville, Spain

Universidad Loyola Andalucía, Department of Quantitative Methods, Seville, Spain

University of Sydney, Mental Health Policy Unit, Brain & Mind Research Institute, Sydney, Australia

Keywords: Relative Technical Efficiency, Monte-Carlo DEA, Simulation, Data Envelopment Analysis, Expert

Knowledge, Decision Support Systems, Operation Research in Health, Small Mental Health Areas.

Abstract: Mental disorders cause an enormous burden to society. Considering the current economic context, an efficient

use of scarce inputs, with an appropriate outcome production, is crucial. This situation defines a classical

Relative Technical Efficiency (RTE) problem. A well-known methodology to assess RTE is the Data

Envelopment Analysis, although it presents some limitations. These may be overcome through a hybrid

strategy that integrates Monte-Carlo simulation and artificial intelligence. This study aims to (1) design of a

Decision Support System for the assessment of RTE of Small Mental Health Areas based on DEA; and (2)

analyse 19 mental health areas of the Bizkaian Healthcare System (Spain) to classify them and to identify

potential management improvements. The results have showed higher global RTE in the output-oriented

orientation than in the input-oriented one. This suggests that a decision strategy based on improving the input

management, within the ranges of the expert-driven model of community healthcare, could be appropriate. A

future research line will focus our attention on the validation process through the analysis of micro-

management interventions and their potential impacts in the real system.

1 INTRODUCTION

The current high levels of mental disorders

prevalence cause an enormous burden to the society

and a devastating impact on health and economy

(WHO, 2003). The factors involved in the

development of these psychopathologies are not only

individual features; social, economic and political

determinants, such as national policies and

community support, have also a relevant influence in

the manifestation of the symptomatology (WHO,

2016).Unfortunately, in high-income countries, 35%-

50% of people who suffer mental disorders do not

receive any treatment; in middle and low-income

countries, this percentage increases till 76%-85%

(WHO, 2016).

To face this problem, the World Health

Organization (WHO) and United Nations (UN) are

carrying out specific macro-level strategies. Firstly,

the WHO designed a ‘Mental Health Action Plan

2013-2020’ (WHO, 2013), in which was emphasized

the importance of assessing the evidence and

developing a deeper research. In addition, this action

plan highlighted the provision of health and social

care from a community-based perspective. On the

other hand, the UN is also supporting the shifting of

mental health treatments from hospital to community-

based care (United Nations, 1991). The community-

based mental health care is focused on caring for

individuals with mental illness from institutional

environments to the community (Moran & Jacobs,

2013; Shen & Snowden, 2014). This paradigm of

intervention presents better outcomes and is more

cost-effective than institution-based care (Gutierrez‐

Recacha, Chisholm, Haro, Salvador‐Carulla &

Ayuso‐Mateos, 2006; WHO, 2005;). According to

this model, an increase in outpatient and day care

services and a decrease in inpatient services is

expected. Therefore, the integration of care and

treatment in general hospitals and primary care as

well as the collaboration between professionals and

informal care providers is fundamental.

In Spain, both the Mental Health Strategy of the

Spanish National Health System (Ministerio de

Almeda N., GarcÃ a-Alonso C., Alberto Salinas-PÃl’rez J., R. GutiÃl’rrez-ColosÃ a M. and Salvador-Carulla L.

Assessment of Relative Technical Efﬁciency of Small Mental Health Areas in Bizkaia (Basque Country, Spain).

DOI: 10.5220/0006195603770384

In Proceedings of the 6th International Conference on Operations Research and Enterprise Systems (ICORES 2017), pages 377-384

ISBN: 978-989-758-218-9

377

Sanidad, Política e Igualdad, 2011) and the ‘Strategy

for tackling the challenge of chronic illness in the

Basque Country’ (Gobierno Vasco, 2011) are now

being developed, among others, for handling

chronical diseases. The main goals of these strategies

and policies are the promotion of mental health, the

provision of care, the enhancement of the recovery

and the reduction of morbidity and disability.

Regarding economics issues, even though a

political responsiveness to burden of mental disorders

is consolidated, the amount of resources destined to

mental health care depends on the “health” of the

economy (Shen & Snowden, 2014). Public mental

health services are highly vulnerable to resources

constraints in compromising economic situations

(Shen & Snowden, 2014). In Spain (2010), according

to the Organization for Economic Cooperation and

Development (OECD), 9% of the Gross Domestic

Product (GPD) was destined to expenditures on

health (OECD, 2010). Mental disorders absorbed an

expense of 46 billion euros (Parés-Badel el al., 2014).

Taking into account the relevance their increasing

prevalence and the involved amount of public

resources, always scarce, an efficient mental health

care system is absolutely crucial in the present

economic situation.

Research and empirical evidences are decisive

elements for designing suitable mental health policies

and, in consequence, improving quality of care.

Identification and assessment of potential

improvements in the system can be used in designing

of new strategies for enhancing efficiency scores of

mental health services in real contexts. Policy

makers’ decisions are usually based on their clinic

experience and thus the decisional risk is pretty high

taking into account the high uncertainty level: inner

and systemic. In the current economic situation, risks

could be reduced using Decision Support Systems.

These tools can help decision makers to have a better

understanding of mental health services performance

in a real, dynamic and uncertain context. The lack of

previous information and empirical evidence about

the potential trade-offs (costs and outcomes) between

different policy options, severely affects the selection

of the most “suitable” decision in a specific

management situation: the “what could or should

happen if… problem”.

The maintenance of the essential balance between

the quality of public mental health services and their

financial sustainability is the next challenge. This not

necessary means that a mental health system should

maximize its outcomes while maintaining the amount

of the consumed resources or, sometimes even worst,

reducing them. The key question is the optimization

of the balance between inputs and outputs in a

complex, interrelated and dynamic system under

uncertainty. Sherman (1984) introduced Data

Envelopment Analysis (DEA) for assessing hospital

Relative Technical Efficiency (RTE). Nowadays,

there is a growing interest in the evaluation of RTE in

health systems (Färe, Grosskopf, Lundström & Roos,

2008; Hollingsworth, 2008; Hollingsworth & Parkin,

2001; Kaya & Cafrı, 2015; Pelone et al., 2012;), but

little is known about it in mental health (Torres-

Jiménez et al., 2015; Tyler, Ozcan & Wogen, 1995;).

Although DEA models have been successfully

applied in health, several relevant drawbacks of this

analysis have been identified in the literature

(Salvador-Carulla et al., 2007; Zhu, 2013): (i)

frequently decision makers have difficulties in

interpreting DEA results, (ii) DEA models are not

appropriate for analysing datasets with low number of

decision making units (observations) and high

number of inputs (usually resources) and outputs

(outcomes of the system), (iii) the management of the

inner uncertainty of the real systems is statistically

complicated (Monte-Carlo simulation) and very

computer demanding and, finally, (iv) real data values

(inputs and outputs) have to be interpreted according

to expert knowledge for avoiding biased results (this

process needs to formalise explicit knowledge in a

knowledge-base).

The main goals of the current research are: 1. The

design of a Decision Support System for the

assessment of RTE of Small Mental Health Areas

based on DEA; and 2. The analysis of 19 mental

health areas of the Bizkaia (Spain) Healthcare System

for identifying potential performance improvements.

2 METHODS

2.1 Inputs, Outputs and

Decision-making Units

Original data were collected from “Mental Health

Atlas of Bizkaia” (Pereira, Gutiérrez-Colosía &

Salinas-Pérez, 2013). In total, the dataset included 52

variables, 39 inputs and 13 outputs, which described

the Mental Health Care System in Bizkaia (Spain).

This system is structured in 19 Small Health Care

Areas that were identified as Decision Making Units

(DMU) (19×52 data matrix). The variables were

coded into main types of care (Table 1) according to

the DESDE-LTC codification system (Salvador-

Carulla et al., 2011) and each code was classified

based on the Basic-Mental Health Community Care

(B-MHCC) paradigm (Salvador-Carulla et al., 2007)

ICORES 2017 - 6th International Conference on Operations Research and Enterprise Systems

378

in the following variable groups: service availability,

amount of places or beds, amount of professionals

and service utilization. Variable values were

transformed into rates per 100,000 population. Health

planers and policymakers validated the variable

(inputs and outputs) set (Table 1).

In order to assess the RTE of the selected areas,

15 scenarios were designed. Each scenario is a set of

variables (a meaningful combination of inputs and

outputs), which describes a specific type of care or a

combination of them selected by experts in mental

health care (Table 2). Thus, these scenarios allow to

study different perspectives of the management and

evolution of the system. By integrating the proposed

scenarios, the Decision Support System offers both

the RTE for any area in each scenario as well as the

global RTE of the system.

Table 1: List of inputs/outputs analyzed for each group of

main types of care (DESDE-LTC) and scenario assigned.

Table 2: Description of the scenarios.

The variables in each scenario were selected by

applying two criterions:

1. Methodological: For developing highly

discriminating DEA models (Alirezaee, Howland &

VandePanne, 1998; Dyson et al., 2001; Staat, 2001;

Torres et al., 2015) the number of variables have to

be controlled (2×(I×O)≤DMU, being I the number of

inputs, O the number of outputs and DMU the original

number of observations, 19 in this case).

2. Technical: All of the scenarios have to be

meaningful for managers and policy makers.

According to this principle, results obtained are easy

to interpret and facilitate the identification of

potential improvements that can be used to design

new real interventions and policies.

For better understanding the inner uncertainty of

the system, each variable value was transformed in a

standard statistical distribution (symmetric triangular

T[minimum value, central estimator, maximum

value] in this case). Therefore, the original 19×52

data matrix was transformed in a 19×52 statistical

distributions matrix. The structure and parameters of

these statistical distributions were selected by a panel

of experts including managers and policy makers

(Torres et al., 2015).

2.2 The Monte Carlo DEA Model and

the Decision Support System (DSS)

A hybrid model was used to assess the RTE of the

small mental health care areas in Bizkaia. This model

integrates classical statistics, mathematical

programming and an approximation to artificial

intelligence. Regarding classical statistics, Monte-

Carlo simulation was used: (1) to incorporate

Assessment of Relative Technical Efﬁciency of Small Mental Health Areas in Bizkaia (Basque Country, Spain)

379

uncertainty in variable measuring by using statistical

distributions rather than the original variable values

(i.e. the original value 0.299 was transformed into

triangular distribution T[0.2691, 0.299, 0.3289] and

(2) to artificially multiply the number observations

(500 replications of each area and scenario) which

makes RTE analysis be more discriminant. In the

proposed model, the Monte-Carlo engine allows the

simulation of inputs and outputs and offers the

statistical distribution of the RTE for each area in

each scenario and, by extension, the corresponding

one for the global system (Torres-Jiménez et al.,

2015).

Once inputs and outputs values were produced by

the Monte-Carlo engine, they are mathematically

(linear monotone increasing/decreasing functions)

interpreted based on expert knowledge formalised in

a IF … THEN … rule-base (knowledge-base), an

embryo of a fuzzy inference engine (Torres-Jiménez

et al., 2015). The rule design was based on the B-

MHCC paradigm (Salvador-Carulla et al., 2007).

Finally, and using the transformed variable

values, the operational algebraic model was designed

and solved. The BCC-DEA model, variable returns to

scale, was selected because there is no evidence of a

constant returns to scale rigid behaviour (Salvador-

Carulla et al., 2007). Both input and output

orientations of the BCC-DEA model were used. Input

orientation refers to maintaining a stable level of

outputs, while trying to minimize the resources

utilized. Output orientation aims to maximize the

outcomes for a constant amount of inputs.

In conclusion, for each scenario and BCC-DEA

orientation, the Decision Support System analysed 20

times (or repetitions) a 19×25×V (being: 19 the

number of areas, 25 the number of simulations and V

the number of variables –inputs and outputs- in the

corresponding scenario) datasets. The number of

simulations and repetitions was controlled by the

Nakayama’s error (Torres-Jiménez et al., 2015) that

should always be lower than 2.5% over the RTE

average.

RTE for each area (19), scenario (15) and

orientation (2) has a probabilistic structure that can be

statistically studied. By aggregation, the global RTE

of the system can also be statistically determined and

studied.

3 RESULTS

The results of the analysis showed the statistical RTE

assessment of mental health services provision and

use in 19 Bizkaia’s small areas. 15 different technical

Figure 1: Box-plots of input-oriented relative technical

efficiency of mental health areas for each scenario.

Figure 2: Box-plots of output-oriented relative technical

efficiency of mental health areas for each scenario.

perspectives (scenarios) of the RTE problem were

taken into account in addition to the two BCC-DEA

orientations: input and output. The analysis of the

resulting RTE statistical distributions allowed to: (1)

rank the areas, and (2) identify and assess potential

improvements in key variables by using a

benchmarking process (the area that showed the best

RTE average and the bigger probability of being

efficient is considered the benchmark).

In DEA models, a RTE equal to 1 means that the

analysed DMU is efficient (when the sum of the

slacks is equal to 0) or weak efficient (when the sum

of the slacks is greater than 0). Values lower than 1

show different levels of inefficiency, the lower the

value the lower the efficiency. Figures 1 and 2 plot

the minimums, maximums, confidence intervals

(two-tailed t-Student, α=0.95 and 29 freedom

degrees), averages and outliers of the resulting ETR

ICORES 2017 - 6th International Conference on Operations Research and Enterprise Systems

380

statistical distributions for each scenario (S#) in both

the input and output orientations.

The input orientation shows a less number of

outliers than the output one. The differences between

areas are greater in S1 and S6 to S15 in both

orientations, S2 to S5 have a relative homogeneous

behaviour.

The mental health areas 4, 6 and 11 appear in three

scenarios as outliers in the input-oriented model. On

the other hand, areas 6 and 15 are the most recurring

as outliers in the output-oriented model. Thus, the

area 6 can be considered as a RTE outlier. The area

15 is also an outlier in the most output-oriented

scenarios because of several missing data (highly

penalised in DEA models) because a private health

organization manages this area under contract

agreements with the public health system so its

information is not integrated.

The most efficient areas reach RTE average

values greater than 0.85 in most scenarios and in both

orientations. The worst RTE average values are lower

than 0.7 in the input orientation a 0.85 in the output

one (Figures 3 and 4).

Figure 3: Ranking of mental health areas for the input-

oriented model.

Figure 4. Ranking of mental health areas for the output-

oriented model.

The global input-oriented RTE average is 0.78

(Figure 3). Four areas have a RTE average values

greater than 0.85, while the lowest value is around

0.6. In the output-oriented model (Figure 4), results

are more homogenous and the RTE average is 0.88.

There are five areas above 0.95 and an outlier (area

15) close to 0.5 (due to its missing values).

4 DISCUSSION

4.1 The Monte Carlo DEA Model and

the Decision Support System (DSS)

The Monte-Carlo DEA model overcomes several

limitations of the traditional DEA models. Firstly, the

expert-based interpretation of input and output values

makes the result interpretation easier for decision

makers (Salvador-Carulla et al., 2007; Torres-

Jiménez et al., 2015) because it includes the specific

interrelations and particular characteristics of mental

health systems. For instance, the classical

assumption: “a situation which combines a low input

consumption with a high outcome production is

positive for the system performance” may not be

always correct or appropriate in mental health care

(Torres-Jiménez et al., 2015). Expert knowledge is

formalised in a rule-base by using the B-MHCC

paradigm (Salvador-Carulla et al., 2007), which

determines an appropriateness degree for each

variable value (“non-appropriate” values are

penalised in their mathematical transformation).

Secondly, the Monte-Carlo DEA model makes

RTE assessment more discriminant by the artificial

replication of the observation number. Datasets are

generated by the Monte-Carlo engine according to

variable values statistical distributions.

Finally, the uncertainty associated to data real

values is managed through the transformation of the

original variable values into standard statistical

distributions. The Monte-Carlo simulation engine

explores the variable values spectrum and offers a

RTE probabilistic view.

4.2 Strengths of the Study of the

Relative Technical Efficiency in

Bizkaia

Previous RTE studies have mainly assessed the

efficiency of complete systems (Kaya Samut & Cafrı,

2015); specific services, such as nursing homes

(Garavaglia, Lettieri, Agasisti & Lopez, 2011;

Kleinsorge & Karney, 1992), hospitals (Dash,

Assessment of Relative Technical Efﬁciency of Small Mental Health Areas in Bizkaia (Basque Country, Spain)

381

Vaishnavi, & Muraleedharan, 2010; Mogha, Yadav &

Singh, 2016) or primary care (Cordero et al., 2015;

Kirigia et al., 2011). However, they have not allowed

to know performance differences within the whole

health system. This research has studied different

RTE scenarios designed to describe the behaviour of

both the partial (i.e. residential care) or mixed (i.e.

day and outpatient care jointly) typologies based on

the B-MHCC paradigm. Thus, these scenarios

incorporate an integrative vision of mental health

care, including all the types of care (from a holistic

perspective) in which health and social care are

highlighted. This fact lets us to understand and assess

specific mental health care itineraries that patients

should follow in order to increase RTE and quality

care. Results include all the RTE statistical

distributions as well as the global RTE of the system.

This study has analysed the provision and

utilisation of mental health services in a real system

through an exhaustive data collection from the

Integrated Mental Health Atlas of Bizkaia (Pereira et

al., 2013; Salvador-Carulla et al., 2011). The use of a

standardized model for mental-health care

description and assessment was absolutely essential

because the name of the service was not enough for

describing its management structure and for making

comparisons. The Mental Health Atlas collected

information about the availability of specific types of

care, placement capacity, availability of workforce

and utilization indicators. The Monte-Carlo DEA

model integrates the uncertain information with an

operational model for assessing RTE and potential

managerial improvements.

4.3 Analysis of the Mental Health

System RTE in Bizkaia

Efficient mental health areas may be identified as

references for benchmarking. The assessed potential

improvements can guide management interventions

on the provision (inputs) and outcomes (outputs). On

one hand, the provision of mental health care in

inefficient areas could be adjusted to the values of the

efficient ones. On the other, interventions on the

service utilization could be direct such as the

optimization of facilities, placements and staff; or

indirect such as economic incentives, training

activities, policy design or good practices promotion.

The global RTE of the system is greater in the

output-oriented model than in the input-oriented one.

This suggests that a decisional strategy based on

optimizing the input amount, within the ranges

stablished by the B-MHCC paradigm, may be more

adequate for the Mental Health System of Bizkaia.

In the input-oriented analysis and in spite of the

existence of outliers, the scenarios where the RTE

scores are more homogenous are those that evaluated

the residential non-acute care and day hospitals (S2 to

S5 and, by extension, S13). This characteristic is the

result of the current and careful political planning.

Obviously, RTE scores can be improved in each

scenario by designing specific policies, especially on

the outlier areas.

In the output-oriented analysis (more

homogeneous), the most efficient scenarios are S11

and S12 (community mental health 1 and 2) and S4

(health-related day care). According to the results, S2

(non-acute residential and hospital care), S3

(residential care), S5 (non-health-related day care)

and s6 (outpatient care) could be main the targets in a

decisional environment based on the improvement of

the RTE. In this DEA orientation, the area number 15

has a relevant impact on the RTE scores because it

was a highly penalised due to the lack of information.

It is highly recommended to increase the day care

resources to be equal, at least, to medical ones. This

intervention should increase both the RTE and the

mental health care quality, in addition to an expected

decrease in the economic burden of the system. In this

sense, there are empirical evidences that show that

community-based care is more cost-effective than

institutional-based care (WHO, 2005).

The proposed DSS can assess the impact of an

almost infinite number of planning interventions.

This process can decrease the intrinsic managerial

risk associated to any real management decision. For

example, it can evaluate the effects on the system of

transferring some professionals from a mental health

area, or areas, to other/s: this implies changes in the

provision, utilization and outcome variables. This

analysis understands that any intervention in a

specific geographical area will probably have an

impact on the others because of the interrelations

between them.

4.4 Limitations

The analysis of RTE in specific mental health areas is

relevant and useful but insufficient to evaluate the

global situation of mental health care. The pathway of

care that should be followed by a specific user has to

be designed depending on his clinic status. In Spain,

the first point of contact in the health care system is

usually located in a primary care service or in a

hospital. From these units, the patient can be derived

to a secondary care service afterwards. All the

mentioned services have been include in this study.

Nevertheless, until the patient arrives to this

ICORES 2017 - 6th International Conference on Operations Research and Enterprise Systems

382

secondary level, he has followed an itinerary that

ought to be studied if RTE scores have to be

increased. To avoid an increase in the re-

hospitalization number, in the number of stays at the

hospitals, in the frequentation, the prevalence or even

in the incidence of mental disorders, a most efficient

care coordination and an integrative professional

practice have to be highlighted (Burns, Goldsmith, &

Sen, 2013; Cordero et al., 2015).

In conclusion, it should be necessary to include

primary care services in RTE assessment in order to

have a complete picture of the mental health system

under analysis.

5 CONCLUSIONS

In the decision making processes based on empirical

evidence, the intrinsic decisional risk decreases.

Therefore, it is fundamental to provide the decision

maker as much reliable information as possible to

understand the real situation (Gibert, García-Alonso

& Salvador-Carulla, 2010).

The Monte-Carlo DEA model has provided high-

level and empirical informed-evidence on the RTE

based on the provision and utilization of mental

health services in small geographical areas of the

Bizkaian Health System. Based on the results, it has

been possible to identify and analyse potential

improvements that can be transformed into decisional

interventions to be checked by modifying input or

output values (statistical distributions) in the DSS.

The obtained results may help decision makers to

prioritise them in an uncertain context dominated by

economic restrictions.

Future research will be focused on the validation

of the DSS analysing real decisional situations with

multiple feasible alternatives. Selected micro-

management interventions, those that imply a relative

small number of variables, based on policymaker

interests will be selected to assess potential

improvements and risks on the system management

prior their implementation, Following this process,

the decision making process is supported by empirical

evidence. This feature matches with the strategies

established in the Mental Health Action Plan 2013-

2020 (WHO, 2013).

ACKNOWLEDGEMENTS

This study is part of a bigger Project named “Atlas de

Salud Mental de Bizkaia” funded by the “Red de

Salud Mental de Bizkaia”. It has been carried out in

collaboration with “Osakidetza” and the

“Departamento de Sanidad y Consumo del Gobierno

Vasco”.

REFERENCES

Alirezaee, M.R., Howland, M. & Van de Panne, C., 1998.

Sampling size and efficiency bias in data envelopment

analysis. Journal of Applied Mathematics and Decision

Sciences, 2(1), pp.51-64.

Burns, L.R., Goldsmith, J.C. & Sen, A., 2013. Horizontal

and vertical integration of physicians: a tale of two tails.

Advances in health care management, 15, pp.39–117.

Cordero, J.M. et al., 2016. Technical efficiency assessment

of public primary care providers in the Basque Country

(Spain), 2010-2013. Gaceta Sanitaria, 30(2), pp.104–

109. Available at:

http://www.scopus.com/inward/record.url?eid=2-s2.0-

84959099383&partnerID=40&md5=3043220dd789be

1b3dd178769affd73c.

Dash, U., Vaishnavi, S.D. & Muraleedharan, V.R., 2010.

Technical Efficiency and Scale Efficiency of District

Hospitals: A Case Study. Journal of Health

Management, 12(3), pp.231–248. Available at:

http://www.scopus.com/inward/record.url?eid=2-s2.0-

77955896077&partnerID=tZOtx3y1 [Accessed March

14, 2016].

Dyson, R.G. et al., 2001. Pitfalls and protocols in {DEA}.

European Journal of Operational Research, 132(2),

pp.245–259. Available at:

http://www.sciencedirect.com/science/article/pii/S037

7221700001491.

Färe, R. et al., 2008. Evaluating health care efficiency.

Advances in health economics and health services

research, 18, pp.209–228. Available at:

https://www.scopus.com/inward/record.uri?eid=2-

s2.0-

58849144444&partnerID=40&md5=0156be57de9f25

4a68436a2db8b6d62f.

Garavaglia, G. et al., 2011. Efficiency and quality of care in

nursing homes: An Italian case study. Health Care

Management Science, 14(1), pp.22–35. Available at:

https://www.scopus.com/inward/record.uri?eid=2-

s2.0-

79551530621&partnerID=40&md5=eddb9c7279a145

d0f1c32c9181972ff9.

Gibert, K., Garcia-Alonso, C. & Salvador-Carulla, L., 2010.

Integrating clinicians, knowledge and data: expert-

based cooperative analysis in healthcare decision

support. Health research policy and systems / BioMed

Central, 8, p.28.

Gobierno Vasco. (2010). Estrategia para afrontar el reto de

la cronicidad en Euskadi, Bilbao, Eusko Jaurlaritza-

Gobierno Vasco.

Gutierrez-Recacha, P. et al., 2006. Cost-effectiveness of

different clinical interventions for reducing the burden

Assessment of Relative Technical Efﬁciency of Small Mental Health Areas in Bizkaia (Basque Country, Spain)

383

of schizophrenia in Spain. Acta psychiatrica

Scandinavica. Supplementum, (432), pp.29–38.

Hollingsworth, B., 2008. The measurement of efficiency

and productivity of health care delivery. Health

economics, 17(10), pp.1107–1128.

Hollingsworth, B. & Parkin, D., 2001. The efficiency of the

delivery of neonatal care in the UK. Journal of public

health medicine, 23(1), pp.47–50.

Kaya Samut, P. & Cafrı, R., 2015. Analysis of the

Efficiency Determinants of Health Systems in OECD

Countries by DEA and Panel Tobit. Social Indicators

Research. Available at:

https://www.scopus.com/inward/record.uri?eid=2-

s2.0-

84941367202&partnerID=40&md5=77b2c000e68c3d

32218b73040f075a46.

Kirigia, J.M. et al., 2011. Technical efficiency of primary

health units in Kailahun and Kenema districts of Sierra

Leone. International archives of medicine, 4, p.15.

Kleinsorge, I.K. & Karney, D.F., 1992. Management of

nursing homes using data envelopment analysis. Socio-

economic planning sciences, 26(1), pp.57–71.

Ministerio de Sanidad, Política e Igualdad (2011).

Estrategia en Salud Mental del Sistema Nacional de

Salud 2009-2013 [pdf] España. Avaliable at

http://www.aepcp.net/arc/SaludMental2009-2013.pdf

[Accessed 24 Oct. 2016]

Mogha, S.K., Yadav, S.P. & Singh, S.P., 2015. Slack based

measure of efficiencies of public sector hospitals in

Uttarakhand (India). Benchmarking: An International

Journal, 22(7), pp.1229–1246. Available at:

http://www.scopus.com/inward/record.url?eid=2-s2.0-

84942784709&partnerID=tZOtx3y1 [Accessed March

14, 2016].

Moran, V. & Jacobs, R., 2013. An international comparison

of efficiency of inpatient mental health care systems.

Health policy (Amsterdam, Netherlands), 112(1-2),

pp.88–99. Available at:

http://www.scopus.com/inward/record.url?eid=2-s2.0-

84885191198&partnerID=tZOtx3y1 [Accessed March

14, 2016].

[OECD] Organisation for Economic Co-operation and

Development. (2010). Health expenditures and

financing. [online] Available at:

https://stats.oecd.org/Index.aspx?DataSetCode=SHA

[Accessed 20 Oct. 2016]

[OECD] Organisation for Economic Co-operation and

Development. (2016). Mental Health and work.

[online] Available at:

https://www.oecd.org/els/emp/mental-health-and-

work.htm [Accessed 20 Oct. 2016]

Pares-Badell, O. et al., 2014. Cost of disorders of the brain

in Spain. PloS one, 9(8), p.e105471.

Pelone, F. et al., 2012. The measurement of relative

efficiency of general practice and the implications for

policy makers. Health policy (Amsterdam,

Netherlands), 107(2-3), pp.258–68. Available at:

http://www.scopus.com/inward/record.url?eid=2-s2.0-

84865970504&partnerID=tZOtx3y1 [Accessed March

14, 2016].

Pereira, C., Gutiérrez-Colosía, M. and Salinas-Perez, J.

(Eds.). (2013).

Mental Health Atlas of Bizkaia. Bilbao:

Osakidetza.

Salvador-Carulla, L., Dimitrov, H., Weber, G., McDaid, D.,

Venner, B., Šprah, L., ... and Johnson, S. (2011).

DESDE-LTC: Evaluation and Classification of

Services for Long Term Care in Europe. Spain: Psicost

and Caixa Catalunya.

Salvador-Carulla, L. et al., 2007. Use of an operational

model of community care to assess technical efficiency

and benchmarking of small mental health areas in

Spain. The journal of mental health policy and

economics, 10(2), pp.87–100.

Shen, G.C. & Snowden, L.R., 2014. Institutionalization of

deinstitutionalization: a cross-national analysis of

mental health system reform. International journal of

mental health systems, 8(1), p.47.

Sherman, H.D., 1984. Hospital efficiency measurement and

evaluation. Empirical test of a new technique. Medical

care, 22(10), pp.922–938.

Staat, M., 2001. The Effect of Sample Size on the Mean

Efficiency in DEA: Comment. Journal of Productivity

Analysis, 15(2), pp.129–137. Available at:

http://dx.doi.org/10.1023/A:1007826405826.

Torres-Jiménez, M. et al., 2015. Evaluation of system

efficiency using the Monte Carlo DEA: The case of

small health areas. European Journal of Operational

Research, 242(2), pp.525–535. Available at:

http://www.scopus.com/inward/record.url?eid=2-s2.0-

84920716285&partnerID=tZOtx3y1 [Accessed March

14, 2016].

Tyler, L.H., Ozcan, Y.A. & Wogen, S.E., 1995. Mental

health case management and technical efficiency.

Journal of medical systems, 19(5), pp.413–423.

United Nations. (1991). The protection of person with

mental illness and the improvements of mental health

care. [online] Available at

http://www.un.org/documents/ga/res/46/a46r119.htm

[Accessed 25 Oct. 2016].

World Health Organization. (2003). Investing in MENTAL

HEALTH [online] Available at

http://www.who.int/mental_health/en/investing_in_mn

h_final.pdf [Accessed 25 Oct. 2016].

World Health Organization. (2005). Mental health policy,

plans and programmes. Mental Health Policy and

Service Guidance Package. [online] Available at

http://www.who.int/mental_health/policy/services/esse

ntialpackage1v1/en/ [Accessed 25 Oct. 2016].

World Health Organization. (2013). Mental Health Action

Plan 2013 – 2020. [online] Available at

http://www.who.int/mental_health/publications/action

_plan/en/ [Accessed 25 Oct. 2016].

World Health Organization. (2016). Mental disorders.

[online] Available at

http://www.who.int/mediacentre/factsheets/fs396/en/

[Accessed 25 Oct. 2016].

Zhu, J., 2003. Imprecise data envelopment analysis

(IDEA): A review and improvement with an

application. European Journal of Operational

Research, 144(3), pp.513–529.

ICORES 2017 - 6th International Conference on Operations Research and Enterprise Systems

384