Health Evaluation in Software Ecosystems

Iuri Carvalho, Fernanda Campos, Regina Braga, José Maria David,

Victor Stroele and Marco Antônio Araújo

Department of Computer Science, Post Graduate Program in Computer Science,

Federal University of Juiz de Fora (UFJF), Brazil

Keywords: SECO Health, Software Ecosystem, Quality, Process, Metrics, Systematic Mapping.

Abstract: Context: The quality of a Software Ecosystem (SECO) platform and its available products are important

characteristics to ensure its success. However, this concept goes beyond the traditional approaches of quality

assurance, including concepts such as SECO´s health. Objectives: The aim of this study is propose an

evaluation process to application of health metrics. In addition, this metrics were formalized to make feasible

your application and improve the obtained results. Method: A systematic mapping was conducted with the

aim of analyzing the SECO quality research area, highlighting the state of the art and identifying its main

characteristics. In addition, the main approaches and metrics present in the literature for SECO quality and

health evaluation are detailed. This work presents an observational study used to define relevant heath metrics

considering an evaluation process. Results: The metrics were formalized and evaluated by specialists. A health

evaluation process was developed to applicate this metrics. This process is supported by an architecture named

HEAL ME.

1 INTRODUCTION

Software development scenario has rapidly been

changing (Jansen 2013). Currently, there is a massive

presence of Software as a Service (SAS) approaches,

driven mainly by ubiquitous computing. In this

context and with the emergence of challenges such as

Distributed Software Development (DSD), just

maintaining a central architecture is not enough for

most enterprise developers. They need to open their

architectures for the collaboration of external

developers (Bosch 2009). This has given rise to a new

concept of development, where several software

solutions, companies and developers adhere to a

common platform. This scenario is called Software

Ecosystem (SECO) (Jansen 2013).

A SECO is considered an open software platform,

is basically composed of a keystone, a platform and a

set of niche agents (Bosch and Bosch-Sijtsema 2010).

The centralizer acts in the development of the

platform and in the management of relationships with

external parties. However, niche agents are those that

influence the development of the ecosystem.

For the companies that maintain the platform, i.e.,

the keystones, there are several advantages of

adopting a SECO approach. Increase their platform

scope, reaching a greater number of users with their

software solutions, is one of them. Also, the cost

reduction with R&D (Research and Development)

can be considered, since several new software

solutions are developed by external companies

(Bosch 2009).

However, the scenario created by SECO brings

new challenges. Among them we can highlight the

quality assurance of products and platform. The

independence of external developers as well as the

platform characteristics can influence the quality of

the entire ecosystem. In addition, due to the

complexity of the scenario, quality assurance in the

context of SECO has its peculiarities (Santos et al.

2014). Unlike traditional development, the SECO

platform maintainer does not have control over

models and development processes used by outside

companies. In this way, the keystone cannot directly

guarantee the quality of the products developed on the

platform (Santos et al. 2014). Some keystones

strengthen the distribution of these products, making

a deep quality control before providing them.

However, certain companies avoid developing on

these platforms due to the difficulty of making their

products available (Jansen 2013). Therefore, in

addition to the quality standards observed in software

Carvalho, I., Campos, F., Braga, R., David, J., Stroele, V. and Araújo, M.

Health Evaluation in Software Ecosystems.

DOI: 10.5220/0006699202630271

In Proceedings of the 20th International Conference on Enterprise Information Systems (ICEIS 2018), pages 263-271

ISBN: 978-989-758-298-1

263

products, considering the product and the process

quality evaluation, there are other dimensions to be

observed in the context of a SECO. Some of them

have a platform-oriented approach, such as the SECO

health dimension (Santos et al. 2014).

Common quality concepts do not cover the

complexity of the environment created by a SECO, as

the large number of employees grouped in

development communities, as well as a large

relationship network (Fotrousi et al., 2014).

Furthermore, independent developers and a larger

number of users, each one with his/her own needs,

make quality assessment processes a challenge. This

situation led to a new evaluation dimension, known

as "SECO health" (Amorin et al., 2017), as mentioned

before.

According to (Santos et al. 2014), SECO health is

the degree to which a Software Ecosystem provides

opportunities for collaborators and for those that are

platform dependent. Health can also be defined from

the point of view of SECO investors. According to

(Bosh 2009), one of the advantages of adopting

SECO is the reduction of investments in R&D by the

keystone, since these investments are carried out by

companies that join the SECO. Investments applied

in a SECO are high, besides the dependency between

the components and the platform. In order to avoid

losing these investments and to meet the expectations

of those who adopt it, there must be no risk of SECO

death or failures (Amorim et al., 2017).

With the goal of identifying researches in SECO

quality, mapping the state of the art and detecting

possible shortcomings and research gaps, a systema-

tic mapping (Kitchenham 2004) of the literature was

carried out. With the results, it was possible to

identify specific key points on SECO quality research

and, to propose a quality assessment approach

focused in health considering SECOs context.

Among the papers selected during the systematic

mapping process, some of them propose health

assessment metrics for SECO. These metrics were

automated by the HEAL ME architecture, described

in (Carvalho et al., 2017), as our first effort to define

the process. However, the applicability and adherence

of these metrics were not evaluated within the

suggested context. We then proposed the evaluation

of these metrics through an observational study,

executed considering SECO´s experts.

Finally, in order to provide better application of

these metrics and evaluation effectiveness, an

assessment process was proposed. This process tries

to formalize the application and validation of the

metrics through the observational study. This

evaluation is important to encourage SECOS´ new

collaborators, users and partners, providing a first

step to ensure SECOS´s health.

This paper is structured as follows. Section two

presents the systematic mapping execution and

results. Section three describes a set of metrics for

SECO quality and health, grouped in an architecture

named HEAL ME. These metrics were extracted

from the systematic mapping, and evaluated by

specialists. Finally, section four presents the final

considerations, threats to the validity, and future

works.

2 SYSTEMATIC MAPPING

Basing the answers of research questions in evidences

is an important approach to results effectiveness. For

this reason, (Kitchenham 2004) presents the

evidence-based Software Engineering paradigm. The

principle of this paradigm is to answer research

questions, based on evidences found in the literature.

Considering this approach (Kitchenham 2004), a

systematic mapping of the literature was carried out

to evaluate areas of interest related to SECO´s quality.

The objective was to identify the main approaches

present in the literature, trends and state of the art of

SECO´s quality assurance. To perform the systematic

mapping, three phases were defined: planning,

conducting and reporting the study (Kitchenham

2004). For the execution of the entire mapping

process, the Parsifal tool was used, available at:

https://parsif.al.

2.1 Planning

The first step in the construction of the protocol was

the definition of mapping and research questions,

with the goal of finding SECO quality assurance

approaches and its state of the art. As ecosystem

health can be considered a sub-area of quality, the

mapping was carried out under the theme of quality

and later the study was specialized for the SECO

health area.

Three mapping questions were presented: MQ1:

What are the main publication venues in the area?

MQ2: How are papers distributed over the years?

MQ3: Which authors are outstanding in the area?

Four research questions were then proposed:

RQ1: Which quality assurance approaches are used

for SECO? RQ2: Which quality attributes are more

often used to evaluate SECO? RQ3: Which model is

the most used in the evaluation of SECO quality?

RQ4: Which model or quality approach is more used

in SECO health assessment?

ICEIS 2018 - 20th International Conference on Enterprise Information Systems

264

The PICOC (Petticrew and Roberts 2008) was

defined as follows: Population: solutions that

address the quality assurance in SECO; Intervention:

attributes and processes for quality or use of actors'

perceptions or health and prosperity assessments;

Comparison: no comparisons were defined;

Outputs: methods, techniques, approaches, models,

solutions, metamodels, dimensions and other

solutions for quality; Context: Software Ecosystems.

Then, the inclusion criteria defined were: IC1:

Publications from the year 2009 ahead, since SECO

area has gained focus from the work published by

(Bosch 2009); IC2: Open publications type, due to the

need to evaluate their contents; IC3: Publications of

the Computer Science area, because of its specificity;

IC4: Publications which main theme is SECO quality

assurance. The exclusion criteria were defined as

follows: EC1: Publications before 2009; EC2:

Publications that are not of the Computer Science

area; EC3: Publications that are not open

publications; EC4: Publications that do not focus on

SECO quality assurance.

In the sequence, the research bases were selected.

They were selected considering publications on the

Computer Science area. Another criterion was the

ability of using advanced searches to apply the

mapping search string. Finally, all the selected bases

should be compatible with the support tool. The

selected bases were: ACM Digital Library

(http://dl. acm.org/); IEEEXplorer

(http://ieeexplore.ieee.org); ScienceDirect

(http://www.sciencedirect.com/); Scopus

(https://www.scopus.com); and Web of Science

(http://apps.webofknowledge.com/).

With the bases defined, the next step is the

definition of the search string. The PICOC items were

used to identify the most relevant words and

expressions. As a result, the following string was

generated: (quality OR "quality assurance") AND

(perception OR perceptions OR attribute OR

attributes OR process OR health OR evolvability)

AND ("software ecosystem" OR "software

ecosystems" OR SECO OR "software digital

ecosystem" OR "software digital ecosystems"). This

is a generic string considering that each base has its

own search syntax. In order to effectively do the

searches, the string was adapted for each base,

maintaining its basic structure.

An important parameter to evaluate the search

effectiveness is the definition of control papers. They

must be important papers in the area, and that is useful

to evaluate if the search is correctly reflecting the

proposed objectives. Five control papers were defined

for this systematic mapping. They were also

presented by (Manikas 2016) as important quality

assurance papers in ECOS. The selected papers are:

(Schugerl et al. 2009), (Hmood et al. 2012), (Jansen

2013), (Stefanuto et al. 2011), (Franco-Bedoya et al.

2014). The next phase is the conduction of the

systematic mapping.

2.2 Conduction

In this phase, each database was accessed and the

search string was executed. The searches were

performed between August and September 2016 and

updated in May 2017. The total number of studies

founded was 109. Table 2 shows the distribution of

these results by bases.

Table 1: Search results.

Database Number of Papers

ACM 16

IEEEXplorer 12

ScienceDirect 9

Scopus 56

Web of Science 16

Total 109

Among these papers, 29 were duplicated and were

automatically excluded by the Parsifal tool, resulting

in 80 papers. Then the titles and summaries were read

and the inclusion and exclusion criteria applied. In

this process, 57 files were excluded, remaining 23 for

text full analysis. After this stage, 11 papers were

excluded, resulting in 12 papers classified as relevant

for the area of SECO quality assurance. Reflecting the

effectiveness of the search, among the relevant papers

were the control papers, that the search string is

consistent with the study objectives.

2.3 Study Report

Analyzing the distribution of publications we can

answer the first question (MQ1): What are the main

publication venues in the area? A widespread venue

distribution is observed as only two papers were

published in the same conference, the proceedings of

the International Conference on Management of

Emergent Digital EcoSystems - MEDES. Another

important note is the fact that 10 papers were

published in proceedings, and only two in journals.

This situation denotes a lack of specific conferences

and journals in the area.

In order to analyze the evolution of the area, we

can observe the graph in Figure 1, where the number

of publications per year since 2009 is presented. With

this data, it is possible to answer the second question

Health Evaluation in Software Ecosystems

265

(MQ2): How are papers distributed over the years? It

is possible to visualize the growing importance of the

area in the last four years. This is an expected

outcome, since SECO approach is increasingly

present in the context of current software

development (Manikas 2016). However, between

2011 and 2012 there is a discontinuity growth, since

the number of publications is lower in 2012 than in

2011. This situation may point out to the immaturity

of the area. In this way, we may consider that perhaps

the basic concepts of SECO are not fully established.

Finally, we can analyze the distribution of

publications by author, answering the third question

(MQ3): Which author or authors are most outstanding

in the area? By observing the results, a dispersion can

be detected. Among the twelve selected papers, only

the ones published by (Alves and Pessôa 2010),

(Stefanuto et al. 2011) and (Alves et al. 2015) have

authors or co-authors in common. The remaining

papers are from different authors.

Therefore, from this systematic mapping, there is

evidence that the area is still emerging and is not fully

explored, considering the small number of works

about SECO quality assurance. In addition, we

identified the importance of software quality

assurance in the context of SECO research. These

results are detailed reported in the following section.

2.4 Results

To answer the first research question (RQ1) - Which

quality approaches are used for SECO? - Several

different approaches can be mentioned. Among the

papers are quality models, QuESo (Franco-Bedoya et

al. 2014), CoCoADvISE (Lytra et al. 2015), SE-

Advisor (Schugerl et al. 2009) and the meta-model

SE-Equam in (Hmood et al. 2012). As quality

frameworks, we can cite the BISA (Kajan et al. 2011).

In (Alves and Pessôa 2010) the framework named

5CQualiBr is presented, and the PRO2PI-MFMOD

framework is present in (Alves et al. 2015). The BPS

Maturity Model is addressed in (Stefanuto et al. 2011)

and (Alves and Pessôa 2010). Finally, (Mhamdia

2013) presents a literature review, listing several

measurement processes applied to ECOS from a

quality perspective, while (Frantz et al. 2015) present

the application of Markov Decision Processes, with

the same objective. The results reflect the area

diversity, considering the identified approaches. Each

proposal has its unique characteristics, ranging from

maturity models to decision-making processes.

However, one can observe the use of common

techniques, such as semantic analysis.

Considering the second research question (RQ2)

- Which quality attributes are more often used to

evaluate SECO? - we can highlight the ones from (da

Silva Amorim et al. 2014), i.e., communication,

teamwork maturity, technology and integration.

These attributes, which are linked to the SECO

software platform, are also addressed by (Franco-

Bedoya et al. 2014), (Jansen 2013) and (Jansen 2014).

These three papers also share other attributes, such as

sustainability and openness. Identifying these

attributes is of great importance since they are the

basis of the main quality models. In this way, it is

evident that such attributes are of extreme importance

for an evaluation and quality assurance in the context

of a SECO.

The third research question (RQ3) - Which model

is the most used in the evaluation of SECO quality

assurance? - can be answered from the two-

dimensional analysis. The first dimension is the

recurrence of models in the papers. It is possible to

highlight the maturity model presented by (Stefanuto

et al. 2011), which is directly or indirectly addressed

in three papers. The second dimension analyzed was

the number of citations of the papers in the bases. As

a result, we can present the (Franco-Bedoya et al.

2014) and (Schugerl et al. 2009) as the most cited

quality models, and again (Stefanuto et al. 2011) as

the most cited maturity model.

Finally, the fourth research question (RQ4) -

Which model or quality approach is most used in

SECO health assessment? – presented the models and

Figure 1: Publications evolution graph.

ICEIS 2018 - 20th International Conference on Enterprise Information Systems

266

metrics proposed by (Franco-Bedoya et al. 2014)

and (Jansen 2014), where SECO platform quality

dimen-sions are directly addressed. Other models that

treat health and prosperity in a less expressive way are

found in (Schugerl et al. 2009) and (Hmood et al.

2012).

After analyzing the results, we can highlight some

points. First, researches in the area follow the trend of

creating quality models for SECO evaluation.

(Franco-Bedoya et al. 2014) and (Jansen 2013) are

concerned with evaluation of software characteristics

and attributes through metrics. Another concern is the

automation of processes and the use of semantic

analysis to carry out the evaluations, due to the

complexity of the processes and the environment to

be analyzed (Schugerl et al. 2009). However, the

papers do not address SECO in a generic way, only

specific contexts are analyzed. In addition, process

automation is still immature, and we believe that it can

be substantially improved through the use of specific

tools and intelligent analysis. Another drawback is the

presentation of the evaluations results of the proposed

models, whereas a more elaborate analysis with

adequate visualization resources is not usual.

Considering these observations, HEAL ME

architecture was proposed (Carvalho et al. 2017). It

aims to evaluate the quality and health of a SECO in

a generic and semi-automated way, analyzing data

using metrics from the literature, ontological rules

and visualization techniques to improve their

understanding.

In order to present the HEAL ME metrics and also

evidences of its usefulness, an observational study

was carried. Specialists validated the set of metrics,

with the aim of performing an initial validation of

HEAL ME utility. In the next section, this process is

detailed, focusing on SECO´s health characteristics.

3 QUALITY AND HEALTH

METRICS

Based on the previous results, 58 quality and health

evaluation metrics were defined and detailed in

HEAL ME architecture (Carvalho et al. 2017)

context. The complete list of metrics is available at:

http://www.ufjf.br/nenc/files/2008/09/Metrics-

HEAL-ME.pdf

HEAL ME architecture aims to automate the

health assessment process of a SECO. Based on the

systematic mapping, the HEAL ME metrics were

reviewed and evaluated by three SECO experts, based

on an observational study. Each specialist

individually assessed each metric to define its

usefulness and importance. The conduction of the

observational study is presented in the next section,

showing the steps from the data collection to the

reporting of results.

3.1 Observational Study

The study was conducted in two steps. At first, the

data was organized, aiming to capture the specialists’

perception about each metric. Then, a structured

questionnaire with all 58 metrics was sent to

specialists. Each of them individually analyzed the

metrics. During the second step, specialists’ answers

were quantified assigning weights to their responses.

Then, it was possible to formalize the metrics and

define their components using the information

obtained from the study results. This formalization

will be presented in section 3.2.

The study was also organized considering five

phases: Definition, Goals, Planning, Execution and

Results. The study definition aims to outline the

initial steps of the research. The goals phase defines

the main objectives of the study. The planning phase

aims to define the application of the study itself, as

well as the required items for its execution and the

selected data sources. The execution phase details the

steps to reach the final results and, finally, at the

results phase, the data analysis is detailed,

emphasizing how the goals were reached (Perry et al,

1998).

Definition

The scope of the observational study was defined

based on the GQM method (Basili et al. 1994): “to

validate the metrics used by the HEAL ME

architecture with the purpose of verifying the

usefulness and importance of each metric from the

point of view of specialists in the context of a SECO".

In order to define the scope, the following

research question was proposed: Which metrics used

by the HEAL architecture are useful for the

evaluation of a SECO health?

Goals

The goal of the observational study was to evaluate

the usefulness and adherence of each metric present

in the HEAL ME architecture from the point of view

of specialists in the context of SECO. To achieve this

goal, the study was applied to the subjects using a

questionnaire as the main instrument. Each subject

evaluated the metrics presented within the defined

scale.

Health Evaluation in Software Ecosystems

267

Planning

To evaluate the proposed metrics, the study was

performed considering SECOs experts and their

background. The subjects were Software Engineering

specialists, two from the industry and one from the

academy. Data collection was based on online

interviews with the subjects, supported by a

structured questionnaire available at: https://drive.

google.com/open?id=11moBFOXfE7CXVDNQVBJ

D_yZ8i0qCfpMKtv3NnZPOoKk. Each subject

should evaluate each metric, considering its utility

within the following scale: Essential, Desirable, and

Dispensable. The use of a multiple-choice scale to

answer the questionnaire reflects the precision needed

in the metric assessment.

Execution

The application of the questionnaire was done online,

without interviewer´s interference during the process.

All subjects answered the entire questionnaire and did

not report any doubts about its completion.

The subjects accessed the questionnaire and

individually selected the responses, on different days

and times. Each subject took, at average, about 20

minutes to complete the questionnaire. An important

point to be emphasized is that all subjects have a good

knowledge about SECO, and one of the subjects is

currently working with SECO in the industry. After

the study was carried out, the responses were

collected, analyzed and the results are reported below.

Results

The results were analyzed to evaluate the adherence

of each metric to the health and quality assessment

process proposed by the HEAL ME architecture.

Each response was distributed considering three

scales: Essential, Desirable and Dispensable. To

perform the analysis, each response received a

weight, respectively, 2, 1 and 0, considering that

essential is a mandatory metric, desirable is a metric

that may be considered and dispensable are the ones

that must be discarded. The weights demonstrate the

relevance of each response set in a numeric way.

These weights were totalized in each metric

considering the response of each subject. The

averages were calculated, and classified in the

following scale: Essential (average greater than 1.66),

Partially Essential (average less than 1.66 and higher

than 1), Desirable (average equal to 1), Partially

Dispensable (average less than 1 and above zero), and

Dispensable (average equal to zero). The results are

shown in Figure 2.

It can be noticed that 81.03% of the metrics were

evaluated between Essential and Desirable, totalizing

46 metrics. This result shows evidences that the

metrics are useful and adherent to the proposed

context. On the other hand, it was detected that

18.97% of the metrics were evaluated between

Partially Dispensable and Dispensable, totalizing 12

metrics. This demonstrates that new studies should be

conducted in order to validate the evidence collected

in this first evaluation. However, with these results,

the research question could be answered, considering

the 46-metrics positive evaluated. It should be noted,

though, that this is a primary study. New experiments

should be conducted in order to conclusively validate

the research question.

Figure 2: Study results.

3.2 Metrics Description

The list of formalized metrics is available at:

http://www.ufjf.br/nenc/files/2008/09/Formalizadas

HEAL-ME.pdf. There is also a need to define

metrics´ components and evaluation procedures.

Therefore, the metrics components are: Description,

which shows the utility of the metric; Measure and

Formula, that details the application of the metric

using its data; Interpretation, which shows the

meaning of metric result; Unit to quantify the result

measure; and Actor, the individual related with the

metric. With these components, the metrics

evaluation precision is increased, and the results

reflects the reality of the evaluated context. In

addition, to facilitate this interpretation, the metrics

were grouped in three SECO dimensions: Technical,

Business and Social (Barbosa et. al. 2013).

3.3 Definition of Evaluation Process

Based on the metrics validation through the

observational study, it was possible to improve the

ICEIS 2018 - 20th International Conference on Enterprise Information Systems

268

health assessment process considering the HEAL ME

architecture, evolving the architecture, as well as

increasing its effectiveness in the evaluation of

SECO´s health. Additionally, the metrics formaliza-

tion improves the effectiveness and result precision.

The metrics grouping in SECO dimensions have the

potential to show the SECO environment, using a

simple and realistic structure.

As stated before, the metrics used by the HEAL

ME architecture were automated through semantic

rules specified using an ontology together with the

Semantic Web Rule Language (SWRL). Therefore,

the SECO data are instantiated into the ontology to be

evaluated. Some metrics are captured automatically

by the HEAL ME architecture, through the

interaction of the architecture with SECO as an

application integrated to SECO´s core. this processes

is described in (Carvalho et al. 2017). This evaluation

process can be seen in Figure 4, using the BPMN

model notation.

The evaluation process consists of five steps. The

first is data collection. The data are collected

considering several SECO parameters, which are

related to development, users, relationships network,

among others. This data is collected automatically

through specific APIs that communicate with existing

repositories and / or with relevant applications and

with the platform. Each data collected is evaluated by

a metric. The second step is the parametrization of the

metric. These parameters are informed by experts and

vary according to the characteristics of the SECO.

The parameters are used to evaluate whether the

collected data are in accordance to the metric. The

third step is the instantiation of the data and

parameters into the ontology. This step is necessary

for the semi-automated execution of the evaluation,

through the semantic rules. The fourth step is the

execution of the rules. Using the parameters, the rules

automatically evaluate if the data is in accordance or

not to the metric. Finally, the last step is to visualize

the results generated by the evaluation process. In this

step, visualization techniques are used in order to

facilitate user’s understanding. Clustering techniques,

filters, among others are used. It is important to point

out that process-based automation is very important

due to the large number of metrics and the complexity

of executing quality and health assessment without an

automatic support.

4 FINAL CONSIDERATIONS

Quality assurance is an important non-functional

parameter in software production and development. It

is also present in SECO context as the number of

SECOs' users grows on a daily basis. This also causes

a growth in the need for new functionalities and

resources to be developed. In addition, the quality

vision in this context goes beyond the traditional one

of software development, for instance, addressing

concepts such as SECO's health. This concept is

stablished considering the reality of distributed

development in the SECO's context. In order to know

more about this panorama, a systematic mapping was

conducted. This mapping made it possible to find

specific papers presenting models, attributes and

solutions for SECO quality assurance.

Through the systematic mapping, health metrics

for SECO were found. These metrics were than

inserted into the HEAL ME architecture, which aims

to perform the SECO's health assessment

automatically. However, assessing the applicability

and usefulness of these metrics was necessary to

allow better reliability on the results of its application.

For these metrics' evaluation, an observational study

was carried out with domain specialists. After the

metrics were evaluated, they were formalized and

their main attributes were described. The

formalization of the metrics allowed the definition of

the evaluation process. This allows the improvement

of the accuracy of the evaluations, as well as

facilitates the automation of the SECO's health

evaluation process. As future work, we can mention

the conduction of formal experiments to validate the

use of the HEAL ME architecture in SECO specific

contexts and the effectiveness of the metrics and the

evaluation process.

ACKNOWLEDGEMENTS

To FAPEMIG, CNPq, UFJF and Capes for financial

support to the project.

Figure 3: Evaluation flow of the metrics in the HEAL ME architecture.

Health Evaluation in Software Ecosystems

269

REFERENCES

S. da S. Amorim, F. Simas S. Neto, J. D. McGregor, E. S.

de Almeida, and C. von F. G. Chavez. 2017. How Has

the Health of Software Ecosystems Been Evaluated? A

Systematic Review. In Proceedings of the 31st

Brazilian Symposium on Software Engineering

(SBES'17). ACM, New York, NY, USA, 14-23.

J. Bosch. From software product lines to software

ecosystems. In Proceedings of the 13th International

Software Product Line Conference, SPLC ’09, pages

111–119, Pittsburgh, PA, USA, 2009. Carnegie Mellon

University.

J. Bosch and P. M. Bosch-Sijtsema. “Softwares product

lines, global development and ecosystems:

Collaboration in software engineering”. In:

Collaborative Software Engineering (pp. 77-92).

Springer Berlin Heidelberg, 2010.

R. Santos, G. Valença, D. Viana, B. Estácio, A. Fontão, S.

Marczak, C. Werner, C. Alves, T. Conte, and R.

Prikladnicki. Quality in Software Ecosystems:

Challenges and Research Opportunities. In

Proceedings of VIII Workshop on Distributed Software

Development, Software Ecosystems and Systems-of-

Systems, pages 41–44, 2014 (in Portuguese).

K. Manikas. Revisiting software ecosystems research: A

longitudinal literature study. Journal of Systems and

Software, 117:84 – 103, 2016.

I. Carvalho, F. Campos, R. Braga, José M. David, V.

Stroele, Marco A. Araujo. HEAL ME - An Architecture

for Health Software Ecosystem Evaluation. In

Proceedings of the 5th International Workshop on

software Engineering for Systems-of-Systems and 11th

Workshop on Distributed Software Development,

Software Ecosystems and Systems-of-System. In ICSE.

IEEE. 2017.

B. A. Kitchenham. Procedures for performing systematic

reviews. Keele, UK, Keele University, 33(2004):1–26,

2004.

M. Petticrew and H. Roberts. Systematic reviews in the

social sciences: A practical guide. John Wiley & Sons,

2008.

V. Basili, G. Caldeira, H. Rombach. Goal Question Metric

Approach, Encyclopedia of Software Engineering, John

Wiley & Sons, Inc., p. 528-532, 1994.

S. Jansen. "Measuring the health of open source software

ecosystems: Beyond the scope of project health."

Information and Software Technology 56.11 (2014):

1508-1519.

D. Dhungana, I. Groher, E. Schludermann, and S. Biffl.

“Software ecosystems vs. natural ecosystems: Learning

from the ingenious mind of nature”. In Proceedings of

the Fourth European Conference on Software

Architecture: Companion Volume, ECSA ’10, pages

96–102, New York, NY, USA, 2010. ACM.

Iansiti, M, Levien, R (2004) The Keystone Advantage:

What the New Dynamics of Business Ecosystems Mean

for Strategy, Innovation, and Sustainability, Harvard

Business School Publishing Corporation, p 304.

Perry, D. E., Siy, H. P. and Votta, L. G. 1998. Parallel

changes in large scale software development: an

observational case study. International Conference on

Software engineering (ICSE) (Kyoto, Japan, Apr.

1998), 251–260.

Barbosa, O., Santos, R. P., Alves, C., Werner, C. and

Jansen, S. "A systematic mapping study on software

ecosystems from a three-dimensional perspective."

Software Ecosystems: analyzing and managing

business networks in the Software Industry (2013): 59-

81.

APPENDIX

Mapping References:

S. Jansen. How quality attributes of software platform

architectures influence software ecosystems. In

Proceedings of the 2013 International Workshop on

Ecosystem Architectures, WEA 2013, pages 6–10, New

York, NY, USA, 2013. ACM.

P. Schugerl, J. Rilling, R. Witte, and P. Charland. A quality

perspective of software evolvability using semantic

analysis. In Semantic Computing, 2009. ICSC ’09.

IEEE International Conference on, pages 420–427,

Sept 2009.

A. Hmood, I. Keivanloo, and J. Rilling. Se-Equam - an

evolvable quality metamodel. In Computer Software

and Applications Conference Workshops

(COMPSACW), 2012 IEEE 36th Annual, pages 334–

339, July 2012.

O. Franco-Bedoya, D. Ameller, D. Costal, and X. Franch.

Queso a quality model for open source software

ecosystems. In Software Engineering and Applications

(ICSOFT-EA), 2014 9th International Conference on,

pages 209–221, Aug 2014.

G. Stefanuto, M. Spiess, A. M. Alves, and P. F. D. Castro.

Quality in software digital ecosystems the users

perceptions. In Proceedings of the International

Conference on Management of Emergent Digital

EcoSystems, MEDES ’11, pages 85–88, New York, NY,

USA, 2011. ACM.

A. M. Alves and M. Pessôa. Brazilian public software:

beyond sharing. In Proceedings of the International

Conference on Management of Emergent Digital

EcoSystems, pages 73–80. ACM, 2010.

A. M. Alves, M. Pessoa, and C. F. Salviano. Proposal for a

framework for quality measurement to the spb–

brazilian public software. Business Process

Management Journal, 21(1):100–125, 2015.

Kajan, E., Lazić, L., & Maamar, Z. (2011, November).

Software engineering framework for digital service-

oriented ecosystem. In Telecommunications Forum

(TELFOR), 2011 19th (pp. 1320-1323). IEEE.

Lytra, I., Engelbrecht, G., Schall, D., & Zdun, U. (2015,

May). Reusable architectural decision models for

quality-driven decision support: A case study from a

smart cities software ecosystem. In Software

Engineering for Systems-of-Systems (SESoS), 2015.

ICEIS 2018 - 20th International Conference on Enterprise Information Systems

270

da Silva Amorim, S., McGregor, J. D., de Almeida, E. S.,

& von Flach G Chavez, C. (2014, August). Flexibility

in ecosystem architectures. In Proceedings of the 2014

European Conference on Software Architecture

Workshops (p. 14). ACM.

Franz, R. Z., Sawicki, S., Roos-Frantz, F., Yevseyeva, I., &

Emmerich, M. T. (2015). On using Markov decision

processes to model integration solutions for disparate

resources in software ecosystems. Springer.

Franz, R. Z., Sawicki, S., Roos-Frantz, F., Yevseyeva, I., &

Emmerich, M. T. (2015). On using Markov decision

processes to model integration solutions for disparate

resources in software ecosystems. Springer.

Health Evaluation in Software Ecosystems

271