Find the Way in the Jungle of Quality of Service in Industrial Cloud: A

Systematic Mapping Study

Malvina Latifaj, Federico Ciccozzi

and S

everine Sentilles

School of Innovation, Design and Engineering, M

alardalen University, Sweden

Keywords:

Industrial Cloud Computing, Quality of Service, Quality Attributes, Systematic Mapping Study.

Abstract:

The rapid development of Industry 4.0 and Industrial Cyber-Physical Systems is leading to the exponential

growth of unprocessed volumes of data. Industrial cloud computing has great potential for providing the

resources for processing this data. To be widely adopted, the cloud must ensure satisfactory levels of Quality

of Service (QoS). However, the lack of a standardized model of quality attributes hinders the assessment

of QoS levels. This paper provides a comprehensive systematically deﬁned map of current research trends,

results, and gaps in quality attributes and QoS in industrial cloud computing. An extract of the main insights is

as follows: (i) the adoption of cloud technologies is closely related to performance indicators, however other

quality attributes, such as security, are not considered as much as they should; (ii) solutions are most often not

tailored to speciﬁc industrial application domains; (iii) research largely focuses on providing solutions without

solid validation, unsuitable for effective and fruitful technology transfer.

1 INTRODUCTION

Industrial cloud computing, which aims to provide in-

dustrial digital information integration and collabo-

ration between enterprises based on a shared under-

standing of concepts (Wlodarczyk et al., 2009), is

considered to be a favorable solution to cope with lim-

itations of hardware and software for multiple enter-

prises. Dependability and software quality are crucial

factors in the success of industrial applications (Guo

and Deze, 2019). Thus, appropriate quality of service

(QoS) level must be attained for cloud platforms to

supply the necessary resources for processing indus-

trial data.

However, the quality attributes needed to assess

QoS have no formal deﬁnition and there does not ex-

ist a complete list of them (Chung et al., 2012; Mai-

riza et al., 2010). There have been some attempts to

the standardization of these attributes, but the clas-

siﬁcation schemes are inconsistent with each other

terminologically and also categorically, and therefore

their deﬁnitions still lack consensus. The absence of

a generic model of quality attributes that can be used

as a reference for speciﬁc domains leads to the lack

of a fundamental set of quality attributes to be used

to evaluate QoS in industrial cloud computing, which

https://orcid.org/0000-0002-0401-1036

https://orcid.org/0000-0003-0165-3743

is necessary due to the peculiarities of the latter. The

existing literature does not provide the necessary ev-

idence that could help in identifying the appropriate

quality attributes to evaluate QoS in industrial cloud

computing. To ﬁll this gap, we conducted a system-

atic mapping study in the context of QoS in industrial

cloud computing. The overall goal was to identify and

classify the quality attributes that are used to evaluate

QoS in industrial cloud computing and the ways to as-

sess them. The main contributions of this study are

as follows:

• an overview of the publication trends on the topic;

• a classiﬁcation of the quality attributes used to

evaluate QoS in industrial cloud computing and

the quality metrics used to assess them;

• a classiﬁcation of the factors that have an impact

on QoS in industrial cloud computing;

• a classiﬁcation of the domains in which QoS in

industrial cloud computing is investigated;

• a classiﬁcation of the primary studies’ objectives

and a description of the means by which these ob-

jectives are achieved.

The target audience of this paper includes re-

searchers and practitioners in the ﬁeld of cloud com-

puting, seeking a better insight on the QoS specif-

ically dealing with quality attributes, or willing to

adopt cloud technologies but needing a reference for

the evaluation of related QoS.

Latifaj, M., Ciccozzi, F. and Sentilles, S.

Find the Way in the Jungle of Quality of Service in Industrial Cloud: A Systematic Mapping Study.

DOI: 10.5220/0010380401510160

In Proceedings of the 11th International Conference on Cloud Computing and Services Science (CLOSER 2021), pages 151-160

ISBN: 978-989-758-510-4

 2021 by SCITEPRESS – Science and Technology Publications, Lda. All r ights reserved

151

The remainder of this paper is organized as fol-

lows. Section 2 provides a background on QoS and

industrial cloud computing and Section 3 presents the

study design. Section 4 reports on the vertical anal-

ysis results, while Section 5 reports on the horizontal

analysis results. Section 6 provides a discussion of the

main results and Section 7 discusses threats to valid-

ity. Section 8 introduces the related work and Section

9 concludes the paper.

2 BACKGROUND

Cloud computing is a cutting-edge technology that

makes possible the sharing of resources between mul-

tiple users. The popularity that cloud computing has

obtained over the last decades, attributable to the

advantages it provides compared to traditional ap-

proaches, has stimulated consumers to adopt it. More-

over, during the last few years, the digital informa-

tion produced by industry has increased rapidly and

the need for collaboration mechanisms is higher. En-

terprises try to handle this ﬂow of information us-

ing their existing structures, but they have bound-

aries on their capabilities. Even though cloud com-

puting can tackle the complexity of these collabora-

tive approaches, none of its deployment models can

overcome all the occurring obstacles. For that rea-

son, the concept of Industrial Cloud Computing was

introduced as a solution in the form of a platform

to exchange, process, and analyze digital informa-

tion (Wlodarczyk et al., 2009). But, to stay compet-

itive as technology evolves, a certain QoS should be

developed to meet the customers’ expectations. QoS

in cloud computing indicates “the levels of perfor-

mance, reliability, and availability offered by an ap-

plication and by the platform or infrastructure that

hosts it” (Ardagna et al., 2014, p. 1), and its main ob-

jective is to make the services acceptable for users.

Dromey (1995) states that to increase the software

quality, there should be a deﬁned quality model that

can make clear and direct links amongst the high-level

quality attributes and speciﬁc product characteristics.

Quality attributes are indeed fundamental elements

of quality models and have a crucial impact on soft-

ware development, but they have been left in the back-

ground compared to functional requirements, and the

results are scattered (Chung et al., 2012). This leads

to quality attributes not being addressed correctly and

software of poor quality. The reasons as to why qual-

ity attributes, which are so pivotal to the quality of

software systems, can be so hard to address correctly

are as follows: (i) their nature is subjective as different

stakeholders interpret and evaluate them differently,

leading to ambiguity, (ii) their achievement is rela-

tive, as there might always be different ways to reach

a satisfactory level, (iii) they can be interactive and

dependent on one another, so there can not be a local-

ized solution. This set of issues makes it difﬁcult to

deal with and even more challenging to measure and

verify quality attributes. To objectively measure these

quality attributes, we use quality metrics (Fernando

et al., 2014). Through this quantitative measurement,

the actual QoS can be compared to the expected QoS

and future steps for improvement can be determined.

3 STUDY DESIGN

In this section, we present the design of our study. To

form a good understanding of the state-of-the-art on

QoS in industrial cloud computing with a focus on

quality attributes, we carry out a systematic mapping

study according to the guidelines of Petersen et al.

(2015).

3.1 Research Goal and Questions

The fundamental goal of this study is to: achieve a

classiﬁcation of the quality attributes addressed the

most in industrial cloud computing and to provide

researchers and practitioners with a mapping of key

QoS attributes and methods to assess them. This

will enable them to make informed decisions in the

context of QoS for industrial cloud computing and

will highlight research gaps, laying the groundwork

for future research. The research questions (RQx) to

be answered by this study are the following:

RQ1. What Are the Publication Trends Regarding

the Quality of Service in Industrial Cloud Com-

puting Research?

Rationale and outcome: Identify the existing state of

research on QoS in industrial cloud computing and

assess the density of scientiﬁc publications.

RQ2. What Are the Most Addressed Quality At-

tributes and Related Aspects in Industrial Cloud

Computing Research?

Rationale and outcome: The concept of QoS is rather

broad and it includes a multitude of quality attributes,

therefore it is important to identify (i) a set of soft-

ware quality attributes addressed in industrial cloud

computing, (ii) a set of factors that affect quality at-

tributes and the overall QoS, and (iii) the industrial

domains in which cloud computing is used.

CLOSER 2021 - 11th International Conference on Cloud Computing and Services Science

152

RQ3. What Are the Most Common Quality

Metrics Used to Assess Quality Attributes in

Industrial Cloud Computing Research?

Rationale and outcome: Identify the metrics used

to assess the quality attributes in industrial cloud

computing and map them to the latter.

RQ4. What Are the Research Objectives of the

Primary Studies in Relation to QoS in Industrial

Cloud Computing and How Are These Objectives

Achieved?

Rationale and outcome: Obtain a classiﬁcation of the

objectives of the primary studies in relation to QoS,

and ways on how these objectives can be achieved.

3.2 Search and Selection

The search and selection process is a decisive multi-

step process that provides wide coverage of the topic

under investigation. Figure 1 illustrates the complete

process and provides information on the size of our

corpus during the entailed four steps.

Figure 1: Search and selection process.

Step 1: Initial Search. To obtain the initial set

of primary studies, automatic searches were carried

out on two of the largest and most complete elec-

tronic databases in software engineering that are well-

established and include a wide spectrum of peer-

reviewed publications: IEEE Xplore Digital Library

and SCOPUS. The search was performed considering

the title, abstract, and keywords of studies and it was

conducted on 25 February 2020.

The search string is constructed using the

PICOC Petticrew and Roberts (2008) criteria and it is

the following:

“cloud” AND “industr*” AND (“quality of ser-

vice” OR “QoS” OR “quality model” OR “software

qualit*” OR “quality propert*” OR “quality at-

tribut*” OR “non functional” OR “extra functional”

OR “NFR” OR “EFR” OR “NFP” OR “EFP”)

Step 2: Merging and Impurity Removal. A

multi-database search provides a broader spectrum

of studies and minimizes publication bias, but simul-

taneously leads to duplicates and repetitive studies.

For IEEE Xplore and Scopus, we identiﬁed 193

duplicates based on an exact match of title, authors,

and publication year; we removed the duplicates. No

studies with different versions were identiﬁed.

Step 3: Application of Selection Criteria. The

selection criteria are crucial elements to the selection

process and therefore, an explicit deﬁnition of these

criteria should be provided to allow the replication

of the study. An initial set of selection criteria was

deﬁned and the papers were screened based on title,

abstract, and keywords. However, there was a set of

150 primary studies that were uncertain with respect

to the inclusion criteria. The inclusion criteria were

reﬁned and the ﬁnal criteria are introduced in the

following paragraph.

Inclusion Criteria

I1. The study reports on an approach that aims to

deﬁne, measure, analyze, evaluate or improve QoS in

the scope of cloud computing.

I2. The study investigates quality attributes, quality

metrics, or quality models and their impact on QoS

in cloud computing.

I3. The study reports on the use or need of cloud

technologies in industry (e.g., banking) or it is

published at an industrial venue.

I4. The study is written in English and is available in

full text.

I5. The study is peer-reviewed.

Exclusion Criteria

E1. The study does not focus on any speciﬁc quality

attribute.

E2. Secondary or tertiary study.

E3. The study is published as tutorial paper, short

paper, poster paper, editorial, book, keynote, tutorial

summary, tool demonstration, panel discussion, tech-

nical report, or other non-peer-reviewed publications.

E4. The study is not in the computer science context.

Step 4: Backward Snowballing. The initial

corpus of primary studies obtained after the ap-

plication of the selection criteria is complemented

with a fully-recursive backward snowballing activity,

according to the guidelines by Wohlin (2014). Out of

790 results, only eight publications were included in

our corpus for a total of 42 primary studies that can

be found in the appendix in the replication package

The main reasons for the exclusion of publications

were as follows: i) the publications retrieved from

the reference list of a speciﬁc study, do not report on

QoS, or quality attributes and metrics, therefore not

https://bit.ly/2LvkORL

Find the Way in the Jungle of Quality of Service in Industrial Cloud: A Systematic Mapping Study

153

satisfying I1 and I2, ii) the publications do not report

on the use or need of cloud technologies in industry,

therefore not satisfying I3.

3.3 Data Extraction and Synthesis

As a means of extracting and classifying the rele-

vant information from the primary studies, we con-

structed the classiﬁcation framework illustrated in Ta-

ble 1. The parameters of the classiﬁcation frame-

work were deﬁned based on (i) the research questions,

and (ii) the keywording method. After the deﬁnition

of the classiﬁcation framework, the full text of the

primary studies was read to collect information ac-

cording to the categories of the deﬁned framework,

and additional information that did not ﬁt in any cat-

egory but was considered relevant. The categories

with the most additional information were the qual-

ity attributes, quality metrics, and means of reaching

the objectives. These categories had to be reﬁned, as

the keywording process’s initially deﬁned parameters

were not representative enough of the primary studies.

At the end of this process, no studies were excluded.

The data synthesis activity is conducted according

to the recommendations by Cruzes and Dyb

a (2011),

and it is divided into two main phases: vertical analy-

sis in which we analyze the extracted data from the

primary studies in order to collect information re-

garding each deﬁned parameter of our classiﬁcation

framework by applying the line of argument synthe-

sis recommended by Wohlin et al. (2012), and hor-

izontal analysis in which we cross-tabulate the data

among the different categories deﬁned for each re-

search question, to ﬁnd similarities, noteworthy dif-

ferences, and recurring patterns.

4 VERTICAL ANALYSIS

RESULTS

The purpose of the vertical analysis is to provide

quantitative results regarding each category in iso-

lation. These results are provided in the following

subsections in a textual and graphical form.

RQ1 – Publication Trends.

Regarding the distribution of primary studies by

type of venue, the most common publication type is

conference papers (21), followed by journal papers

(15) and workshop papers (2). Four primary studies

pertain to industrial venues (P1, P2, P4, and P5),

and only one venue hosts more than one primary

study (P1 and P5 published at IEEE Transactions on

Industrial Informatics).

Moreover, according to the extracted data, the main

contribution types are model (18) and method (18),

with the rest distributed as follows: framework (4),

tool (2), process (1) and metric (1).

With respect to the research type, among our

primary studies, there were only occurrences of

solution proposal (40) and evaluation research (3)

papers, where the majority were solution proposals

that proposed novel ways on how to deﬁne, measure,

analyze, evaluate, or improve QoS in industrial cloud

computing. P28 and P37 are evaluation research

papers, while P38 is both a solution proposal and an

evaluation research paper.

In terms of validation, the majority of primary

studies validate their approaches by means of eval-

uation (24), but these approaches are not validated

formally nor empirically. A set of 11 primary

studies conducts a more extensive validation of

the proposed approaches through analysis, while a

smaller set of primary studies focus on validation

by example (4). Only two primary studies fall in

the persuasion validation category and only one

primary study falls in the blatant assertion category.

The absence of experience papers may suggest a

lack of maturity in this research area, as existing so-

lutions have not been validated by the community yet.

RQ2 – Quality Attributes and Related As-

pects.

Quality Attributes. Figure 2 illustrates the de-

tails of our quality attributes categorization by

presenting the quality attributes mentioned in more

than one primary study. Our investigation identiﬁed

18 considered quality attributes in total. Nine of them

are mentioned only once through all the primary

studies and seven of these attributes are mentioned

in P35, which is also the study that identiﬁes most

quality attributes (14/18). This paper contributes

with a framework that aims to evaluate the quality

of cloud services by systematically measuring QoS

attributes and using them as a basis for ranking

cloud providers. The majority of the primary studies

addresses performance as a relevant quality attribute

in industrial cloud computing. The overall results

indicate a signiﬁcant gap between performance and

all other quality attributes. This suggests that the

adoption of cloud technologies in industry is very

related to the performance indicators offered by these

technologies.

Factors: Figure 3 provides an illustration of the

factors that affect quality attributes. Being that

cloud technologies operate on the grounds of virtu-

CLOSER 2021 - 11th International Conference on Cloud Computing and Services Science

154

Table 1: Classiﬁcation framework overview.

RQ Category Possible values

RQ2 Quality

attributes

performance (P), security (SE), efﬁciency (E), reliability (RL), availability (A), scalability (SC), us-

ability (U), elasticity (EL), stability (ST), privacy (PR), trust (T), responsiveness (RS), transparency

(TP), interoperability (I), adaptability (AD), sustainability (SUS), suitability (SUI), accuracy (AC)

Factors virtualization (V), data storage (DS), network (N), energy (E), other (O)

Domain gaming (G), mobile (M), power trading (PT), big data (BD), sensor cloud (SC)

RQ3 Quality

metrics

response time (RT), resource utilization (RU), make span (MS), latency (L), request completion

time (RCT), waiting time (WT), CPU utilization (CU), failure rate (FR), throughput (TH), execution

time (ET), rejection probability (RP), request arrival rate (RAR), delay (D), jitter (J), performance

deviation (PD), mean time to failure (MTTF), packet delivery ratio (PDR), guarantee ratio (GR),

drop rate (DR), time to adapt to upgrades (TAU), access rate (AR), request loss probability (RLP),

scheduling time (ST), system overhead rate (SOR), mean time between failures (MTBF) , bandwidth

(B), deadline miss ratio (DMR), job loss ratio (JLR)

RQ4 Objectives improve (I), evaluate (E), select provider (SP), measure (M), deﬁne (D)

Means scheduling (S), security mechanisms (SM), load balancing (LB), error-prone conditions (EPC), al-

location (A), edge servers (ES), queuing model (QuM), request trafﬁc control (RTC), workload

variability (WV), SPs coalition (SPC), ranking cloud services (RCS), quality model (QlM), dynamic

optimal routing (DOR), VM live migration (LM)

Figure 2: Quality attributes.

alization, it is not surprising that quality attributes

are frequently affected by virtualization solutions.

Virtualization is followed by data storage archi-

tecture which is not surprising either, considering

the fact that enterprises use cloud to store sensitive

data, and that raises numerous security concerns.

Furthermore, six studies mention that cloud providers

often have to trade-off between offered levels of QoS

and energy consumption. Network architecture is

also mentioned in six primary studies and is mostly

investigated in cloud gaming, because the majority

of computational operations are performed in the

cloud, and, in high-action games, network latency is

the greatest concern. A total of four primary studies

(P1, P4, P11, P35) fall into the other category as they

focus on providing quality models and mechanisms

to rank service providers and are not affected by

speciﬁc factors.

Domains. From the set of 42 primary studies, 32

of them investigate QoS in cloud computing by

providing insights on the use or need of cloud tech-

nologies in industry, but without a clear indication of

speciﬁc industrial domains. The other ten primary

studies identify ﬁve industrial domains, as illustrated

in Figure 4. Overall, the results indicate that even

though cloud technologies are very beneﬁcial to

Figure 3: Factors. Figure 4: Domains.

various industries and enterprises, research on their

use in speciﬁc industrial domains still lacks maturity.

RQ3 – Quality Metrics.

Our investigation identiﬁed 28 relevant quality

metrics. Figure 5 illustrates the quality metrics that

are mentioned in more than one primary study. Top

three quality metrics are response time, followed by

resource utilization and make span. Response time

is repeatedly emphasized for the impact it has on

the performance of cloud technologies. Thus, being

that performance is the quality attribute mentioned

the most in our primary studies this result is not

surprising. It is worth mentioning that, while all

other studies only suggest the use of speciﬁc metrics,

P35 proposes new approaches for assessing quality

attributes through an extensive set of formulas, but

with no speciﬁc terminology regarding the quality

metric, thus they are not included as part of the

quality metrics list.

RQ4 – Objectives and How to Reach Them.

Objectives. Regarding the objectives of primary

studies the results indicate that the most common

objective is to improve the QoS offered by the service

Find the Way in the Jungle of Quality of Service in Industrial Cloud: A Systematic Mapping Study

155

Figure 5: Quality metrics.

providers as it is crucial for the adoption of cloud

technologies. Even though a considerable difference

is observed between the number of studies that try to

improve and evaluate, research efforts on the latter do

not go unnoticed. A total of nine primary studies fall

into this category. Three primary studies investigate

approaches on how to select the most suitable cloud

provider according to speciﬁc requirements and only

two primary studies fall into the deﬁne category,

where the authors aim to deﬁne quality attributes

and a quality model. The low number of primary

studies in this category highlights the fact that the

community is still lacking agreement when deﬁning

QoS and quality attributes.

Figure 6: Objectives.

Means. Our investigation identiﬁed 14 approaches

that can help researchers achieve the objectives spec-

iﬁed above. As shown in Figure 7, the ﬁve top ap-

proaches are: scheduling, security mechanisms, load

balancing, error-prone conditions, and allocation.

These results imply that the most effort has been put

into scheduling, which is not surprising since, with

the increasing amount of data that is being processed

in the cloud, effective scheduling mechanisms are

crucial to providing satisfactory levels of QoS. Fur-

thermore, being that security was the second most

mentioned quality attribute in industrial cloud com-

puting (see Figure 2), it was expected that the secu-

rity mechanisms would be ranked among the top ap-

proaches.

Figure 7: Means of reaching the objectives.

5 HORIZONTAL ANALYSIS

RESULTS

The purpose of the horizontal analysis is to iden-

tify possible connections between the categories

of our classiﬁcation framework. To do that, we

cross-tabulated the extracted data across different

categories and analyzed the relevant insights.

Quality Attributes and Metrics. Figure 8 provides

evidence on the quality metrics that are used to assess

speciﬁc quality attributes. We have only included the

metrics that have more than one occurrence among

the primary studies and the quality attributes assessed

by them. Only ﬁve quality attributes are assessed by

quality metrics that are mentioned in more than one

primary study. The majority of quality metrics are

used to assess performance and only a few metrics

are used to assess other quality attributes. This was

in part expected, considering the signiﬁcant gap in

occurrence between performance and other quality

attributes identiﬁed during the vertical analysis.

Figure 8: Quality attributes and metrics.

Factors and Quality Attributes. Figure 9, provides

evidence that performance is mostly affected by the

virtualization solutions of the cloud provider. Dif-

ferent virtualization solutions provide varying levels

of performance due to the technologies and strategies

used. Another factor that has a signiﬁcant impact on

CLOSER 2021 - 11th International Conference on Cloud Computing and Services Science

156

performance is the network infrastructure, as a poor

infrastructure is bound to negatively affect the appli-

cation’s performance. Furthermore, security is mostly

affected by the data storage architecture. Considering

the higher exposure that data encounters on the cloud

than on-premises, it is surprising to see that only one

primary study mentions the impact that the virtual-

ization solutions have on security. This result sug-

gests the lack of research on how to mitigate security

threats when providing virtualization solutions. Ef-

ﬁciency is affected by the energy management solu-

tions offered by the cloud provider, as the high con-

sumption of energy for cloud data centers is raising

economic and environmental concerns.

Figure 9: Factors and quality attributes.

Objective of Study and Means of Reaching It. Fig-

ure 10 illustrates the relation between the objectives

of primary studies and means of reaching them. From

the results it can be seen that the most mentioned ap-

proach to improve QoS is scheduling, followed by

security mechanisms and load balancing. Through

scheduling, the resources can be utilized efﬁciently,

and the overall performance of the system can expe-

rience a signiﬁcant improvement. In order to evalu-

ate QoS, the majority of primary studies use simula-

tion of error-prone conditions, such as VM and server

failure, followed by edge servers, queuing model and

workload variability. The most common approach to

help select a provider is by ranking the cloud services

based on the users’ requirements and providers’ capa-

bilities. This approach is also used to measure QoS of

different cloud providers.

Contribution Type and Validation Type. Figure

11 illustrates a high concentration of primary studies

that use evaluation to validate the proposed models

and methods. As a consequence, the scientiﬁc ev-

idence that they provide cannot be considered par-

ticularly strong. However, models surpass methods

when it comes to more extensive and systematic val-

idation, such as analysis. This suggests that mod-

els have been provided with higher quality evidence

Figure 10: Objectives and means.

in terms of their applicability. Metrics, tools, and

processes have only been validated through analysis,

while frameworks have an even distribution with one

occurrence in each category, except for blatant asser-

tion.

Figure 11: Contribution and validation type.

6 DISCUSSION

In order to describe and interpret the signiﬁcance of

our ﬁndings, this section discusses the obtained re-

sults of the vertical and horizontal analysis introduced

in the previous section.

Regarding publication trends (RQ1), the major-

ity of primary studies are published at conferences,

followed by journals and workshops, which is com-

mon in the software engineering literature. Further-

more, the research on this topic can be categorized as

solution-seeking, with ∼ 95% of the primary studies

being solution proposals. Among them, the major-

ity contribute with a model or method. We believe

that more effort should be put on the evaluation and

validation of solutions in industry. More speciﬁcally,

more attention should be put on formal or empirical

validation to properly assess the limitations and ben-

eﬁts of the proposed solutions.

Regarding quality attributes (RQ2), the results

provide evidence that one particular quality attribute

Find the Way in the Jungle of Quality of Service in Industrial Cloud: A Systematic Mapping Study

157

is addressed more frequently than others: perfor-

mance, addressed in ∼ 80% of the primary stud-

ies. More effort should be directed to other qual-

ity attributes since desired levels of QoS are usu-

ally achieved through a combination of quality at-

tributes (Software & Systems Engineering Standards

Committee, 1998). Furthermore, different primary

studies report on different factors that affect quality

attributes and, consequently, the overall QoS. Virtual-

ization is covered in more than ∼ 50% of the primary

studies, and it is mostly mentioned as a factor affect-

ing performance. This result was expected as virtu-

alization is a ground technology that enables the use

of cloud computing and improves the utility of phys-

ical machines. Furthermore, a notable number of pri-

mary studies investigate data storage architecture as

a factor affecting QoS and more speciﬁcally, security.

This was expected as numerous enterprises are mov-

ing their data to the cloud because of its simplicity,

but fear the security aspect. We believe additional re-

search must be conducted in how virtualization solu-

tions affect security because each virtualization layer

can be a target to malicious attacks and compromise

the entire cloud infrastructure. Moreover, our study

revealed that only ∼ 24% of the primary studies re-

ported on QoS with respect to a particular domain,

therefore, it is our belief that more research efforts

should be spent on investigating QoS in speciﬁc in-

dustrial domains.

Regarding quality metrics (RQ3), the majority

of primary studies focuses on three quality metrics:

response time, resource utilization and make span.

These results do not imply that the remaining quality

metrics have not received enough attention from the

research community, but only that they have a more

restricted application in the assessment of quality at-

tributes mentioned in the selected primary studies.

The aforementioned quality metrics are, to a large ex-

tent, used to assess performance. However, ∼ 18% of

quality metrics have two occurrences in the primary

studies, while ∼ 43% have only one. Furthermore,

only one primary study contributed by proposing met-

rics for the assessment of speciﬁc quality attributes.

This makes us believe that more research should be

conducted on the deﬁnition of more speciﬁc and en-

hanced quality metrics. Moreover, the mapping be-

tween quality attributes and quality metrics showed

that ∼ 67% of the quality attributes mentioned in the

primary studies were not assessed by any quality met-

ric. We foresee a need for further research targeting

these quality attributes with the goal of proposing and

identifying quality metrics to assess them.

Regarding objectives (RQ4) of the primary stud-

ies, we found that the majority were interested in

improving QoS. We could discuss how the research

community could have focused more on other ob-

jectives such as measuring or deﬁning QoS, rather

than on its improvement, but being that QoS is cru-

cial for the adoption of cloud technologies, we un-

derstand the rationale behind this. Moreover, the ap-

proaches that strive to improve QoS involve the wide

use of scheduling techniques and mechanisms. How-

ever, our suggestion would be to focus on providing

scheduling solutions that take into consideration mul-

tiple QoS attributes rather than a single one. Further-

more, we observed a noteworthy focus on using secu-

rity mechanisms to improve QoS.

7 THREATS TO VALIDITY

External Validity. One major threat that could limit

the generalizability of our study is having a set of

primary studies that is not representative enough of

the research on this topic. To mitigate this threat, we

conducted an automatic search on two of the largest

and most complete electronic databases in software

engineering, which was complemented with a fully-

recursive backward snowballing activity to eventually

enrich the initial corpus of primary studies. Another

threat that can jeopardize the external validity of our

study is the exclusion of primary studies in languages

other than English. Regardless, being that English is

the de-facto standard language used for scientiﬁc pa-

pers, this threat is negligible. Moreover, we only tar-

geted peer-reviewed publications as they are expected

to provide scientiﬁc work of certain quality assessed

by peers, thus excluding gray literature.

Internal Validity. To mitigate biases regarding the

degree of inﬂuence of external variables on the de-

sign of the study, we rigorously deﬁned and validated

a detailed study protocol that follows the guidelines

proposed by Petersen et al. (2015), Kitchenham and

Charters (2007), and Wohlin et al. (2012). Further-

more, we deﬁned a classiﬁcation framework for the

extraction of data from the primary studies, that was

iteratively revised. The keywording process was also

used to transform qualitative data into quantitative

data, while quantitative data was analyzed using de-

scriptive statistics, making the data analysis validity

threats minimal. Finally, a complete replication pack-

age

is available for the independent replication of our

study.

Construct Validity. To mitigate the threat of hav-

ing a set of primary studies that is not representa-

tive enough of the population deﬁned by the research

questions, we conducted an automatic search on two

electronic databases and complemented the search

CLOSER 2021 - 11th International Conference on Cloud Computing and Services Science

158

with a fully-recursive backward snowballing activity.

Furthermore, we conducted preliminary searches and

reﬁned the search string based on the analysis of a set

of sample studies. Finally, all relevant studies were

screened according to deterministic and clear selec-

tion criteria.

Conclusion Validity. Our study has been con-

ducted using widely accepted systematic methods

documented in the study design. The study design

and the replication package can be used by other re-

searchers to verify and replicate our work. We are

aware that they may design a classiﬁcation framework

with different categories and data items than ours,

thus leading to different results. However, we miti-

gated this potential threat by: (i) documenting how we

constructed the classiﬁcation framework, (ii) reﬁning

the categories and data items throughout the entire

data extraction process, and (iii) making the replica-

tion package publicly available for transparency and

replicability.

8 RELATED WORK

This section discusses other existing systematic stud-

ies related to QoS in cloud computing. However, as

to our best knowledge, there is no existing systematic

mapping study on QoS in industrial cloud computing

with a focus on quality attributes. Therefore, in the

following, we provide only an overview of the few

existing studies related to our topic.

Lehrig et al. (2015) conducted a systematic lit-

erature review in the cloud computing context with

respect to three quality attributes (scalability, elas-

ticity, and efﬁciency), to recommend relevant def-

initions and metrics. Their ﬁndings can be used

to evaluate the QoS of cloud technologies and as a

starting point to derive new quality metrics. Ab-

delmaboud et al. (2015) conducted a systematic map-

ping study to offer insights on the state-of-art of QoS

approaches in cloud computing. Most of the studies

focus on Infrastructure-as-a-Service and Software-as-

a-Service. The challenges and gaps identiﬁed in the

study conﬁrm that more research must be conducted

on QoS approaches. Prakash et al. (2019) conducted

a literature review and a comparative study of QoS

management techniques in cloud computing. They

present a list of eight quality attributes and their re-

spective metrics that can help with the improvement

of performance in cloud systems and compare various

QoS techniques. Nevertheless, they state that there

are still lots of possibilities on how QoS in cloud com-

puting can be further enhanced.

The related works presented in this section high-

light the fact that the existing research conducted

at the intersection of cloud computing and QoS is

generic and does not focus on the industrial side of

cloud computing, which we instead focus on.

9 CONCLUSIONS

This paper reports on a systematic mapping study

on QoS in industrial cloud computing with the goal

of identifying and classifying the most commonly

addressed quality attributes and related metrics, and

highlighting the research gaps. The initial set of po-

tentially relevant studies consisted of 1063 publica-

tions. The selection of the primary studies through a

rigour and well-documented process led to a ﬁnal set

of 42 primary studies. For the extraction of relevant

information from the set of studies, we deﬁned and

incrementally reﬁned a classiﬁcation framework that

can also be used for future research. For data synthe-

sis, we carried out a vertical and horizontal analysis.

The main results of this study are the following:

• Research on QoS in industrial cloud computing

largely focuses on providing solution proposals

that contribute with novel models and methods.

• There is a need for more solid validation of pro-

posed solutions that can contribute to the maturity

of the research area and the adoption of these so-

lutions in practice.

• The adoption of cloud technologies in industry is

closely related to the performance indicators of-

fered by these technologies. However, research

on other quality attributes is quite limited.

• Despite the undeniable importance of security in

cloud technologies, the results suggest a lack of

research on security as a quality attribute and on

how to mitigate the security threats when provid-

ing virtualization solutions.

• The most mentioned quality metrics in industrial

cloud computing are response time, resource uti-

lization and make span, that to a large extent are

used to assess performance.

• The approaches are in most cases not targeting

explicitly a speciﬁc industrial domain, potentially

hampering the identiﬁcation of relevant quality at-

tributes and metrics for speciﬁc applications.

• There is strong scientiﬁc research on the im-

pact the virtualization solutions have on QoS

and speciﬁcally, performance, that can help cloud

providers make more informed QoS-aware archi-

tectural decisions.

• The focus of research in this area seems to be ori-

ented towards the improvement of QoS by using

Find the Way in the Jungle of Quality of Service in Industrial Cloud: A Systematic Mapping Study

159

better scheduling techniques to tailor quality at-

tributes according to users’ expectations.

To summarize, these conclusions provide evi-

dence that there have been research efforts on this

topic, but the focus of the current literature still leaves

open research challenges. Overall, we believe that our

study will be helpful to the research community for

the following reasons: (i) the catalog of quality at-

tributes and metrics can be useful for evaluating QoS

in industrial cloud computing, and (ii) the catalog of

factors affecting QoS can help make more informed

decisions regarding cloud architecture with respect to

how it affects speciﬁc quality attributes.

ACKNOWLEDGMENTS

This work was supported by Vinnova through the

ITEA3 BUMBLE project (rn. 18006).

REFERENCES

Abdelmaboud, A., Jawawi, D. N., Ghani, I., Elsaﬁ, A.,

and Kitchenham, B. (2015). Quality of service ap-

proaches in cloud computing: A systematic mapping

study. Journal of Systems and Software, 101:159–179.

Ardagna, D., Casale, G., Ciavotta, M., P

erez, J. F., and

Wang, W. (2014). Quality-of-service in cloud comput-

ing: modeling techniques and their applications. Jour-

nal of Internet Services and Applications, 5(1):11.

Chung, L., Nixon, B. A., Yu, E., and Mylopoulos, J. (2012).

Non-functional requirements in software engineering,

volume 5. Springer Science & Business Media.

Cruzes, D. S. and Dyb

a, T. (2011). Recommended steps for

thematic synthesis in software engineering. In 2011

international symposium on empirical software engi-

neering and measurement, pages 275–284. IEEE.

Dromey, R. G. (1995). A model for software product

quality. IEEE Transactions on software engineering,

21(2):146–162.

Fernando, W., Wijayarathne, D., Fernando, J., Mendis,

M., and Guruge, I. (2014). The importance of soft-

ware metrics: Perspective of a software development

projects in sri lanka. In SAITM Research Symposium

on Engineering Advancements.

Guo, S. and Deze, Z. (2019). Cyber-Physical Systems: Ar-

chitecture, Security and Application. Springer Inter-

national Publishing, 1st edition.

Kitchenham, B. A. and Charters, S. (2007). Guidelines

for performing systematic literature reviews in soft-

ware engineering technical report. Software Engineer-

ing Group, EBSE Technical Report, Keele University

and Department of Computer Science University of

Durham, 2.

Lehrig, S., Eikerling, H., and Becker, S. (2015). Scala-

bility, elasticity, and efﬁciency in cloud computing:

A systematic literature review of deﬁnitions and met-

rics. In Proceedings of the 11th International ACM

SIGSOFT Conference on Quality of Software Archi-

tectures, pages 83–92.

Mairiza, D., Zowghi, D., and Nurmuliani, N. (2010). An in-

vestigation into the notion of non-functional require-

ments. In Proceedings of the 2010 ACM Symposium

on Applied Computing, pages 311–317.

Petersen, K., Vakkalanka, S., and Kuzniarz, L. (2015).

Guidelines for conducting systematic mapping stud-

ies in software engineering: An update. Information

and Software Technology, 64:1–18.

Petticrew, M. and Roberts, H. (2008). Systematic reviews in

the social sciences: A practical guide. John Wiley &

Sons.

Prakash, S. et al. (2019). Review of quality of service based

techniques in cloud computing. In Data Science and

Big Data Analytics, pages 255–265. Springer.

Software & Systems Engineering Standards Committee

(1998). IEEE Std 1061-1998—IEEE standard for a

software quality metrics methodology. IEEE Com-

puter Society, Tech. Rep.

Wlodarczyk, T. W., Rong, C., and Thorsen, K. A. H. (2009).

Industrial cloud: Toward inter-enterprise integration.

In IEEE International Conference on Cloud Comput-

ing, pages 460–471. Springer.

Wohlin, C. (2014). Guidelines for snowballing in system-

atic literature studies and a replication in software en-

gineering. In Proceedings of the 18th international

conference on evaluation and assessment in software

engineering, pages 1–10.

Wohlin, C., Runeson, P., H

ost, M., Ohlsson, M. C., Reg-

nell, B., and Wessl

en, A. (2012). Experimentation in

software engineering. Springer Science & Business

Media.

CLOSER 2021 - 11th International Conference on Cloud Computing and Services Science

160