Evaluation of Non-Functional Requirements for IoT Applications
Joseane O. V. Paiva
a
, Rossana M. C. Andrade
b
and Rainara Maia Carvalho
c
Group of Computer Network, Software Engineering and Systems, Federal University of Cear
´
a, Fortaleza, Brazil
Keywords:
Internet of Things, Software Quality, Non-Functional Requirements, Systematic Mapping.
Abstract:
Internet of Things (IoT) is a paradigm that enables physical objects to interact and to work together. IoT
applications have particular characteristics, such as context-awareness, interconnectivity, and heterogeneity,
and particular types of interaction, user interaction with devices (called human-thing interaction), and the
interaction between devices (called thing-thing interaction). These characteristics represent the expectation
around the system and are also known as Non-Functional Requirements (NFRs). So, during the requirements
elicitation of such systems, they can appear as NFRs, and their combination often increases the complexity
of the IoT application development and evaluation. Thus, this work aims to identify which approaches and
NFRs have been considered in the literature to evaluate IoT applications and the main challenges faced by
the evaluators. We use the systematic mapping methodology to provide a comprehensive view of approaches,
methods, tools, and processes. As a result, we identified two tools, six approaches, one method, and one
process that can be used to evaluate NFRs, a set of 42 NFRs that can be considered for IoT applications, and
the main challenges related to the NFRs evaluation for the IoT applications.
1 INTRODUCTION
Internet of Things (IoT) can be defined as a paradigm
where intelligent objects interact in an environment
through a wireless connection, being able to cooper-
ate to provide services and to achieve common goals
(Atzori et al., 2010).
IoT applications are increasingly present in our
daily lives, on the streets, in the malls, at work, or
in our homes (Rowland et al., 2015). Users have
been adapted themselves to the presence of devices
that, through sensors, capture data about their envi-
ronment, their health, their behavior, and learned to
use the facilities provided by these solutions.
The growing presence around us of intelligent ob-
jects or things like smartphones, smartwatches, intel-
ligent lamps, among others, shows that IoT has be-
come a reality in our lives, being it in the industrial or
personal context.
On the other hand, the way users interact with
these solutions may differ from how s/he interacts
with traditional applications.
IoT applications deal with countless other forms
of interactions such as gestures, vocal commands,
a
https://orcid.org/0000-0003-1700-8765
b
https://orcid.org/0000-0002-0186-2994
c
https://orcid.org/0000-0002-4806-0542
or the total absence of user actions (Rowland et al.,
2015).
Besides the user interaction with devices (i.e.,
Human-Thing Interaction), IoT applications present
a new type of interaction, the Thing-Thing Interac-
tion, which is the interaction between intelligent ob-
jects. The connected devices relate without human
intervention to perform a task (Andrade et al., 2017).
Moreover, IoT applications have particular char-
acteristics, such as context-awareness, interconnectiv-
ity, and heterogeneity, that should be considered in the
requirements elicitation and evaluation.
In this scenario, due to the inherent complexity
of IoT applications, the traditional evaluation meth-
ods available in the literature may not cover all the
quality properties that need to be considered. These
quality properties or attributes are also known as non-
functional requirements (NFRs) (Uckelmann et al.,
2011).
NFRs can be defined as a description of a prop-
erty or characteristic that a system must exhibit or a
constraint that it must respect (Wiegers and Beatty,
2013).
Meeting the chosen NFRs is vital to the success
of the software (Chung and Prado Leite, 2009). How-
ever, ensuring that the system meets NFRs is not a
trivial task and there is a need to find out how NFRs
Paiva, J., Andrade, R. and Carvalho, R.
Evaluation of Non-Functional Requirements for IoT Applications.
DOI: 10.5220/0010461901110119
In Proceedings of the 23rd International Conference on Enterprise Information Systems (ICEIS 2021) - Volume 2, pages 111-119
ISBN: 978-989-758-509-8
Copyright
c
2021 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
111
of IoT applications have been evaluated.
Quality evaluation for IoT applications is a topic
that has received attention from the scientific commu-
nity. Kim (2016), for example, proposed a quality
model for IoT applications and specific measures for
this type of application.
Andrade et al. (2017) discuss how we can benefit
from the ubiquitous field of systems interaction eval-
uation to evaluate interaction with IoT applications,
focusing on both systems’ main differences and simi-
larities and, Bures et al. (2020) present a set of quality
characteristics for IoT applications which specifically
emphasizes the aspects of security, privacy, reliability
and usability.
However, we did not find out a work that summa-
rizes how the evaluation of NFRs for IoT applications
has been conducted. Thus, in order to contribute with
the NFR evaluation field, this work aims to provide
a comprehensive view on approaches, methods, tools,
and processes used for the evaluation of NFRs for IoT
applications as well as the main challenges faced by
the evaluators.
For this, we apply the systematic mapping
methodology, which is designed to give an overview
of a research topic by classifying and counting contri-
butions in relation to the categories of that classifica-
tion (Petersen et al., 2015).
The remainder of this paper is organized as fol-
lows: Section 2 provides a theoretical background,
discussing IoT challenges, and the non-functional re-
quirements. Section 3 describes our methodology;
Section 4 presents the results; in Section 5, we dis-
cuss the results; and in Section 6 we expose our final
considerations and future work.
2 BACKGROUND
This study aims to provide an overview of how NFRs‘
assessments have been conducted for the IoT applica-
tions. Therefore, we consider it appropriate to provide
a theoretical foundation regarding the IoT and NFRs
as follows.
2.1 Internet of Things
The term Internet of Things was introduced by Kevin
Ashton in 1999 and was associated with RFID tech-
nology (Ashton, 2009).
Since then, the purpose and use of this technol-
ogy have evolved, and today IoT can be defined as a
paradigm where smart objects interact in an environ-
ment through a wireless connection, being able to co-
operate in providing services and achieving common
goals (Atzori et al., 2010).
IoT can be considered an extension of Ubiquitous
Computing (Carvalho et al., 2020), which refers to
devices connected everywhere in such a transparent
way that we will not realize they are there (Weiser,
1991).
Due to their similarities, such as mobility and
context-awareness, the artifacts that come from Ubiq-
uitous Computing are suitable for IoT applications
(Andrade et al., 2017). Therefore, in this work, we
consider the NRFs focused on Ubiquitous Computing
as suitable for IoT applications.
On the other hand, IoT applications have also
some singularities, such as interconnectivity, hetero-
geneity, and services related to objects.
We present then the following list of IoT features
and their definitions (Patel et al., 2016):
Interconnectivity: In IoT applications, differ-
ent objects can be connected, sharing information
within a communication infrastructure.
Object-related Services: IoT applications can
provide services related to the objects that inte-
grate the solution, such as privacy protection and
semantic consistency between objects.
Heterogeneity: An IoT application is composed
of heterogeneous devices, both in aspects related
to hardware and aspects related to how they were
implemented. However, these devices can interact
with other devices or service platforms through
different networks.
Dynamicity: The state of devices and the context
changes dynamically. Devices can, for example,
be connected or disconnected, vary in location,
have their settings changed, among others. The
number of devices that makes up an IoT applica-
tion can also vary dynamically, depending on the
environment and other entities’ presence (for ex-
ample, objects and people).
Scalability: IoT applications provide the recog-
nition of each object that makes up a solution and
ensure communication between them. Since the
number of objects that makes up an IoT solution
can vary, the system must remain operational re-
gardless of the number of connected objects.
Security: IoT applications must be designed with
the security of their use in mind; this includes per-
sonal data and their user’s well-being.
Interconnectivity: Allows network access and
devices’ ability to consume and produce data
within an IoT solution.
ICEIS 2021 - 23rd International Conference on Enterprise Information Systems
112
Limitations related to these features diminish their
potential and impact usability (Rowland et al., 2015).
Therefore, it is critical to ensure that these properties
are performing as expected in IoT applications.
2.2 Non-Functional Requirements
Non-Functional Requirements (NFRs) can be defined
as requirements that are not directly related with spe-
cific functionalities offered by the software. They can
be related to the emerging properties of the system,
such as Reliability and Security (Sommerville, 2011).
According to (Wiegers and Beatty, 2013), quality
characteristics are also known as a type of NFRs and
they describe the product’s characteristics in various
dimensions considered important by the stakeholders,
such as security and usability.
Thus, the software quality assurance demands that
quality characteristics (or NFRs) are specified, mea-
sured, and evaluated, whenever possible, using vali-
dated or widely accepted measures and measurement
methods (ISO/IEC 25000, 2011).
Another important point to stress in this paper
concerning the evaluation of NFRs is that they can
be correlated, which means that one NFRs can has
a impact in another one. This impact could be pos-
itive (helps) or negative (hurts) (Wiegers and Beatty,
2013).
3 METHODOLOGY
As we mentioned in the introduction section, this re-
search aims to identify tools, methods, approaches,
and processes for evaluating NFRs in IoT applica-
tions.
To this end, we conducted a systematic mapping
of the literature (SML), following the guidelines pro-
posed by (Kitchenham and Charters, 2007), which di-
vides the research conduction into three stages: Plan-
ning (Section 3.1), Conduction (Section 3.2), and
Analysis and Reporting (Section 4).
3.1 Planning
In the first stage of the SML methodology, we set
up all the research parameters for conducting the re-
search in a document called the research protocol. In
this protocol, we defined the research questions, the
search strategy, the selection criteria, and the data to
be extracted.
3.1.1 Research Questions
According to the aim of this study, the following re-
search question was defined:
RQ1 - How Are the Non-Functional Requirements
Evaluations Conducted in IoT Applications?
To better answer this question, we had also de-
fined a set of secondary questions as follows:
SQ1 - Which are the approaches, tools, methods,
and processes used to evaluate non-functional re-
quirements in IoT applications?
SQ2 - Which are the non-functional requirements
observed when evaluating IoT applications??
SQ3 - Which are the challenges faced when eval-
uating non-functional requirements?
3.1.2 Search Strategy
The first step in selecting the papers that answered
our search questions was to conduct an automated
database search.
For this, we used PICO (Population, Intervention,
Comparison, and Outcomes), which is a strategy sug-
gested by Kitchenham and Charters (2007) to identify
keywords related to the defined search questions for
the search string.
Population: In software engineering, the popu-
lation can represent a specific software engineer-
ing function, an application area, or an industry
group. In our context, the population was defined
as IoT applications, and we also considered ubiq-
uitous and pervasive systems due to their similar-
ities;
Intervention: Intervention is defined with the
methodology, tool, technology, or software proce-
dure used to address a specific issue. In our case,
the intervention was defined as the non-functional
requirements;
Comparison: Comparison can be defined as the
methodology, tool, technology, or procedure with
which the intervention is being compared. For this
research, we do not define elements of compari-
son;
Outcome: Outcomes should relate to factors of
importance to practitioners in the study area. In
our case, we selected terms that represented eval-
uation approaches as outcomes.
For each PICO element, we also selected syn-
onyms to minimize the risk of restricting the search
results. Table 1 shows the terms chosen by each cate-
gory.
Evaluation of Non-Functional Requirements for IoT Applications
113
Table 1: Selecting search terms.
PICO Strategy
Population
(“Internet of Things” OR IoT OR
“Pervasive*” OR “Ubiquitous”)
Intervention
“non-functional requirement” OR
nfr OR “non-functional property”
OR “quality characteristic” OR
“quality attribute” OR
“quality requirement” OR
“extra-functional requirement” OR
“non-behavioural requirement” OR
“quality factor”)
Comparison No terms selected
Outcome
(evaluation OR assessment OR
verification OR validation)
AND (method OR tool OR
technique OR approach OR
process)
When connecting the search terms with logical
operators, we elaborated the following search string:
(“Internet of Things” OR IoT OR “per-
vasive” OR ubiquitous”) AND ( “non-
functional requirement*” OR nfr OR “non-
functional propert*” OR “quality character-
isti*” OR “quality attribute*” OR “qual-
ity requirement*” OR “extra-functional re-
quirement*” OR “non-behavioural require-
ment*” OR “non-behavioral requirement*”
OR “quality factor*” ) AND ( evaluation OR
assessment OR verification OR validation)
AND ( method* OR tool* OR technique* OR
approach* OR process*)
3.1.3 Research Sources Selection
We applied the search string in two databases that
were chosen due to their good coverage and stabil-
ity: Scopus
1
and Web of Science
2
. Therefore, Scopus
incorporates other important bases for the Software
Engineering field, such as ACM and IEEE.
In addition, we also performed a snowballing for-
ward procedure, which means identifying new papers
1
http://www.scopus.com/search
2
http://www.webofknowledge.com
based on a set of papers that cite the paper being ex-
amined (Wohlin, 2014).
3.1.4 Study Selection Criteria
We selected for this research, papers that present
tools, approaches, methods, or processes for evaluat-
ing non-functional requirements for IoT applications.
As for time range, once IoT was first cited in 1999
(Ashton, 2009), articles published between 2000 and
2019 were included in the current research. To ensure
the correct interpretation of our research results and
the replicability of the study, we limited our search to
papers written in English.
Regarding the exclusion criteria, we did not con-
sider eligible for this research papers that:
are secondary or tertiary works;
are books, technical reports, white papers, short
papers (less than 5 pages);
are not available for download; and
do not present tools, approaches, methods, or pro-
cesses of non-functional requirements evaluation
for IoT applications.
3.1.5 Data Extraction
After we select the studies, they were submitted to
the data extraction step that consists of exploring the
work selected to extract information that will help us
to answer our research questions.
Table 2 presents the data to be extracted in the se-
lected papers.
3.2 Conducting
This conducting stage was divided into two steps.
First, we applied the search string in the Web of Sci-
ence and Scopus databases. We selected the papers
through the study selection criteria. At the end of this
process, we obtained a set of approved papers. Then,
we used the selected papers to apply the snowballing
forward procedure (Wohlin, 2014).
3.2.1 Fist Step: Search on Databases Sources
We started by applying the string search to the se-
lected databases.
The search string returned 174 papers, 121 by
Scopus and 51 by Web of Science. After removing
49 duplicated studies, we obtained 125 papers for the
title/abstract screening. At this step, we excluded 88
papers based on our study selection criteria. Thus, we
got a set of 37 papers eligible for the entire reading.
ICEIS 2021 - 23rd International Conference on Enterprise Information Systems
114
Table 2: List of data to be extracted.
Data extraction form
Study metadata
Title, authors,
and year of publication.
Approaches,
methods, tools,
or process
Strategy adopted
to evaluate the NFRS.
Type of system
Ubiquitous System
or IoT application.
NFRs
The NFRs evaluated.
Evaluation focus
System in development,
system implemented
or both.
Challenges
The challenges on
evaluating NFRs
reported within the paper.
After downloading and reading the full text of the
37 papers, we excluded 27 papers that had not an-
swered our research questions. Thus, we got a set of
10 papers for data extraction.
3.2.2 Second Step: Snowballing Forward
To complement the search results through the
databases, we decided to perform the Forward Snow-
balling procedure, which consists of choosing key pa-
pers from the study area and analyzing the papers that
have cited them (Wohlin, 2014).
For this research, we analyzed the citations refer-
ring to the 10 papers selected during the first phase.
We performed these search through Google Scholar
3
, by searching for each key papers, and then for all
the papers that cited them.
Thus, 80 papers were identified and submitted to
the same steps and study selection criteria used in the
first stage. At the end, we excluded 78 papers that
did not answer our research questions and selected 2
papers for data extraction.
4 RESULTS
This section describes the third step of our SML
(i.e., Analysis and Reporting), where we analyzed
3
http://scholar.google.com
and summarized the research results to answer the re-
search question: RQ1 - How are the non-functional
requirements evaluations conducted in IoT applica-
tions?
Since we highlight the singularities of IoT appli-
cations, it is necessary to understand how these solu-
tions have been evaluated.
Finally, we selected 12 documents for data extrac-
tion, following the data extraction form. After that,
we analyzed the results and answered our research
questions. Table 3 shows the final set of studies and
their respective ID.
Table 3: List of final set of studies.
ID Reference
S1 (Ruiz-L
´
opez et al., 2012)
S2 (Ruiz-L
´
opez et al., 2013)
S3 (Kim et al., 2017)
S4 (Rocha et al., 2017)
S5 (Munnelly and Clarke, 2008)
S6 (Jazdi et al., 2016)
S7 (Ruiz-L
´
opez et al., 2013)
S8 (Badii et al., 2018)
S9 (Ghasemi et al., 2019)
S10 (Chauhan and Babar, 2017)
S11 (Filho et al., 2020)
S12 (Alegre-Ibarra et al., 2018)
Figure 1 represents the PRISMA Flow Diagram
(Moher et al., 2009) that describes the process
adopted to conduct this research and its results.
Figure 2 and Figure 3 show, respectively, the dis-
tributions of studies by the type of systems that they
were focused on and the type of NFR evaluation, if
they were focused on systems in development or sys-
tems already implemented.
SQ1 - Which Are the Approaches, Tools, Methods,
and Processes Used to Evaluate Non-Functional
Requirements in IoT Applications?
We identified 2 tools, 6 approaches, 1 method, and 1
process that can be used for the evaluation of NFRs.
Table 4 presents the strategies identified for evaluat-
ing NFRs in IoT systems.
SQ2 - Which Are the Non-Functional Require-
ments Observed When Evaluating IoT Applica-
tions?
Once we understood what had been used to conduct
the evaluations, it is important to identify if the eval-
Evaluation of Non-Functional Requirements for IoT Applications
115
Figure 1: PRISMA flow diagram.
uators had considered specific NFRs to evaluate IoT
applications.
We found 7 studies that described one or more
NFRs as suitable for evaluating ubiquitous systems or
IoT applications.
A total of 42 NFRs were identified
4
. Table 5
shows the list of NFRs considered during quality eval-
uations in IoT applications.
SQ3- Which Are the Challenges Faced When Eval-
uating Non-Functional Requirements?
By analyzing the final set of studies, we also identified
some challenges concerning the evaluation of NFRs
in IoT applications.
The most recurrent challenge is the necessity to
deal with context-awareness. Other problems are the
4
The list of 42 NFR with its descriptions is available on
https://github.com/great-ufc/NFRs4IoT
Figure 2: Type of system prioritized by the studies.
Figure 3: Focus of NFR Evaluation prioritized by the stud-
ies.
lack of a systematic approach and appropriate meth-
ods for evaluating non-functional requirements in IoT
applications.
5 DISCUSSION
We performed a systematic mapping to answer the
following primary research question: How are the
non-functional requirements evaluations conducted in
IoT applications?
Moreover, to better address the aspects involved in
this question, we decided to divide this macro ques-
tion into three secondary questions (SQs). The an-
swers came from a detailed analysis of 12 out of 174
papers returned from the search in databases and the
snowballing forward application.
Regarding SQ1, we noticed that most of the ap-
proaches, methods, and tools are focused on the re-
quirements elicitation stage and not on evaluating if
the IoT systems meet the NFRs previously estab-
lished.
ICEIS 2021 - 23rd International Conference on Enterprise Information Systems
116
Table 4: Strategies identified by each study.
Study Proposal Type
S1 MD-UBI Process
S2 MD-UBI Process
S3
Architecture-based
energy evaluation
Approach
S4 HUbis Tool
S5 GQM Method
S6
Determination of the
software reliability
approach
Approach
S7 REUBI Method
S8 Performance evaluation Approach
S9
ATAM scenario-based
approach
Approach
S10
Reference Architectures
for Design and Evaluation
of Web of Things Systems
Approach
S11 MALTU Approach
S12 RC-ASEF Tool
Also, we only identified one process by this re-
search, MD-UBI (Ruiz-L
´
opez et al., 2013). This pro-
cess was defined to support developers and evalua-
tors in elicitation, representation, and evaluation of re-
quirements, focusing on NFRs. This process is model
oriented and aims to develop high-quality Ubiquitous
Systems, covering the entire software development
lifecycle.
Regarding to SQ2, we identified 42 non-
functional requirements.
Among the 42 NFRs identified, 14 of these
are present as characteristics and sub-characteristics
in the quality models proposed by ISO/IEC 25000
(2011).
System/software product quality: adaptation,
availability, ease of use, interoperability, main-
tenability, modifiability, performance, reliability,
reusability, security, testability and usability.
Quality in use: efficiency and effectiveness.
Two of the selected papers have also cited 27 qual-
ity characteristics for ubiquitous systems. This set
is proposed by Carvalho et al. (2017) and has some
characteristics in common with the two quality mod-
els proposed by ISO/IEC 25000 (2011), but it con-
tains specific characteristics for ubiquitous systems:
Table 5: NFRs distribution per study.
NFRs Cited by
Acceptability S4, S11
Adaptation S4, S11
Attention S4, S11
Availability S4,S9, S10, S11
Calmness S4, S11
Context-awareness S4,S11
Comprehensibility S5
Data Input S4,S11
Device Capability S4, S11
Ease of use S4,S11
Effectiveness S4, S11
Efficiency S4,S11
Elasticity S10
Energy efficiency S3
Familiarity S4,S11
Flexibility S4,S11
Information display S4, S11
Interconnectivity S4, S11
Interoperability S9, S10
Maintenability S5
Maneageability S5
Mobility S4, S11
Modifiability S9
Multi-tenancy S10
Network Capability S4,S11
Performance S8, S9
Positioning of
components S4,S11
Predictability S4, S11
Privacy S4,S11
Reliability S4,S6, S10, S11
Reusability S5
Robustness S4,S11
Safety S4,S11
Scalability S4,S5, S8, S10, S11
Security S4,S9, S10, S11
Simplicity S4,S11
Testability S5
Transparency S4,S11
Trust S4, S11
Usability S4,S5, S9, S11
User Satisfaction S4,S11
Utility S4,S11
Acceptability, attention, calmness, context-
awareness, device capability, familiarity,
interconnectivity, mobility, network capabil-
ity, predictability, privacy, robustness, safety,
scalability, simplicity, transparency, trust, user
satisfaction, and utility.
Evaluation of Non-Functional Requirements for IoT Applications
117
Given the similarities between IoT applications
and ubiquitous systems, the use of this set of quality
characteristics is also applicable for conducting qual-
ity evaluations on IoT systems.
The most evaluated NFR was Scalability, which is
the ability to provide services to a few or a large num-
ber of users (Carvalho et al., 2017). Scalability was
considered an important NFR to evaluate by 41,6%
of the studies (5 papers). Another NFR identified is
related with scalability:
Elasticity: The ability of the system to provide
particular service on demand during a time inter-
val (Chauhan and Babar, 2017).
Availability, reliability, security and usability were
also highlighted. They were both cited by 33% of the
studies (4 papers).
For SQ3, we identified that the most significant
challenge when evaluating IoT systems is the context-
awareness, which is the system’s ability to discover
and take advantage of contextual information such as
user location, time of day, and user activity (Musumba
and Nyongesa, 2013). Thus, it is necessary to ensure
that the system correctly identifies the contextual in-
formation, so the users will be benefited instead of
affected by this non-functional requirement.
Another challenge is the lack of specific ap-
proaches for the evaluation of NFRs in IoT applica-
tions. This result is in line with the discussion held at
SQ1, where we identified only 3 studies that focused
on the evaluation of NFRs satisfaction and only one
process that has a step to evaluate NFRs in the final
product.
6 CONCLUSION
IoT applications are gaining more and more space in
our daily lives. Therefore, it is necessary to look at
how these solutions have been evaluated, mainly con-
cerning non-functional requirements, because they
are directly related to the software quality evaluation
and the user expectations through the system.
In this work, we perform a systematic mapping on
the evaluation of non-functional requirements in IoT
systems to provide a comprehensive view about this
field.
As the main contributions, we identify artifacts
(tools, methods, approaches, and processes) to eval-
uate NFRs and we perceived the lack of artifacts
and systematic approaches to evaluate whether IoT
applications satisfy the previously established non-
functional requirements or not.
We also identified 42 NFRs that can be considered
to design and evaluate of IoT applications. Most ex-
isting artifacts are focused on the usability and relia-
bility of the solutions, which may represent a concern
about how the users interact and how they relate to
IoT applications.
As future work, there is a need to develop and
validate a process to systematize the NFR evaluation
steps for IoT applications with a particular eye to the
human-thing and thing-thing interactions.
ACKNOWLEDGEMENTS
We would like to thank CNPq for the Productiv-
ity Scholarship of Rossana M. C. Andrade DT-2
(N
315543 / 2018-3) and UFC, FASTEF, and Dell
cooperation using the Brazilian Informatics Law (No
10.176of 1/11/2001) incentives that help us to support
our research laboratory.
REFERENCES
Alegre-Ibarra, U., Augusto, J. C., and Evans, C. (2018).
Rc-asef: An open-source tool-supported requirements
elicitation framework for context-aware systems de-
velopment. In 2018 Federated Conference on Com-
puter Science and Information Systems (FedCSIS),
pages 829–838.
Andrade, R. M., Carvalho, R. M., de Ara
´
ujo, I. L., Oliveira,
K. M., and Maia, M. E. (2017). What changes from
ubiquitous computing to internet of things in inter-
action evaluation? In Lecture Notes in Computer
Science (including subseries Lecture Notes in Artifi-
cial Intelligence and Lecture Notes in Bioinformatics),
volume 10291 LNCS, pages 3–21. Springer Verlag.
Ashton, K. J. (2009). That ‘internet of things’ thing.
Atzori, L., Iera, A., and Morabito, G. (2010). The In-
ternet of Things: A survey. Computer Networks,
54(15):2787–2805.
Badii, C., Belay, E. G., Bellini, P., d. Cenni, Marazzini, M.,
Mesiti, M., Nesi, P., Pantaleo, G., Paolucci, M., Val-
tolina, S., Soderi, M., and Zaza, I. (2018). Snap4city:
A scalable iot/ioe platform for developing smart city
applications. In 2018 IEEE SmartWorld, Ubiquitous
Intelligence Computing, Advanced Trusted Comput-
ing, Scalable Computing Communications, Cloud Big
Data Computing, Internet of People and Smart City
Innovation (SmartWorld/ SCALCOM/ UIC/ ATC/ CB-
DCom/ IOP/ SCI), pages 2109–2116.
Bures, M., Bellekens, X., Frajtak, K., and Ahmed, B. S.
(2020). A comprehensive view on quality charac-
teristics of the iot solutions. In Jos
´
e, R., Van Laer-
hoven, K., and Rodrigues, H., editors, 3rd EAI In-
ternational Conference on IoT in Urban Space, pages
59–69, Cham. Springer International Publishing.
Carvalho, R. M., Andrade, R. M., and Oliveira, K. M.
(2020). How developers believe invisibility impacts
ICEIS 2021 - 23rd International Conference on Enterprise Information Systems
118
nfrs related to user interaction. In 2020 IEEE 28th
International Requirements Engineering Conference
(RE), pages 102–112. IEEE.
Carvalho, R. M., Castro Andrade, R. M., Oliveira, K. M.,
Sousa Santos, I., and Bezerra, C. I. (2017). Quality
characteristics and measures for human—computer
interaction evaluation in ubiquitous systems. Software
Quality Journal, 25(3):743–795.
Chauhan, M. A. and Babar, M. A. (2017). Chapter 7
- using reference architectures for design and eval-
uation of web of things systems**this chapter de-
scribes a method for software architecture design,
analysis, evaluation and evolution of individual inter-
net of things (iot) subsystems and how individual iot
subsystems can be used for design of web of things
(wot) systems.: A case of smart homes domain. In
Sheng, Q. Z., Qin, Y., Yao, L., and Benatallah, B., ed-
itors, Managing the Web of Things, pages 205 228.
Morgan Kaufmann, Boston.
Chung, L. and Prado Leite, J. C. (2009). On Non-
Functional Requirements in Software Engineering,
page 363–379. Springer-Verlag, Berlin, Heidelberg.
Filho, J. C. S., Brito, M. R. F., and Sampaio, A. L.
(2020). Comparing heuristic evaluation and maltu
model in interaction evaluation of ubiquitous systems.
In Anais Estendidos do XIX Simp
´
osio Brasileiro so-
bre Fatores Humanos em Sistemas Computacionais
(IHC’20), Porto Alegre, RS, Brasil. SBC.
Ghasemi, F., Rezaee, A., and Rahmani, A. M. (2019).
Structural and behavioral reference model for IoT-
based elderly health-care systems in smart home.
International Journal of Communication Systems,
32(12):e4002.
ISO/IEC 25000 (2011). Iso/iec 25000 - systems and soft-
ware engineering - systems and software quality re-
quirements and evaluation (square). Technical report.
Jazdi, N., Oppenlaender, N., and Weyrich, M. (2016).
Quantification of the quality characteristics for
the calculation of software reliability. IFAC-
PapersOnLine, 49(30):1 5. 4th IFAC Symposium
on Telematics Applications TA 2016.
Kim, D., Choi, J.-Y., and Hong, J.-E. (2017). Evaluating
energy efficiency of internet of things software archi-
tecture based on reusable software components. In-
ternational Journal of Distributed Sensor Networks,
13(1):1550147716682738.
Kim, M. (2016). A quality model for evaluating IoT appli-
cations. International Journal of Computer and Elec-
trical Engineering, 8(1):66–76.
Kitchenham, B. and Charters, S. (2007). Guidelines for per-
forming systematic literature reviews in software en-
gineering. Technical Report EBSE 2007-001, Keele
University and Durham University Joint Report.
Moher, D., Liberati, A., Tetzlaff, J., and Altman, D. G.
(2009). Preferred reporting items for systematic re-
views and meta-analyses: the prisma statement. An-
nals of Internal Medicine, 151(4):264–269.
Munnelly, J. and Clarke, S. (2008). Infrastructure for ubiq-
uitous computing: Improving quality with modulari-
sation. In Proceedings of the 2008 AOSD Workshop
on Aspects, Components, and Patterns for Infrastruc-
ture Software, ACP4IS ’08, New York, NY, USA. As-
sociation for Computing Machinery.
Musumba, G. and Nyongesa, H. (2013). Context awareness
in mobile computing: A review. International Journal
of Machine Learning and Applications, 2(1):5.
Patel, K. K., Patel, S. M., and Scholar, P. G. (2016). Inter-
net of Things-IOT: Definition, Characteristics, Archi-
tecture, Enabling Technologies, Application & Future
Challenges. International Journal of Engineering Sci-
ence and Computing.
Petersen, K., Vakkalanka, S., and Kuzniarz, L. (2015).
Guidelines for conducting systematic mapping stud-
ies in software engineering: An update. Information
and Software Technology, 64:1 – 18.
Rocha, L. C., Andrade, R. M. C., Sampaio, A. L., and Lelli,
V. (2017). Heuristics to evaluate the usability of ubiq-
uitous systems. In Streitz, N. and Markopoulos, P.,
editors, Distributed, Ambient and Pervasive Interac-
tions, pages 120–141, Cham. Springer International
Publishing.
Rowland, C., Goodman, E., Charlier, M., Light, A., and Lui,
A. (2015). Designing connected products: UX for the
consumer Internet of Things. ” O’Reilly Media, Inc.”.
Ruiz-L
´
opez, T., Noguera, M., Rodr
´
ıguez F
´
ortiz, M. J., and
Garrido, J. L. (2013). Requirements systematization
through pattern application in ubiquitous systems. In
van Berlo, A., Hallenborg, K., Rodr
´
ıguez, J. M. C.,
Tapia, D. I., and Novais, P., editors, Ambient Intelli-
gence - Software and Applications, pages 17–24, Hei-
delberg. Springer International Publishing.
Ruiz-L
´
opez, T., Rodr
´
ıguez-Dom
´
ınguez, C., Noguera, M.,
and Rodr
´
ıguez, M. J. (2012). A model-driven ap-
proach to requirements engineering in ubiquitous sys-
tems. In Novais, P., Hallenborg, K., Tapia, D. I., and
Rodr
´
ıguez, J. M. C., editors, Ambient Intelligence -
Software and Applications, pages 85–92, Berlin, Hei-
delberg. Springer Berlin Heidelberg.
Ruiz-L
´
opez, T., Rodr
´
ıguez-Dom
´
ınguez, C., Noguera, M.,
Rodr
´
ıguez, M. J., Benghazi, K., and Garrido, J. L.
(2013). Applying model-driven engineering to a
method for systematic treatment of nfrs in ami sys-
tems. J. Ambient Intell. Smart Environ., 5(3):287–310.
Sommerville, I. (2011). Software engineering 9th edition.
ISBN-10, 137035152:18.
Uckelmann, D., Harrison, M., and Michahelles, F. (2011).
An Architectural Approach Towards the Future Inter-
net of Things, pages 1–24. Springer Berlin Heidel-
berg, Berlin, Heidelberg.
Weiser, M. (1991). The computer for the 21 st century. Sci-
entific american, 265(3):94–105.
Wiegers, K. and Beatty, J. (2013). Software Requirements,
Third Edition. Microsoft Press.
Wohlin, C. (2014). Guidelines for snowballing in system-
atic literature studies and a replication in software en-
gineering. In Proceedings of the 18th International
Conference on Evaluation and Assessment in Software
Engineering, EASE ’14, New York, NY, USA. Asso-
ciation for Computing Machinery.
Evaluation of Non-Functional Requirements for IoT Applications
119