Expert Review of Taxonomy based Testing: A Testing Framework for
Medical Device Software
Hamsini Ketheswarasarma Rajaram
1a
, John Loane
1b
, Silvana Togneri MacMahon
2c
and Fergal Mc Caffery
1d
1
Regulated Software Research Centre, Dundalk Institute of Technology, Dundalk,
Ireland
2
School of Computing, Dublin City University, Dublin, Ireland
Keywords: The US FDA, Taxonomy, SW91, ISO/IEC/IEEE 29119, IEC 62304, Framework, Expert Review,
Software Testing, Recalls.
Abstract: This paper details the expert review of a framework developed to implement a novel testing approach called
taxonomy-based testing (TBT) for the medical device software domain. This framework proposes three
approaches to implement TBT and has been validated by experts from the software testing industry and the
medical device software domain. This paper details the results from the expert review. The expert review
focused on validating the three approaches to TBT, the benefits of TBT to medical device software
development, the accuracy of mappings of testing techniques from ISTQB and ISO/IEC/IEEE 29119-4:2015
to defects from a defect taxonomy, the integration of TBT into the standard test processes, ISTQB and
ISO/IEC/IEEE 29119-2:2013 and the structure of the framework. The contribution of this paper is to reveal
that (i) the framework is implementable in medical device software organisations that follow the IEC
62304:2006+A1:2015 software development process or that use standard test processes, (ii) using a defect
taxonomy could standardise the application of experience-based approaches to software testing and (iii)
considering potential defects before writing test cases could identify additional defects for test cases.
1 INTRODUCTION
This research proposed a testing approach called
taxonomy-based testing (TBT) to improve medical
device software (MDS) quality and to reduce adverse
events caused by MDS defects (Alemzadeh et al.
2013; Rajaram et al. 2020; Felderer and Beer 2013).
This research uses a defect taxonomy called SW91,
Classification of Defects in Health Software. SW91
is a standard for health software which has been
developed by the Association for the Advancement of
Medical Instrumentation (AAMI) in collaboration
with the US FDA (AAMI 2018). SW91 includes a
total of 186 defects from the planning phase to the
maintenance phase of a system. In TBT, the
requirements will be mapped into potential SW91
defects, and test cases will be written based on the
requirements and the mapped defects. Test cases will
a
https://orcid.org/0000-0002-3294-3906
b
https://orcid.org/0000-0002-9285-5019
c
https://orcid.org/0000-0003-0179-2436
d
https://orcid.org/0000-0002-0839-8362
be executed to verify whether the software complies
with the relevant requirements and does not contain
the mapped defects. By applying TBT, an
organisation can achieve benefits such as root cause
analysis, risk minimisation and early detection of
defects (Rajaram et al. 2020).
This research identified the need for a framework
detailing the TBT implementation process and the
alpha TBT TBT) framework was developed to
meet this need (Rajaram(&) et al. 2019). The main
purpose of this framework was to implement TBT in
any MDS organisation with limited resources and
without the researcher's involvement with the
organisation's requirement and defect data.
The literature revealed that expert review is
important to identify the usefulness and applicability
of research. (Offenberger et al. 2019; Sjøberg et al.
2007; Dumas and Sorce 1995; MacMahon et al.
Rajaram, H., Loane, J., MacMahon, S. and Caffery, F.
Expert Review of Taxonomy based Testing: A Testing Framework for Medical Device Software.
DOI: 10.5220/0010458503310339
In Proceedings of the 16th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE 2021), pages 331-339
ISBN: 978-989-758-508-1
Copyright
c
2021 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
331
2014). This research used expert review to validate
whether the α TBT framework was implementable
in MDS organisations. This will be followed by
implementing the framework in MDS organisations
to provide additional validation.
This paper details the expert review of the α
TBT framework. Section 2 explains the α –TBT
framework and the validation points considered in the
expert review. Section 3 details the process followed
during this expert review. Section 4 details reviews of
the α – TBT framework. Section 5 details subsequent
changes considered in the next version of the
framework. Section 6 is future work. It details how this
research plans to implement TBT in MDS organisa-
tions to complete the validation process. Section 7
details the summary and conclusion of this paper.
2 ALPHA TBT FRAMEWORK
The α–TBT framework details what TBT is and the
following three approaches to TBT:
Approach A: TBT using all SW91 defects.
Approach B: TBT using testing technique mappings.
Approach C: TBT at different phases of software
development.
The purpose of providing three different
approaches is that the organisation can select an
appropriate approach in light of their resources and
testing practices. Also, the three approaches help to
narrow down the selection of SW91 defects from 186
and allow the implementation of TBT at any phase of
MDS development. The α TBT framework details
the TBT implementation steps - review SW91
defects, plan-TBT, map requirements to SW91
defects, write test cases, execute tests and analyse
results. These steps have been integrated into the two
standard test processes: the ISO/IEC/IEEE 29119-
2:2013(ISO/IEC/IEEE 2013); and the test process of
the International Software Testing Qualifications
Board (ISTQB) (ISTQB 2010). These integrations
were published in (Rajaram(&) et al. 2019). The rest
of this section details the three approaches to TBT and
the points considered in the expert review of the α
TBT framework.
Approach A - TBT using all SW91 Defects: In this
approach, the organisation has to search and map the
potential defects for each requirement to all SW91
defects. In this approach, the participants of TBT
must have a good understanding of all SW91 defects
and their classification.
Approach B - TBT using Testing Technique
Mappings: This approach uses testing techniques
from the ISO/IEC/IEEE 29119-4: 2015 and ISTQB
(Graham et al. 2006; ISO/IEC/IEEE 2015). These
testing techniques have been mapped into their
potential SW91 defects, which can be identified when
applying these techniques. The purpose of this
mapping is to reduce the number of SW91 defects that
need to be considered for each requirement.
Approach C - TBT at Different Phases of Software
Development: This approach allows an organisation
to narrow down the number of SW91 defects to those
specific to a particular software development phase.
Table 1 shows the number of SW91 defects from each
phase of software development that needs to be
considered when using TBT approach C.
Table 1: SW91 defects.
Parent level defects Number of defects
Planning (1) 2
Requirements (2) 15
Architecture and
Design (3)
28
Implementation (4) 106
Test (5) 18
Release Defects (6) 4
Maintenance (7) 13
2.1 Validation of the α – TBT
Framework
The α TBT framework involves software testing,
MDS development, application of defect taxonomies,
and use of the ISO/IEC/IEEE 29119-2:2013 and
ISTQB test processes. The goal of this expert review
was to validate the following five points of the α
TBT framework:
1. The three approaches to TBT.
2. The three approaches to TBT for MDS
development.
3. Testing technique mappings.
4. Standard test processes and TBT.
5. The structure of the α – TBT framework.
After the expert review, the TBT framework will
be implemented in a number of MDS organisations to
provide further validation. This validation plan is
presented in Section 6. The next section details the
process followed in this expert review and Section 3.3
details the reviews.
ENASE 2021 - 16th International Conference on Evaluation of Novel Approaches to Software Engineering
332
3 EXPERT REVIEW PROCESS
AND EXPERT PROFILES
The following steps have been followed in this expert
review: (1) Find relevant experts. (2) Develop
questionnaires and request permission. (3) Receive
and analyse reviews.
3.1 Find Relevant Experts
As detailed in Section 2.1 the α TBT framework has
different parts to validate. It was not possible to find
individuals with expertise in all areas of the TBT
framework. Therefore, we looked to find one expert
for each point that we were looking to validate. There
was some overlap in expertise between experts. This
accounts for the difference in numbers of experts
validating each point. In order to validate the different
parts of the α – TBT framework, the following search
criteria have been used in expert selection:
Expertise in software testing
Expertise in defect taxonomy application
Expertise in testing techniques application
Expertise in MDS development
Expertise in SW91 defects
Expertise in the standard test processes such
as ISO/IEC/IEEE 29119-2:2013 and ISTQB.
Six experts were selected from authors who have
conducted taxonomy-supported testing and published
several papers, from the software testing industry and
from the MDS industry. Table 2 details the experts'
expertise considered for the framework validation.
Table 2: Experts.
Experts Expertise considered in this research
Expert 1 More than twenty years of experience in
software testing. Trainer for different levels of
ISTQB certification for more than fifteen
years. Leading active member of testing
standard family ISO/IEC/IEEE 29119.
Expert 2 More than fifteen years of experience in the
software development industry.
Internationally recognised researcher who has
conducted several research projects in
software industry including defect taxonomy-
supported testing by applying the ISTQB test
process.
Expert 3 Industrial testing consultant with knowledge
in the application of defect taxonomy-
supported testing in industrial projects along
with testing techniques and the ISTQB test
process. Thirty years' experience in test
management, test automation and quality
assurance.
Expert 4 Testing expertise with agile and the ISTQB
test process.
Over thirty years' experience in
the software development industry.
Expert 5 Medical device regulatory expert who works
at U.S. FDA for over ten years. Active and
leading member of the standard development,
AAMI SW91 "Classification of Defects in
Health Software".
Expert 6 Quality assurance manager for over ten years
at a leading multi-national software testing
organisation.
3.2 Develop Questionnaires and
Request Permission
As the framework is complex and had multiple
distinct parts to validate, three sets of open-ended
questionnaires (questionnaires A, B and C) were
developed. All three questionnaires had 80% overlap.
Table 3 details the questionnaires and experts who
received those questionnaires. Questionnaire B
included the same questions as Questionnaire A and
it added a set of questions focused on the testing
technique mappings. Questionnaire C included
questions from Questionnaire A, but it added
questions focused on finding the applicability of the
framework to the MDS domain and did not focus on
validating the point about standard test processes and
TBT.
In order to develop questionnaires, the question
design process detailed by Dawson was followed in
this study (Dawson 2009). Questions were developed
to validate different points of the α TBT framework
detailed in Section 2.1. After developing questions, a
draft of the questions was reviewed by another
researcher to determine how well the questions
Table 3: Questionnaires, validation points and experts.
Questionnaires Validation points Experts
Questionnaire A Three approaches to TBT. Expert 1
Expert 2
Expert 4
Standard test processes and
TBT.
The structure of the α –
TBT framework.
Questionnaire B Three approaches to TBT. Expert 3
Expert 6
Testing technique
mappings.
Standard test processes and
TBT.
The structure of the α –
TBT framework.
Questionnaire C Three approaches to TBT
for MDS development.
Expert 5
The structure of the α –
TBT framework.
Expert Review of Taxonomy based Testing: A Testing Framework for Medical Device Software
333
fulfilled their purpose and to ensure that they did not
contain any ambiguity.
Since expertise between experts was overlapped,
different numbers of experts were considered for
validating each point of α TBT framework. Due to
this, an imbalance of experts to validate each point
was observed. As this research will further validate
the framework with MDS organisations, the
imbalance of experts for each point was not
considered as a major validity issue to this research.
Expert 5 was involved in the development of
SW91. This is considered a strength rather than a
limitation of this expert review. Their in-depth
knowledge of SW91 along with their medical device
software regulatory expertise means that they were an
ideal candidate to validate the framework.
After identifying the experts and developing the
questionnaires, emails were sent to the experts
requesting that they review the α TBT framework.
The experts were also asked to participate in a focus
group interview to discuss the findings from their
reviews.
3.3 Receive and Analysis of Reviews
All experts agreed to provide written feedback and to
join a focus group interview or individual interview
after their initial reviews were analysed. Experts 1
and 3 participated in the focus group interview. The
other experts were unable to join at the requested
time. Additional interviews were held with Experts 5
and 6. Expert 4 was also contacted for further
clarification and he has provided this clarification via
email. The researcher analysed the reviews collected
from questionnaires, focus group interviews and
individual interviews. The next section presents the
analysis of the reviews.
4 EXPERT REVIEW OF THE
ALPHA TBT FRAMEWORK
The remainder of this section details the reviews on
the five points of α TBT framework listed in Section
2.1.
4.1 Three Approaches to TBT
This part was validated by experts 2, 3, 4 and 6. This
part of the expert review focused on validating the
benefits of the three approaches to TBT in practice
and implementation of the three TBT approaches.
Expert 2 and 3 did not point out any difficulties
with the three approaches to TBT. This is significant,
as these two experts have experience in the
application of defect taxonomy-supported testing.
They prefer approach B, which uses testing
techniques. They said that this approach facilitates an
efficient design of test cases and it will be a good
testing strategy to consider when planning the testing
for projects which use testing techniques.
Expert 2 said that by adopting any of the
approaches, the implementation phase of a project
would have enhanced traceability by checking code
against requirements and mapped defects.
Expert 3 said that since TBT uses a defect
taxonomy from the MDS industry, it should include
defects which are familiar and that TBT could be
easily implemented into a MDS organisation.
Expert 4 said that the three TBT approaches could
be beneficial. He suspects only a minority of MDS
organisations will use it. Unless the adherence to
standards is made mandatory by regulatory bodies,
the overhead of application is too great for most
companies. He also said that TBT could be hard to
sell to an organisation due to the cost of the third step
of TBT, mapping requirements to SW91 defects.
Expert 6 said that defect taxonomies are often
used informally as an experienced-based technique
during test case generation. The TBT framework
could be seen as standardising, something that was
previously based primarily on experience. TBT can
be used as an additional review before or after the
creation of the initial test specification. During this
review using the lens of the mapped defect taxonomy,
the test cases could be reviewed for coverage.
However, he also suspected that TBT would be hard
to sell unless defect taxonomy analysis was part of the
existing test creation phase.
Experts 2 and 3 have provided positive feedback.
Experts 4 and 6 suspect that TBT is hard to sell to
organisations. Since both experts 4 and 6 have
provided feedback based on their expertise in non -
MDS, their feedback was discussed with a MDS
expert, expert 5. Expert 5 strongly disagreed with
their comments and said that while the "the overhead
of application is too great for most companies might
be legitimate", there is still no proper evidence to
show the cost or time of the mappings from TBT
implementation. Expert 5 said the benefits of TBT
such as early detection of defects, root cause analysis
have to be prioritised first and that the time needed to
conduct the mappings has to be identified through
industrial validation of this framework. The time
taken to conduct the mapping between requirements
and SW91 defects will be identified through
ENASE 2021 - 16th International Conference on Evaluation of Novel Approaches to Software Engineering
334
validation with MDS organisations. This is detailed
in Section 6.
MDS expert 5 did not agree with expert 4's
comment that the regulatory bodies should make
adherence to standards mandatory. Expert 5 said that
regulations aim to ensure safe and reliable software
development and to assist the MDS organisations in
developing software in the right way. MDS
organisations should not be governed by regulatory
bodies. They should understand the purpose and
benefits of regulations.
4.2 Review of the Three Approaches to
TBT for MDS Development
This part of the expert review has focused on finding
out the benefits of the three approaches to TBT for
MDS development. This part was validated by expert
5.
Expert 5 provided a review of the three
approaches to TBT and their implementation to
different situations in a MDS organisation. Expert 5
said that approach C seems a good starting point that
will bring benefits to MDS development. The
selection of an approach has to be based on the
software development methodology. Existing
development methodologies at a MDS organisation
could determine which approach can be selected. For
example, test-driven development can use approach
B. Approach C is implementable in agile
development. However, approach A may also work
with an agile methodology where the focus is on all
the defects that match the current sprint. Expert 5
suggested detailing how TBT can be implemented at
a company that uses agile.
Expert 5 said that since IEC 62304:2006+A1:2015
is a process standard that does not include test criteria,
it is hard to tell whether or not the three approaches
comply with the processes of IEC
62304:2006+A1:2015. As there were no test criteria in
IEC 62304:2006+A1:2015 (Bujok et al. 2017), expert
5 suggested that the TBT framework should include
the following three direct links between IEC
62304:2006+A1:2015 and SW91 defects:
Link 1: The defect category Security (3.8) from
SW91 was developed with IEC
62304:2006+A1:2015 in mind. MDS organisations
can use this category and its definition to refer to
defects related to security.
Link 2: Segregation necessary for risk control (3.2)
is one defect category from SW91. This category
aligns with Section 5.3.5 of IEC
62304:2006+A1:2015 (IEC 2015).
Link 3: The framework suggests considering potential
defects of SW91 in each phase before that phase. This
can be linked with Section 5.1.12 from IEC
62304:2006+A1:2015. Section 5.1.12 details the
identification and avoidance of common software
defects.
The researcher extended the study on the link
between IEC 62304:2006+A1:2015 and SW91
defects and found the following fourth link:
Link 4: Section 5.1.6 of IEC 62304:2006+A1:2015
details that each life cycle activity has to include
verification. This can be linked with the TBT
framework, approach C which details how SW91
defects can be used in the verification at different
phases of MDS development.
The links above between SW91 and IEC
62304:2006+A1:2015 show that the application of
the α TBT framework could help MDS
organisations to achieve regulatory compliance.
4.3 Review of Testing Technique
Mappings
Testing techniques from the ISTQB and the
ISO/IEC/IEEE 29119-4:2015 were mapped into
potential SW91 defects. These mappings are used in
TBT approach B. These mappings were validated by
expert 3 and expert 6.
This part of the validation focused on establishing
the accuracy of the mappings. Expert 3 said that the
"mappings in the excel sheets are understandable and
plausible". Expert 3 said that this mapping could help
testers to design test cases effectively and that it would
also significantly improve the design of test cases.
Expert 6 said that these mappings are good and
comprehensively cover the testing techniques in the
software testing industry. Expert 6 said that the
granularity of defects in the mappings is much finer
than what his organisation currently works with. For
example, the mapping of equivalence partitioning
testing technique includes defects such as Data
definition (4.1), Scalar data type (4.1.1) and Scalar
Data Operations (4.1.1.1). Expert 6 said that testers
would not log this granularity of defects to a root
cause in their defect management/software life cycle
application.
Since expert 6 is from non-MDS testing, this
comment was discussed with expert 5 to know about
the granularity of defects in MDS development.
Expert 5 said MDS organisations should consider root
causes and contributing factors to ensure quality via
mitigation of root causes.
Expert Review of Taxonomy based Testing: A Testing Framework for Medical Device Software
335
4.4 Standard Test Processes and TBT
The six steps of the TBT implementation process are
integrated into the two standard test processes,
ISO/IEC/IEEE 29119-2:2013 and ISTQB. These
integrations were detailed and published in
(Rajaram(&) et al. 2019).
This part of the validation
focused on finding out:
Whether or not the two standard test processes are
aligned correctly.
Whether the six steps of TBT are correctly
integrated into the standard test processes.
Whether TBT is implementable in an
organisation which follows a standard test
process.
This part was validated by experts 1, 2, 3 and 4.
Experts 1 and 3 said that both standard test processes
have different scopes and that the researcher had not
explained this during the alignment of both standard
test processes. The ISO/IEC/IEEE 29119-2:2013 test
process has three sub processes, from the organisation
level to the project level. The scope of the ISTQB test
process is project-level testing only. It cannot be
aligned with the organisational test process from
ISO/IEC/IEEE 29119-2:2013. The experts suggested
detailing the difference in scope in the framework.
Regarding the integration of TBT into the test
processes ISO/IEC/IEEE 29119-2:2013 and ISTQB,
all the experts provided positive comments on the
integration. They suggested that TBT is currently
integrated into the older version of the ISTQB test
process and should be integrated into the newer
version of the ISTQB test process.
Regarding the six steps of TBT, the experts
suggested that Step 1: Review SW91 defects, should
not be considered in the test processes. This step
should be considered as knowledge sharing and
conducted at the requirement engineering phase,
which is outside of the test process. If this step is
considered at the requirement engineering phase, this
will provide an opportunity to consider all potential
defects from all phases of software development. The
experts also suggested minor changes related to
terminologies and their explanations in both test
processes. The experts said that both test processes,
the ISTQB and the ISO/IEC/IEEE 29119-2:2013
process allow the selection of appropriate testing
techniques for a project. TBT is a new testing
technique and could be considered when a testing
team selects testing techniques for a project in an
organisation. TBT is implementable in an
organisation using either standard test process, the
ISTQB or the ISO/IEC/IEEE 29119-2:2013.
4.5 Review of the Structure of the
Framework
This part was validated by experts 1, 2, 3, 4 and MDS
expert 5. The purpose of this part of the validation is
to find out whether the framework is coherent. It also
aimed to assess if the framework was implementable
in MDS organisations.
Expert 1 and expert 3 said that the framework is
readable, understandable and well structured. The six
steps are explained very well. If the researcher
includes examples throughout the framework, it will
be clearer to the reader. Expert 2 and expert 4 said that
the α TBT framework is clearly structured, although
they pointed out the challenges on the implementation
of TBT due to the time it will take to conduct the
mappings between requirements and SW91 defects.
Expert 5 said that by including identified pros and
cons for each approach, it would make the decision-
making easier for MDS organisations by showing that
possible failure points have been taken into
consideration.
In order to make the implementation of TBT
easier, they said that the researcher should perform
training on conducting mappings using sample
requirements and SW91 defects at the organisation
which agreed to implement it. The experts suggested
that by implementing TBT in an organisation,
efficiency can be measured and that this detail should
be included in the framework. They suggested giving
thought to automating this framework.
5 CHANGES RESULTING FROM
EXPERT REVIEW
The experts provided their reviews on the α–TBT
framework. This section summarises the changes
resulting from the expert review that will be built into
the next version of the framework, the β – TBT
framework.
The first part of the α–TBT framework validation
focused on the three approaches to TBT. No changes
to the α–TBT framework were suggested apart from
finding the time needed to conduct the mappings
between requirements and SW91 defects. This will be
identified through the validation of TBT approach B
in a MDS organisation and it will be included in the
β - TBT framework. This has been detailed in future
work, Section 6.
The second part of the α–TBT framework
validation focused on the three approaches to TBT for
MDS development. This was validated by expert 5,
ENASE 2021 - 16th International Conference on Evaluation of Novel Approaches to Software Engineering
336
who suggested detailing the direct links between IEC
62304:2006+A1:2015 and SW91. This suggestion
has been adopted and these links have been included
in the β - TBT framework. Expert 5 also suggested
including details on how TBT can be used with an
agile methodology. A document will be developed
detailing how to implement TBT when using an agile
methodology. This document will be added to the β -
TBT framework.
The third part of the α–TBT framework validation
focused on testing technique mappings. Testing
techniques from the ISTQB and the ISO/IEC/IEEE
29119-4:2015 were mapped into the potential SW91
defects and used in approach B of TBT. This part was
validated by experts 3 and 6, who provided positive
comments about the accuracy of the mappings.
Expert 6 said that the mappings contain much finer
defect details than they typically work with. Testers
would not log the finer detail of defects in their defect
management/software life cycle. This review was
discussed with expert 5, who said that MDS
organisations should consider finer detail of defects
to find root causes. Taking into account the statement
made by Expert 5 on finer defect details, the use of
finer defect details will be investigated through the
TBT approach B implementation with MDS
organisations. This has been detailed in future work,
Section 6.
The fourth part of the α–TBT framework
validation focused on standard test processes and
TBT. The experts said that TBT is implementable in
an organisation which follows the standard test
process, either ISO/IEC/IEEE 29119-2:2013 or
ISTQB. However, the experts suggested the
following changes, which have been adopted into the
β - TBT framework:
Do not integrate the first step of TBT into both
standard test processes, ISTQB and
ISO/IEC/IEEE 29119-2:2013. Consider it at the
requirement gathering phase.
Detail the scope of both test processes ISTQB,
ISO/IEC/IEEE 29119-2:2013 and where TBT can
fit.
Integrate TBT into the new version of the ISTQB
test process and provide additional detail to the
phases of both test processes ISTQB and
ISO/IEC/IEEE 29119-2:2013.
The last part of the α–TBT framework validation
focused on the structure of the framework. All the
experts provided positive comments on the structure
of the αTBT framework. However, the experts
suggested performing training on how to conduct
mappings between requirements and SW91 defects at
the organisation which agreed to implement TBT.
The experts suggested including the efficiency, pros
and cons of TBT approaches in the framework to
make the selection easier for the organisation.
This research will validate TBT approach B in an
MDS organisation. From this validation, the
efficiency, pros and cons of TBT approach B will be
investigated and includes in the β - TBT framework.
This has been detailed in future work, Section 6.
Table 4 summarises the changes suggested from the
expert review. Changes in bold have already been
adopted into the β TBT framework. The rest of the
changes will be addressed through future work.
Table 4: Changes for building β – TBT framework.
Validation points Suggested changes
Three approaches
to TBT.
Investigate the time taken to conduct
the mappings.
Three approaches
to TBT and MDS
development.
Develop an approach for
implementing TBT in Agile
development.
Examine the direct links between
SW91 and IEC
62304:2006+A1:2015.
Testing technique
mappings.
Include the benefits of finer defects to
MDS development.
TBT and standard
test process.
Consider the first step of TBT at the
requirement gathering phase.
Provide the scope of both test
processes ISTQB and
ISO/IEC/IEEE 29119-2:2013 and
TBT.
Integrate TBT into the newer
version of ISTQB.
The structure of
the α TBT
framework.
Detail the pros and cons of three
approaches to TBT.
Provide examples through the
framework.
6 FUTURE WORK
As detailed in Section 1, the expert review on the α –
TBT framework is one part of the validation. This
validation suggested including the following points in
the next version of the framework:
Pros and cons of the three approaches to TBT.
Time taken to conduct mappings between
requirements and SW91 defects.
The efficiency of three approaches to TBT.
Benefits of finer defects to MDS development.
In order to validate the points raised in the expert
reviews, a research collaboration was established
Expert Review of Taxonomy based Testing: A Testing Framework for Medical Device Software
337
with an MDS organisation, Company B in the U.K.
The researcher presented the three approaches to
TBT. By considering resources and project time at
Company B, they agreed to implement TBT approach
B. During this implementation, five requirements
from a past release at Company B will be used to map
into SW91 defects using the testing technique
mappings. Based on the mappings, the researcher will
interview the test engineer at Company B to
investigate the points raised in the expert review.
7 CONCLUSION
This paper has detailed a part of the validation, expert
review of the α–TBT framework. Six experts from the
software testing industry and the MDS industry have
reviewed the α–TBT framework. The expert review
of α–TBT framework focused on validating
approaches to TBT, the benefits of TBT to MDS
development, the accuracy of mappings of testing
techniques from ISTQB and ISO/IEC/IEEE 29119-
4:2015 to defects from SW91, the integration of TBT
into the standard test processes such as ISTQB ,
ISO/IEC/IEEE 29119-2:2013 and the structure of the
framework.
Experts provided positive feedback on the three
approaches to TBT. The review identified that the
three approaches to TBT could be implemented in
different situations at MDS organisations.
Approaches A and C can fit into agile development.
Approach B can fit into test-driven development and
it enables the standardisation of the experience-based
application of defect taxonomies in software testing.
Since the α–TBT framework includes mappings of
testing techniques to SW91 defects, it will be
beneficial to consider potential defects before writing
test cases.
The experts said that the framework is readable
and well structured. It is implementable in MDS
organisations which use the
IEC62304:2006+A1:2015 process. It enables the
implementation of TBT into existing standard test
processes such as ISTQB and ISO/IEC/IEEE 29119-
2:2013. Experts said that the testing technique
mappings comprehensively cover the testing
techniques in the software testing industry and the
mappings are correct. These mappings would
significantly improve the design of test cases.
Experts suggested including additional details in
the next version of the framework such as (i) pros,
cons and efficiency of the three approaches to TBT,
(ii) time taken to conduct mappings between
requirements and SW91 defects and (iii) benefits of
reporting fine grained defects to MDS development.
These points will be investigated through validation
with MDS organisations. This will be included in the
next version of the framework.
ACKNOWLEDGEMENT
This work was supported with the financial support
of the Science Foundation Ireland grant 13/R.C./2094
and co-funded under the European Regional
Development Fund through the Southern & Eastern
Regional Operational Programme to Lero - the Irish
Software Research Centre (www.lero.ie).
REFERENCES
AAMI. (2018). American National Standard ANSI / AAMI
SW91: 2018.
Alemzadeh, H., Iyer, R.K., Kalbarczyk, Z., Raman,
Jaishankar and Raman, Jai. (2013). Analysis of safety-
critical computer failures in medical devices. IEEE
Security and Privacy Magazine, 11(4), pp.14–26.
Black, R. (2008). Advanced Software Testing - Vol. 1. 2nd
ed. Santa Barbara: Rocky Nook Inc.
Bujok, A.B., MacMahon, S.T., Grant, P. and McCaffery, F.
(2017). Approach to the Development of a Medical
Device Software Quality Assurance Framework. In:
STV17 and INTUITEST 2017.
Chillarege, R., Bhandari, I.S., Chaar, J.K., Halliday, M.J.,
Ray, B.K. and Moebus, D.S. (1992). Orthogonal Defect
Classification-A Concept for In-Process
Measurements. IEEE Transactions on Software
Engineering, 18(11), pp.943–956.
Dawson, C. (2009). Introduction to Research Methods.
Dumas, J. and Sorce, J. (1995). Expert reviews: how many
experts is enough?. Proceedings of the Human Factors
and Ergonomics Society, 1(October 1995), pp.228–232.
Felderer, M. and Beer, A. (2013). Using defect taxonomies
to improve the maturity of the system test process:
Results from an industrial case study. In: SWQD 2013.
pp.125–146.
Graham, D., Veenendaal van, E., Evans, I. and Black, R.
(2006). Foundations of software testing; ISTQB
Certification. London: Cengage Learning Emea.
IEC. (2015). Medical device software — Software life-
cycle processes. Bs En 62304:2006 +a1:2015,
3(November 2008), p.88p.
ISO/IEC/IEEE. (2013). BSI Standards Publication
Software and systems engineering — Software testing
Part 2: Test processes.
ISO/IEC/IEEE. (2015). BSI Standards Publication
Software and systems engineering Software testing
Part 4 : Test techniques. , 2015.
ISTQB. (2010). Certified Tester Foundation Level
Syllabus.
ENASE 2021 - 16th International Conference on Evaluation of Novel Approaches to Software Engineering
338
Karahroudy, A.A. and Tabrizi, M.H.N. (1996). Software
Defect Taxonomy , Analysis and Overview. , 1996.
MacMahon, S.T., McCaffery, F. and Keenan, F. (2014).
The MedITNet assessment framework: development
and validation of a framework for improving risk
management of medical IT networks. Journal of
Software: Evolution and Process, 26(12), pp.1172–
1192.
Offenberger, S., Herman, G.L., Peterson, P., Sherman,
A.T., Golaszewski, E., Scheponik, T. and Oliva, L.
(2019). Initial Validation of the Cybersecurity Concept
Inventory: Pilot Testing and Expert Review.
Proceedings - Frontiers in Education Conference, FIE,
2019-Octob, pp.1–9.
Rajaram(&), H.K., Loane, J., MacMahon, S.T. and Fergal,
M. (2019). A Framework for Taxonomy Based Testing
Using Classification of Defects in Health Software-
SW91. Systems, Software and Services Process
Improvement, 2019, pp.606–618. Available from:
https://link.springer.com/chapter/10.1007/978-3-030-
28005-5_47.
Rajaram, H.K., Loane, J., MacMahon, S.T. and Caffery,
F.M. (2020). A Retrospective Study of Taxonomy
based Testing using Empirical Data from a Medical
Device Software Company. In: ICSOFT 2020.
Sjøberg, D.I.K., Dybå, T. and Jørgensen, M. (2007). The
future of empirical methods in software engineering
research. FoSE 2007: Future of Software Engineering,
2007, pp.358–378.
Expert Review of Taxonomy based Testing: A Testing Framework for Medical Device Software
339