Testing Environment Emulation - A Model-based Approach
Jian Liu
1
, John Grundy
2
, Mohamed Abdelrazek
2
and Iman Avazpour
2
1
School of Software and Electrical Engineering,
Swinburne University of Technology, Hawthorn, VIC 3122, Australia
2
School of Information Technology, Deakin University, Burwood, VIC 3125, Australia
Keywords: Model-driven Engineering, Domain-specific Visual Modeling Language, Testing Environment Emulation.
Abstract: Modern enterprise software systems often need to interact with a large number of distributed and
heterogeneous systems. As a result, integration testing has become a critical step in their software
development lifecycle. Service virtualization is an emerging technique for creating testing environments
with realistic executable models of server side production-like behaviours. However, building models in
existing service virtualization approaches is very challenging, requiring either significant human effort or
the availability of interactive tracing records. In this paper, we present a domain-specific modeling approach
to generate complex, virtualized testing environments. Our approach allows domain experts to use a suite of
domain-specific visual modeling languages to model key interface layers of applications at a high level of
abstraction. These layered models are then transformed into a testing runtime environment for application
integration testing. We have conducted a technical comparison with two other existing approaches and also
carried out a user study. The user study demonstrated the acceptance of our new testing environment
emulation approach from software testing experts and developers.
1 INTRODUCTION
Testing environment emulation provides integration
testing to an enterprise System Under Test (SUT) that
interacts with many external systems. Currently, there
are two kinds of approaches to develop such
integration testing environments. Specification-based
approaches are used by IT professionals to develop
simplified versions of applications with external
behaviour only (often called “endpoints”) manually
(Hine et al., 2009, Yu et al., 2012). They perform this
using available knowledge of the underlying message
syntax, interaction protocol and system behaviour.
Interactive tracing data record-and-replay based
approaches (called “interactive tracing” hereafter)
create endpoint models from recorded request-
response pairs between the endpoint system and an
earlier version of a SUT automatically (Du et al.,
2013). Each endpoint’s simulated response is
generated by finding a close-match request in the
previously recorded trace database.
Both approaches have their strengths but also
shortcomings. Specification-based approaches have
high development and set-up cost and require access
to a detailed system specification and/or
implementations, if available. Interactive tracing
approaches depend on the availability of trace records
for all integration testing cases between a SUT and its
operational environment. In recent years, the
interactive tracing approaches are getting more
popular, and many major players have released their
commercial products (Giudice, 2014). However,
these products need to have a complementary
specification-based development tool for modeling
those endpoints, which do not have all interactive
tracing data available.
Aiming to achieve high development productivity
and ease of use for domain experts, we have
developed a novel specification-based Domain-
Specific Modeling (DSM) approach for testing
environment emulation. Our approach is based on
model-driven engineering, where users build high
level abstract service models and executable code will
be generated automatically by transforming these
models using code generators. Our DSM approach
divides software interfaces into different abstraction
layers, where each layer represents a modeling
problem domain – e.g. endpoint signature, protocol or
behaviour. The approach introduces a suite of
Domain-Specific Visual Languages (DSVLs) for
Testing Environment Emulation (TeeVML) (Liu et
al., 2016), each is dedicated to a specific interface
layer. Users use TeeVML to model testing endpoints
112
Liu J., Grundy J., Abdelrazek M. and Avazpour I.
Testing Environment Emulation - A Model-based Approach.
DOI: 10.5220/0006194601120124
In Proceedings of the 5th International Conference on Model-Driven Engineering and Software Development (MODELSWARD 2017), pages 112-124
ISBN: 978-989-758-210-3
Copyright
c
2017 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
in layers. A testing runtime environment is then
provided by Axis2 Web Service platform
(Jayasinghe, 2008), together with the Java code
generated automatically from endpoint models.
The key contributions of this paper include: (1) a
solution to model endpoint external behaviours for
enterprise application integration testing, (2) a
software interface description framework to abstract
testing endpoints into multiple logic layers, (3) and a
model-driven, domain-specific approach to generate
emulated testing endpoints using our TeeVML
toolset. The scope of our approach is for emulating
complex interactions between a SUT and an endpoint.
Thus other interactions behind the endpoint for
providing composite services are not considered in
this study. The applications include those using
Remote Procedure Call (RPC) communication style
and with stateful session management as a typical
business scenario. However our approach can be
generalized to real-world examples.
The remainder of this paper is organized as
follows: Section 2 motivates our work with an
example case study, followed by an introduction of
our approach in Section 3. In Section 4, we show how
an endpoint is modeled and then describe the steps to
convert endpoint models into testing runtime
environment. In Section 5, we evaluate our approach
and discuss the key findings from the results of a
technical comparison and a user survey. This is
followed by a review of related work in section 6.
Finally, we conclude this paper and identify some key
future works in Section 7.
2 MOTIVATION
Assume a company has an in-house Enterprise
Resource Planning (ERP) system to support its daily
operations. For the purpose of streamlining its sales
process and improving internal efficiency, the
company plans to introduce a public cloud-hosted
Customer Relationship Management (CRM)
application. To ensure the interconnectivity and
mutual operability between the ERP system and
CRM application, integration testing must be
conducted before putting the CRM in production. For
this study, we treat the cloud CRM as the SUT, and
the ERP as the testing endpoint to be emulated.
The sequence diagram in Figure 1 illustrates a
typical purchase process, where a sales representative
uses the CRM application to place a Purchase Order
(PO) for his/her client. Our main interest is on the
interactions between the testing endpoint and the SUT,
which represent the endpoint protocol behaviours. We
describe the interactive behaviour between the CRM
and ERP below.
Figure 1: A CRM and an ERP sales process flow diagram.
Whenever the endpoint receives a logon request
(#1) from its SUT, it transits from Idle state to Home
state and an interactive session starts. The next valid
operation is a PO request (#2), and followed by an
inventory check (#3). The returned value of the
inventory check will determine whether or not
supplier chain related steps will be executed. If the
purchase item has enough stock for the PO, the
process flow will jump over those supplier purchasing
steps and directly go to a payment request (#8) state.
Otherwise, we have to go through all supplier
purchase steps (#4, #5, #6 and #7) to buy the missing
quantity of the PO item. Supplier PO approval (#5)
and approval notification (#6) are iteration operations,
informing all approvers one-by-one to give his/her
approval. If all required approvals for the supplier PO
have been obtained, the rest of purchasing steps will
be executed in the order as in Figure 1. Otherwise, the
purchase process will be aborted without success.
In addition, there are some other important
protocol behaviours: (1) Timeouts – a timeout event
will automatically terminate an interactive session and
the endpoint state will be changed from Home to Idle,
if no valid operation request is received within a
defined period of time; (2) Synchronous operation
simulation – if an endpoint operation is in
synchronous mode, all further operation requests will
be rejected when it is processing the operation; and (3)
Unsafe operation simulation -- i.e. not an idempotent
operation that will produce the same result if
executed once or multiple times. The payment request
Testing Environment Emulation - A Model-based Approach
113
(#8) is considered to be an unsafe operation, and
multiple requests for a same operation are not
allowed.
It is infeasible to test the CRM with the
production ERP system, and there is large cost
involved in duplicating the ERP. Conventional
interactive tracing and specification-based approaches
are similarly infeasible or difficult to use, as the
former relies on existing interactive tracing data and
the latter requires development of detailed endpoint
model implementations.
Just as any other software development tools,
users’ primary concerns about our endpoint modeling
approach will be: what can it do for their service
emulation modeling and generation, will it improve
endpoint development productivity, how easily can it
be used.
Therefore, we have defined the following three
research questions to be addressed by our approach
described in this paper:
1) RQ1 – Can we emulate an integration testing
environment capable of detecting interface defects
of an existing or a non-existing system under test
from a high level service model?
2) RQ2 – Would such a model-based approach
improve testing environment development
productivity, compared to using third generation
languages or specification-based manual coding
approaches?
3) RQ3 – Can we develop a user centric approach,
easy enough to learn and use for specifying testing
endpoints by domain experts?
3 OUR APPROACH
A testing endpoint is a server-side application,
receiving, validating and processing operation
requests from a SUT. Our goal is to make the
emulated testing environment rich enough to “fool”
the SUT that it is talking to the real system.
Specifically, an endpoint is a simplified version of its
real system with three assumptions: (1) only external
behaviours of the real system are considered and all
internal implementations will be ignored; (2) only the
operations of the real system to be invoked by the
SUT are provided; and (3) all SUT interface defects,
together with their types and origin information,
should be able to be detected and reported.
3.1 Domain Analysis
To identify endpoint common entities and find out
their relationships, we conducted our testing
environment emulation domain analysis by
investigating three applications interacting with their
clients. These applications included the ERP system
introduced in Section 2, a LDAP server and an e-
commerce application. These applications represent a
variety of application domains in a typical enterprise
environment. The domain analysis focused on two
areas: the interaction abstraction between a service
provider and a service consumer, and the requirement
on integration testing environment. From the domain
analysis, we proposed a layered software interface
description framework for testing environment
emulation, and defined interface defect types to be
detected by endpoints. Consecutively, we designed
our modeling approach for each interface layer. The
detailed design of the TeeVML visual notations are
out of scope for this paper. Interested readers can refer
to our previous publication (Liu et al., 2016).

3.2 Software Interface Description
Framework
Our new layered software interface description
framework builds on top of Han’s comprehensive
interface definition framework for software
components (Han, 2000). Our framework logically
separates software interfaces into three horizontal and
two vertical layers. Horizontal layers include
signature, protocol and behaviour. Vertical layers
include data store (data persistence) and Quality-of-
Service (QoS) (or call non-functional requirement). A
SUT operation request is processed horizontally by an
endpoint step by step from signature, protocol, down
to interactive behaviour layer. Whenever an error
occurs at any layer, the request process will be
terminated.
The signature and protocol layers act as message
pre-processors for checking the correctness of an
operation request syntax and temporal sequence,
before handing it over to the behaviour layer for
generating response. Vertical layers are not directly
involved in request processing, but provide support to
horizontal layers. We use a modular development
approach to model an endpoint – i.e. each module
represents a particular interface layer.
In this paper, we focus on modeling endpoint
functional layers. The data store layer is integrated in
the behaviour layer. The Quality of Service
requirements is part of our future work.
3.3 Interface Defects
Corresponding to endpoint horizontal and vertical
layers, there are also two types of interface defects:
MODELSWARD 2017 - 5th International Conference on Model-Driven Engineering and Software Development
114
functional defects, which are directly related to
operation request processing, such as incorrect
message signature and invalid operations; and non-
functional defects, such as non-compliance with
endpoint security policy. Table 1 lists all the
functional defect types a SUT operation request may
contain.
Table 1: Operation request defect types.
No Defect Type Description
Signature
S1
An operation request is not an operation provided
by endpoint.
S2
The parameters in an operation request are not
matched with the parameters of the corresponding
operation provided by endpoint, in terms of
parameters’ name, data type and order in the
operation request.
S3
One or more operation request mandatory
parameter(s) is (are) missing.
S4
One or more parameter(s) in an operation request is
(are) beyond the defined value range of the
corresponding endpoint operation.
Protocol
P1
An operation request is invalid for the current
endpoint state.
P2
An operation request is invalid for the current
endpoint state, as one or more parameter(s)
violate(s) the defined constraint condition(s).
P3
An operation request is invalid for the current
endpoint state, as one or more returned value(s)
from a previous operation request violate(s) the
defined constraint condition(s).
P4
An operation request is invalid, due to endpoint
state transition driven by some internal event, such
as time out.
P5
An operation request is invalid, as endpoint is in
processing a synchronous operation request.
P6
An operation request is invalid, as one such request
for an unsafe operation has been received by
endpoint.
Table 1 does not include any behaviour defect
types. This is because a SUT’s obligation is to send
correct operation requests to endpoint and the way
these requests are to be processed is defined in the
endpoint’s internal implementation. The reason why
we still model endpoint behaviour is that the validity
of alternative requests may depend on what values are
returned in a response message it has received for a
previous operation request (refer to P3 defect type of
Table 1). Below we discuss how these request defect
types can be detected.
3.4 Signature Modeling Approach
Endpoint signature is modeled by a signature DSVL.
We used Web Service Definition Language (WSDL)
1.1 as its model and adopted a three-level architecture
design (refer to Figure 2). The top-level WSDL DSVL
(refer to Figure 2a) is used to define the five WSDL
entity types: Service, Port, Binding, PortType and
Operation. To link these entities together, we added
two relationships: Composition and Association. The
middle-level operation DSVL (refer to Figure 2b) is
used to define request and/or response message(s)
contained in an operation. The bottom-level message
DSVL (refer to Figure 2c) is based on the W3C XML
Schema 1.1 for defining complex elements in a
message. By using the multi-level modeling approach,
lower level models can be reused by upper level
models.
Figure 2: Signature metamodel.
The benefits from using WSDL specification as
our signature DSVL metamodel include: (1) WSDL
supports RPC communication style, covering a wide
range of endpoint signature types; (2) we can use
Axis2 wsdl2java utility to generate Axis2 Web
Service engine as our domain framework
automatically; and (3) Axis2 engine provides a
signature defect detection mechanism from a WSDL
file definition. There are some open-source or
commercial WSDL tools available, such as Eclipse
WTP Plugin and XMLSpy. The motivation for
developing our own WSDL tool is to increase the
consistency among different parts of TeeVML.
Behaviour model imports operations and their
parameters from the corresponding signature model;
and message DSVL is reused to define data store
model.
Testing Environment Emulation - A Model-based Approach
115
The signature defects S1 to S3 in Table 1 can be
detected by the Axis2 Web Service engine. For S4
defect debugging, two fields are added to the element
type of message DSVL for specifying the minimum
and maximum values of a request parameter.
3.5 Protocol Modeling Approach
A standard operation-driven Finite State Machine
(FSM) is often used to represent endpoint protocol
behaviour (Hine et al., 2009). To deal with incomplete
protocol specification problems and capture runtime
constraints, we used an Extended Finite State
Machine (EFSM) to enrich our protocol modeling
capability with dynamic protocol aspects. Our EFSM
adds one entity type and two entity properties (marked
yellow in Figure 3). The entity type is the
InternalEvent, which is used to define state transitions
triggered by a time event. One of the entity properties
is the StateTransitionConstraint of transition entity,
and it is for specifying either static or dynamic
constraints on state transition function. Another one is
the StateTimeProperty of state entity, which allows
users to simulate synchronous and unsafe operations.
As endpoint protocol modeling is relatively simpler
than the other two layers, we use a flat view
presentation structure.
Figure 3: Protocol metamodel.
All the protocol defect types listed in Table 1 can
be detected by a testing endpoint, developed by a
modeling tool based on the EFSM model: (1) P1 – the
operation-driven state transition FSM; (2) P2 and P3
the StateTransitionConstraint property of transition
entity; (3) P4 – the InternalEvent entity; and (4) P5
and P6 – the StateTimeProperty of state entity.
3.6 Behaviour Modeling Approach
Endpoint behaviour DSVL was designed based on the
dataflow programming paradigm (Sousa, 2012). We
chose this metaphor as it allows complex specification
of behaviour models but is understandable by a wide
range of software stakeholders. The dataflow
programming execution model is represented by a
directed graph; the nodes of the graph are data
processing units, and the directed arcs between the
nodes represent data dependencies. Data flow in each
node from its input connector; and the node starts to
process and convert the data whenever it has the
minimum required parameters available. The node
then places its execution results onto its output
connector for the next nodes in the chain. Data store
operators are used to access and manipulate persistent
data.
To handle complicated business logics, we
designed our behaviour DSVL using hierarchical
structure. The benefits from using the hierarchical
structure are two-fold: First, we can reuse some of the
nodes, if they perform exactly the same task but are
located at different components. Second, it can help us
to manage diagrammatic complexity problem. At the
bottom level, a node consists of some primitive visual
constructs for performing operations on data and flow
controls. At the top level, there are discrete service
nodes to represent all operations provided by an
endpoint. To prevent the data inconsistence between
behaviour model and signature model, each of the
service nodes imports its request and response
parameters from the same endpoint signature model.
3.7 Testing Runtime Environment
Our testing runtime environment is generated by
transforming the endpoint models using a set of code
generators. There are four code generators: (1) a
signature code generator to transform an endpoint
signature model to a WSDL file, (2) a protocol code
generator to transform an endpoint protocol model to a
protocol processing class, (3) a behaviour code
generator to transform an endpoint behaviour model to
several behaviour model classes for handling all
operation requests from a SUT, and (4) a data store
code generator to transform data store and operation
models to JDBC classes.
An Axis2 Web Service engine, generated by Axis2
wsdl2java utility, binds both server side and client side
implementations to a service contract defined by a
signature WSDL file. On the server side, Axis2
provides a skeleton class as interface for integrating
business logic processing Java classes. On the client
side, Axis2 also has a stub class for allowing client to
access the server operations. We developed Java API
classes for facilitating the integration with SUTs. To
provide the integration testing service to a SUT, Axis2
MODELSWARD 2017 - 5th International Conference on Model-Driven Engineering and Software Development
116
service must be placed into a servlet container. We use
Tomcat 7.0 as our Web application server. Figure 4
illustrates our integration testing runtime environment,
where a SUT is on the right-hand side and a testing
endpoint is on the left-hand side. They interact with
each other by SOAP over HTTP protocol.
Figure 4: Testing runtime environment deployment view.
4 USAGE EXAMPLE
We use the ERP system from Section 2 as an example
to show how a testing endpoint is modeled by use of
TeeVML and how an integration testing runtime
environment is built.
4.1 Signature Modeling
We start signature modeling by specifying the five
WSDL entity types using WSDL DSVL: Service,
Port, Binding, PortType and Operation. Then, we link
them together by using either a composition or an
association relationship. All the entity types are
instantiated by providing their names. In addition, the
operation has a pattern property with a value of
“in/out”, “in-only” or “out-only”; and the port must
specify the endpoint service address.
We use an operation named paymentrequest as an
example to show how operations can be modeled. The
operation contains paymentrequest_request and
paymentrequest_response messages, and they are
defined by operation DSVL. The message label is “in”
for the request message and “out” for the response
message. The elements in the request and response
messages are defined by using message DSVL. The
request message contains only one element pono
(purchase order number), and it is defined as integer
and mandatory. Since a valid pono is a five-digit
integer, the element’s minimum field is specified as
10000 and maximum field as 99999. The response
message consists of three elements: amount,
errorcode and errormessage. They are placed in the
message in alphabetic order. The amount is a float
data type, errorcode integer and errormessage string.
Figure 5 illustrates the
hierarchical signature
model of the ERP system endpoint. It contains the top-
level WSDL model (refer to Figure 5a; for a better
view representation, we only show five operations),
the middle-level paymentrequest operation model
(Figure 5b), and the bottom-level request and response
message models (Figure 5c).
Figure 5: Example signature model.
4.2 Protocol Modeling
We use protocol DSVL to model the testing endpoint
protocol layer. Figure 6 illustrates the endpoint
protocol model, where the emulated enterprise
purchase process flows in clockwise direction. To
explain how the endpoint protocol is modeled, we
select three typical protocol behaviours of interactive
session management, constraint state transition and
transition iteration. They are marked as A, B and C in
the diagram, respectively.
A -- Session management: Endpoint protocol
modeling starts from specifying an interactive session
by using a logon transition relationship from Idle state
to Home state. On the opposite direction, a logout
transition relationship terminates a session. A session
can also be terminated by a timeout event, which is
Testing Environment Emulation - A Model-based Approach
117
defined by using a timeout relationship linking a from
state to a to state.
B – Constraint transition relationship: When the
endpoint is at inventorycheck state, there are
alternative flows either to supplierpo or to
paymentrequest. The choice of the flows is subject to
whether the purchase item stock can meet the PO
requirement. We use a constraint transition
relationship to link the inventorycheck state to the
supplierpo state. Its constraint condition is specified in
the relationship dialog box by comparing the quantity
parameter of porequest request with the inventory
parameter of inventorycheck response. If the former is
greater than the latter, the state transition will happen.
Similarly, we specify another constraint transition
from the inventorycheck state to the paymentrequest
state, and the constraint condition is the item stock
less than or equal to the PO quantity.
C – Transition iteration: A loop relationship is used
to specify that all the operations between the from
state and the to state of the loop relationship will be
repeatedly executed. We use a loop relationship to
specify the approval process of a supplier PO, which
includes an approvalnotification and a
supplierpoapproval operations. The approval process
starts from the immediate manager of the purchaser
until the manager with authority for the PO amount.
Figure 6: Example protocol model.
4.3 Behaviour Modeling
We use one operation paymentrequest as example to
explain how endpoint behaviour is modeled by our
behaviour DSVL. We model an operation behaviour
by using a service node construct and instantiate it by
providing the operation name. The operation request
and response parameters are imported from the
corresponding signature operation model
automatically. The paymentrequest operation node
consists of two sub nodes: poinformationretrieve to
retrieve PO, product and client information, and
poamountcalculation to calculate the total PO amount.
These two nodes are placed between a pair of input
and output bars.
The poinformationretrieve node is used to show
how behaviour DSVL visual constructs are used to
implement business logics. Figure 7 illustrates the
operations and dataflows within the
poinformationretrieve node. The node has one input
parameter pono, and four output parameters: quantity,
unitprice, discount and errormessage. The node
includes three data query operations: (1) to retrieve
PO category, item, quantity and clientname from
PurchaseOrderTable by the pono; (2) to retrieve
unitprice from ProductTable by the category and
item; and (3) to retrieve discount from ClientTable by
the clientname. If searching records are found,
searching results will be placed on the normal output
port (black circle) of data store operator. Otherwise, a
FatalError variable will be assigned by following the
exceptional output port (yellow circle).
Figure 7: Example behaviour model.
4.4 Testing Environment Creation
We build our testing runtime environment by
converting the above models into executable codes to
be run inside Axis2. We use Eclipse as our Java IDE;
and two Java projects purchaseserver and
purchaseclient are created for hosting server and
client side codes. The testing environment creation
process is described as followings:
Testing environment platform creation: The
signature model is transformed to a WSDL file, and
the file is copied to both the client and server project
folders. Then, the WSDL files in both project folders
are converted to Axis2 Web Service platform by
using Axis2 wsdl2java utility.
MODELSWARD 2017 - 5th International Conference on Model-Driven Engineering and Software Development
118
Protoc
ol and behaviour models transformation and
integration: The endpoint protocol and behaviour
models are transformed to Java classes; and the Java
classes are copied to the purchaseserver folder. Then,
these Java classes are integrated into an Axis2
skeleton class. Figure 8 shows a code snippet of the
skeleton class for paymentrequest operation. Signature
parameters are verified against defined value ranges
first at Line 8. Then, the protocol validity of the
paymentrequest operation is tested at Line 9. If both
the signature and protocol are verified correctly, the
operation response message will be generated from
Line 14 to 17.
Figure 8: A code snippet of Axis2 skeleton class.
Axis2 Web Service generation and deployment:
We developed an Apache Ant build XML file to
build the endpoint Axis2 Web Service automatically,
and the built service purchaseservice.aar file is
loaded to Tomcat webapps folder for providing the
endpoint testing service.
SUT integration: A Java API file is provided for
integrating a SUT with Axis2 stub file in the
purchaseclient folder. Figure 9 shows the
paymentrequest API file. A SUT accesses the
endpoint testing service through a Tomcat application
server URI at Line 8. The paymentrequest_request is
sent by providing input parameter pono at Line 10.
The Paymentrequest _response method is defined and
assigned the response parameters from the testing
endpoint at Line 12 and 13. Finally, these response
parameters are assigned to the API return string array
“retMessage” from Line 15 to 17.
x
Figure 9: An Axis2 stub class API.
5 EVALUATION
To assess how well the three research questions are
addressed by our new endpoint modeling approach,
we define three corresponding evaluation criteria: (1)
testing endpoint functionality (addressing RQ1) – the
approach should be able to develop various types of
testing endpoints, which could be used to detect all
sorts of interface defects; (2) development
productivity (addressing RQ2) – the approach should
ideally have high endpoint development productivity
with less development effort and time; and (3) ease of
use (addressing RQ3) – the approach should be easy
to learn and adapt.
These criteria were assessed by a technical
comparison of the two currently available testing
endpoint emulation approaches: specification-based
manual coding and interactive tracing approaches.
This comparison motivated our new specification-
based DSM approach. Thus, we provide details of
how our approach compares with these two other
approaches as well. After our approach was ready to
use, we conducted a user study to evaluate to what
extent our approach is accepted by IT professionals in
respect to each of these criteria. It is also good to
mention that we have made some improvements on
our early versions of TeeVML based on the feedbacks
from the user study.
5.1 Technical Comparison
We conducted the technical comparison by looking
into what key techniques these approaches adopt to
address the issues related to the evaluation criteria
Testing Environment Emulation - A Model-based Approach
119
Table 2: Approaches’ techniques.
Manual coding Interactive tracing Domain-specific modeling
Functionality
The key motivation of these
approaches is to provide SUT
performance testing by emulating
large number of endpoints of the
same type. To achieve this
objective, these approaches adopt a
light-weight architecture design
(Hine et al., 2009) and some testing
features are deliberately neglected.
Dynamic protocol behaviour cannot
be modeled, as state transition is
triggered only by an operation.
Unless great effort is made,
behaviour layer modeling will be
limited.
To provide integration testing, these
approaches search for the right
request matching on data byte level
without any knowledge about upper-
level message syntax. They can only
tell whether a test is passed or
failed, but cannot provide any defect
information. These approaches are
not usable for testing a new
application, as its trace data are not
available.
Our testing endpoint provides
integration testing from signature,
protocol and behaviour abstraction
layers. The signature layer model
supports all RPC style
communications; the protocol layer
can model both static and dynamic
protocol behaviours; and the
behaviour layer uses a hierarchical
structure dataflow programming for
modeling complicated logic
implementation.
Productivity
The approaches adopt a modular
architecture design, where an
endpoint type dependent message
engine module is separated from an
endpoint type independent network
infrastructure and a system
configuration module (Hine et al.,
2009). However, as the message
engine is coded manually,
significant amount of development
effort is needed for each new
endpoint type.
Testing endpoint is created by
recording the interactive tracing
data between an endpoint and an
earlier version of the SUT
application. These approaches do
not need any endpoint development
work, but some effort on trace data
recording.
An endpoint is modeled by layers,
and layer models are transformed to
executable source codes. The key
solution to productivity
improvement is to maximize
components reusability. We have
adopted multi-level design for the
signature DSVL and node
hierarchical structure for the
behaviour DSVL.
Ease of Use
To develop an endpoint, developers
must have both business domain
knowledge and programming skills.
Neither business domain knowledge
nor programming skills are
required. However, during the trace
recording, developers must have a
certain level of understanding of the
endpoint application. Also, users
must be trained to use the tool.
Developers must have business
domain knowledge, and some
modeling skill is preferred. To
achieve ease of use, we applied the
principles of Physics of Notations
(Moody, 2009) to optimize our
visual nation designs.
(refer to Table 2). We have added some attributes to
these evaluation criteria. We then gave a four-point
ranking subject to the level of support (N – n/a, L -
low, M - medium or H - high) the approaches provide
for each attribute and each evaluation criterion (refer to
Table 3). The overall ranking of each evaluation
criterion summarizes individual attribute’s ranking and
takes their importance into consideration.
The interactive tracing approaches have the H
ranking for both the productivity and ease of use, as
they create testing endpoints from interactive tracing
data automatically. However, these approaches have
two key shortcomings in terms of testing functionality.
One is their usability, which is subject to the
availability of interactive tracing data; another one is
that they cannot report defect type and cause
information. So, we give these approaches the L
ranking for the testing functionality.
In contrast, both specification-based types of
approaches need to develop endpoint by using
different techniques. As our DSM approach uses
higher level of abstraction models than code to express
design intent, we have given our approach the M
ranking and manual coding the L ranking for both the
productivity and ease of use. Both types of approaches
can report static defects, but our DSM approach can
report dynamic defects as well. So, DSM approach
ranks H for the testing functionality, while manual
coding is given the M ranking.
5.2 User Study
Our user study was conducted in two phases to
measure the perceived usefulness and perceived ease
of use (Davis, 1989) of our endpoint modeling tool,
respectively. Phase One study was conducted through
MODELSWARD 2017 - 5th International Conference on Model-Driven Engineering and Software Development
120
interviews, including an introduction to the toolset
and a Q&A session. We invited 16 testing experts,
who had two or more years industry testing
experience and were knowledgeable about integration
testing, to take part in the survey. For Phase Two, we
asked participants to model the deposit operation of a
banking system endpoint. We then collected their
opinions based on their experience with our TeeVML
toolset. Total of 19 software developers and IT
research students took part in the survey. Most of
them had five or more years software development
experience and were familiar with at least one third
generation language.
Table 3: Approaches comparison.
(MC: Manual coding, IT: Interactive tracing,
DM: Domain-specific modeling).
Attribute Description
MC IT DM
Functionality
The approach detects all interface defects.
MH H
The approach reports signature defects.
H N H
The approach reports static protocol
defects.
H N H
The approach reports dynamic protocol
defects.
L N H
Business scenario can be simulated (time
event, synchronous and unsafe operations).
L N H
Overall ranking of functionality
ML H
Productivity
Endpoint is generated automatically.
N H N
The approach supports high-level
abstraction.
L N H
The approach supports components reuse.
MN H
The approach has built-in error prevention
mechanisms.
H N M
Network interface is generated
automatically.
L N H
Overall ranking of productivity
L H M
Ease of Use
Special training is needed.
N L M
Endpoint application knowledge is needed.
L N L
Programming skills are needed.
L N N
Visual notations are used.
N N H
Overall ranking of ease of use
L H M
The questionnaires included 5-point Likert Scale,
single-choice and multi-choice questions. We counted
the frequency of participant responses to measure the
degree of acceptance to a question statement. We had
total 58 questions and selected some of them for this
paper results presentation. The full result reports can
be accessed at:
https://sites.google.com/site/teevmlapsec/
5.2.1 Phase One Results
The main objective of Phase One survey is to
evaluate the first criterion -- testing endpoint
functionality. We selected four questions from Phase
One questionnaire (refer to Table 4) to analyse this
criterion from two different angles. The first one is
about participants’ acceptance of emulated testing
endpoint in general and by each interface layer. The
second one is to find out possible reasons why the
participants would consider using (or not using) our
emulated testing endpoints in their future projects.
Table 4: Phase One user study results.
No Statement
Fre
q
uenc
y
5 4 3 2 1
Q8
In your opinion, an emulated
testing environment is useful
for an application inter-
connectivity and inter-
operability test.
8 6 0 1 1
Q9
What kinds of testing features do you want to see
an emulated testing environment provides to
system under test for interconnectivity and inter-
operability test?
Correctness of message signature
13
Correctness of interactive protocol
16
Correctness of interactive behaviou
r
14
Correctness of non-
f
unctional requirement
11
Othe
r
1
Q
13
What are the main motivations for you to use
emulated testing environment?
Cost saving on application hardware and
s
oftware investment
14
Effort saving on application installation and
maintenance
10
Lack of application knowledge
5
Early detection of interface defects
15
Q
14
What are your main concerns, which could prevent
you from using emulated testing environment?
Extra development effort on testing endpoints
6
L
earning a new technology
6
Inadequate testing functionality
7
Emulation accuracy
7
Result reliability
12
Q8 on usefulness of emulated testing
environments received 14 out of 16 in favour
response rate (scoring 4 or 5). This is a good
indication of participants’ acceptance of the overall
usefulness of testing endpoints. From Q9 we see that
protocol layer received in favour responses from all
participants. The reason could be that most
applications do not have a well-documented protocol
specification. SUT protocol related defects can only
be found by conducting integration testing.
Testing Environment Emulation - A Model-based Approach
121
Responses to Q13 indicate that the top reason for
using endpoints is early detection of interface defects.
Integration testing is normally conducted during the
later stages of software development lifecycle. This is
partly because integration testing environment is not
available before then. If a rapid and cheap solution
for testing environment deployment was available,
developers may have preferred to conduct at least part
of integration testing earlier. Responses to Q14
indicates that most participants’ concerns are on the
reliability of testing endpoint results. We believe the
main reason behind is that software developers are
used to using real applications for their integration
testing. However, an endpoint is actually a simplified
version of its real application. This might have some
impacts on SUT testing results.
5.2.2 Phase Two Result
In Phase Two, we evaluated the ease of use and
development productivity criteria. The former used
the ten questions from Software Usability Scale
(SUS) (Brooke, 1996) (refer to Q12 to Q21 of Table
5), and the latter was captured by the three questions
specifically related to performing the assigned task
(refer to Q9, Q10, Q22 of Table 5).
The responses to the SUS questions were quite
positive with average 85% in favour. Particularly, all
participants did not agree (scoring 1 or 2) with Q17
negative statement on tool’s inconsistency. Q15
received 8 out of 19 in favour responses and 9
participants voted neutral. This result is confirmed by
Q10, where only 4 participants did not ask for support
for finishing their tasks. We believe this is due to the
fact that our introduction video was targeted toward
introduction of the tool and the approach in general
rather than stepwise instruction of using the tool for
similar examples. As a result, more participants felt
they needed to ask for instructor’s support. We
believe this will be rectified overtime with more
usage of the approach and tool support.
Q22 captures participants’ opinions on how much
of their time and effort will be reduced through using
our approach. 12 out of 19 respondents chose “50% -
80%” and “80%+” and no participant voted “almost
the same”. As a result, we can conclude that most
participants believed that our approach could increase
endpoint development productivity. Confirming this
is the fact that 15 out of 19 participants finished their
task in less than 30 minutes (see responses to Q9).
Based on this result, we can generalize that it is
possible to model a relatively complex endpoint with
more than ten operations within a day through using
our tool support for testing environment emulation.
Table 5: Phase Two user study results.
No Statement
Fre
q
uenc
y
5 4 3 2 1
Q9
How long did it take you to complete the task?
10
15 minutes
1
16 – 20 minutes
4
21
25 minutes
7
26
30 minutes
3
30+ minutes
4
Q
10
How many times have you asked for support?
N
one
4
One time
4
Two times
4
Three times
5
Four times or more
2
Q
12
You would like to use the tool
in your future project.
7 11 1 0 0
Q
13
You found the tool
unnecessarily complex.
0 1 2 12 4
Q
14
You found the tool was easy to
use.
8 10 1 0 0
Q
15
You would need support to be
able to use the tool.
0 2 9 8 0
Q
16
You found the various features
of the tool were well integrated.
8 10 0 1 0
Q
17
You found there was too much
inconsistency in the tool.
0 0 0 11 8
Q
18
You would image that most
people would learn to use the
tool very quickly.
5 12 1 1 0
Q
19
You foun
d
the tool very
cumbersome to use.
0 0 2 10 7
Q
20
You felt very confident using
the tool.
4 13 2 0 0
Q
21
You needed to learn a lot of
things before you could get
going with the tool.
0 1 3 8 7
Q
22
In your opinion, comparing to a third generation
language (e.g. Java) you are familiar with, how
much would a typical endpoint development effort
be reduced by using the tool?
A
lmost the same
0
10 – 25%
2
26
50%
6
51
80%
9
81%+
2
6 RELATED WORK
Testing distributed systems is a complex problem.
Ghosh and Mathur raised nine issues to be addressed
in testing distributed systems (Ghosh and Mathur,
1999). Method stubs, mock objects, and existing
emulation approaches have been used to emulate the
behaviour of endpoint systems (Gibbons, 1987,
Freeman et al., 2004, Hine et al., 2009, Du et al.,
2013, Yu et al., 2012, Giudice, 2014). However,
MODELSWARD 2017 - 5th International Conference on Model-Driven Engineering and Software Development
122
some of these approaches introduce a large
implementation overhead; while the others depend on
the availability and accessibility of existing SUT
applications.
Pact is an open-source tool, enabling consumer
contract driven testing (Pact, 2016). It provides a
DSL for users to specify operation requests and
expected responses from service provider. However,
each interaction in Pact is verified in isolation without
context maintained from previous interactions.
Therefore, Pack is not suitable for performing
protocol testing.
Model-driven engineering is an avenue to raise the
level of abstraction beyond programming by
specifying solution directly using problem domain
concepts (Poole, 2001). UML is a general purpose
modeling language, its visual presentation makes it
understandable by a wide range of software
stakeholders. UML Testing Profile (UTP) provides a
generic extension mechanism for the automation of
test generation processes (Schieferdecker et al., 2003);
state chart simulates finite-state automaton (Zhang and
Liu, 2010); and activity diagram graphically
represents workflows of stepwise activities and
actions (Dumas and Terhofstede, 2001). However,
there are two main problems with using UML to
define new modeling languages (Abouzahra et al.,
2005): one is usually hard to remove parts of UML
that are not relevant in a specialized language; another
one is that all diagram types have restrictions based on
UML semantics.
DSLs are a model-driven development approach,
where the first class entities are the models and the
model transformations (Selic, 2003). DSLs often
support higher-level abstraction than general purpose
modeling languages (e.g. UML), so they require less
programming effort and low-level details to specify a
given system. DSLs have been widely used in various
of business and technical domains, such as, WebDSL
for web applications (Visser, 2008), MaramaEML for
business process modeling (Li et al., 2014) and SDL
(Kim et al., 2015) for supporting statistical survey
process. In contrast, we use a suite of DSVLs tailored
to modeling signature, protocol and behaviour aspects
of endpoints.
7 SUMMARY
Modern software development needs quick yet cost
effective solutions to develop and deploy integration
testing environment. Due to their unique advantage on
endpoint development productivity over others,
interactive tracing based service virtualization
approaches are gaining momentum in recent years.
But they still need a specification-based tool to specify
testing endpoint, as application trace data may be
neither available nor usable.
Our model-driven approach divides a software
interface into three abstraction functional layers, and
a suite of domain-specific visual languages have been
developed for modeling these layers. By this layered
modeling, we have achieved high development
productivity and rich testing functionality. In
addition, our approach supports partial endpoint
development, where a testing endpoint may have only
one or two of these layers to meet SUT testing
requirement.
Existing specification-based testing environment
emulation approaches cannot validate SUT’s runtime
protocol behavior, as they check the validity of an
incoming service request based on endpoint state
only. On the other hand, our protocol model is based
on an EFSM that allows our behavior models to
capture dynamic protocol aspects. Furthermore, our
testing environment provides a rich set of functions
for simulating typical business scenarios, such as
time-driven state transition, synchronous and unsafe
services.
In a realistic enterprise environment, application
security requirements may put extra constraints on the
validity of operation requests. Some of the constraints
are role-based, so that some operations are accessible
only to a certain group of users. Other constraints are
security policy related, such as restriction on available
time or specific pattern required for some operation
parameters. Also, there are some performance
limitations such as response time, and robustness
requirements such as handling endpoint malfunctions.
These and other QoS requirements are part of our
future works.
ACKNOWLEDGMENTS
The authors gratefully acknowledge support for this
research by an Australian Post-graduate Award and
an Australian Research Council Discovery Project
DP140102185 grant.
REFERENCES
Abouzahra, A., Bézivin, J., Del Fabro, M. D., Jouault, F.,
2005. A Practical Approach to Bridging Domain
Specific Languages with UML Profiles. Proceedings of
The Best Practices For Model Driven Software
Development At Oopsla, Citeseer.
Testing Environment Emulation - A Model-based Approach
123
Brooke, J., 1996. Sus-A Quick and Dirty Usability Scale.
Usability Evaluation in Industry, 189, 4-7.
Davis, F. D., 1989. Perceived Usefulness, Perceived Ease
Of Use, And User Acceptance Of Information
Technology. Mis Quarterly, 319-340.
Du, M., Schneider, J.-G., Hine, C., Grundy, J. & Versteeg,
S., 2013. Generating Service Models by Trace
Subsequence Substitution. Proceedings of the 9th
International Acm Sigsoft Conference on Quality of
Software Architectures. Canada: Acm.
Dumas, M., Terhofstede, A., 2001. Uml Activity Diagrams
as a Workflow Specification Language. Uml
2001—The Unified Modeling Language. Modeling
Languages, Concepts, and Tools. Springer.
Freeman, S., Mackinnon, T., Pryce, N., Walnes, J., 2004.
Mock Roles, Objects. In Companion to the 19th Annual
Acm Sigplan Conference on Object-Oriented
Programming Systems, Languages, and Applications.
Canada: Acm.
Ghosh, S., Mathur, A. P., 1999. Issues in Testing
Distributed Component-Based Systems. First Icse
Workshop on Testing Distributed Component-Based
Systems, Citeseer.
Gibbons, P. B., 1987. A Stub Generator for Multilanguage
Rpc in Heterogeneous Environments. Ieee Transactions
On Software Engineering, 13, 77-87.
Giudice, D. L., 2014. The Forrester Wave™: Service
Virtualization and Testing Solutions. In: Forrester
(Ed.).
Han, J., 2000. Rich Interface Specification for Software
Components. Peninsula School of Computing and
Information Technology Monash University,
Mcmahons Road Frankston, Australia.
Hine, C., Schneider, J.-G., Han, J., Versteeg, S., 2009.
Scalable Emulation Of Enterprise Systems. Software
Engineering Conference, Australian, Ieee, 142-151.
Jayasinghe, D., 2008. Quickstart Apache Axis2, Packt
Publishing Ltd.
Kim, C. H., Grundy, J., Hosking, J., 2015. A Suite of
Visual Languages for Model-Driven Development of
Statistical Surveys and Services. Journal of Visual
Languages & Computing, 26, 99-125.
Li, L., Grundy, J., Hosking, J., 2014. A Visual Language
and Environment for Enterprise System Modelling and
Automation. Journal of Visual Languages &
Computing, 25, 253-277.
Liu, J., Grundy, J., Avazpour, I., Abdelrazek, M., 2016. A
Domain-Specific Visual Modeling Language for
Testing Environment Emulation. Ieee Symposium on
Visual Languages and Human-Centric Computing.
Cambridge, Uk.
Moody, D. L., 2009. The “Physics” Of Notations: Towards
A Scientific Basis For Constructing Visual Notations In
Software Engineering. Software Engineering, Ieee
Transactions On, 35, 756-779.
Pact., 2016. Enables Consumer Driven Contract Testing
[Online]. Available: Https://Github.Com/Realestate-
Com-Au/Pact [Accessed].
Poole, J. D., 2001. Model-Driven Architecture: Vision,
Standards and Emerging Technologies. Workshop on
Metamodeling and Adaptive Object Models, Ecoop.
Schieferdecker, I., Dai, Z. R., Grabowski, J., Rennoch, A.,
2003. The Uml 2.0 Testing Profile and Its Relation to
Ttcn-3. Testing Of Communicating Systems. Springer.
Selic, B., 2003. The Pragmatics of Model-Driven
Development. Ieee Software, 20, 19.
Sousa, T. B., 2012. Dataflow Programming Concept,
Languages and Applications. Doctoral Symposium on
Informatics Engineering.
Visser, E., 2008. Webdsl: A Case Study in Domain-
Specific Language Engineering. Generative and
Transformational Techniques in Software Engineering
Ii. Springer Berlin Heidelberg.
Yu, J., Han, J., Schneider, J.-G., Hine, C., Versteeg, S.,
2012. A Virtual Deployment Testing Environment for
Enterprise Software Systems. Proceedings of the 8th
International Acm Sigsoft Conference on Quality of
Software Architectures. Italy: Acm.
Zhang, S. J., Liu, Y., 2010. An Automatic Approach to
Model Checking Uml State Machines. Fourth
International Conference on Secure Software
Integration and Reliability Improvement Companion
(Ssiri-C), 1-6.
MODELSWARD 2017 - 5th International Conference on Model-Driven Engineering and Software Development
124