A Comprehensive Review of Testing Blockchain Oriented Software
Mariam Lahami
1
, Afef Jmal Maalej
1
, Moez Krichen
1,2
and Mohamed Amin Hammami
1
1
ReDCAD Laboratory, National Engineering School of Sfax, University of Sfax, Tunisia
2
Faculty of CSIT, Al-Baha University, Saudi Arabia
Keywords:
Blockchain, Review, Dynamic Testing, Smart Contracts, BoS.
Abstract:
This work presents a short review on the most relevant studies in the context of testing Blockchain-Oriented
Software (BoS), especially at the smart contract level. Focusing on dynamic testing, we first provide a clas-
sification of 20 studies according to the accessibility of smart contract code. Second, we give an overview
of each identified work while highlighting its advantages and limitations. Third, we discuss challenges and
opportunities in this research area. After carrying out this review, it was noticed that there is much work to be
done, especially in the context of model-based testing of smart contracts which can be a good research line for
detecting defects and enhancing the quality of such applications.
1 INTRODUCTION
Nowadays, one of the most popular trends of In-
formation Technology (IT) is the blockchain (Jab-
bar et al., 2022). Firstly introduced in the white pa-
per of Nakamoto et al. in 2008 (Nakamoto et al.,
2008), Blockchain (BC) gathered a lot of interest in
the last decade by several organizations, especially
financial ones. BC is a distributed ledger that fol-
lows the Peer-to-Peer architecture in which each node
maintains a copy of the blockchain ledger. Its struc-
ture makes it distinguishable since it is composed of
blocks, linked together by means of cryptographic
functions, in which data are securely stored and can-
not be changed or tampered. All the participant nodes
in the blockchain network are running with respect
to a common consensus algorithm without the need
of a third party, such as financial institutions. Cur-
rently, blockchain platforms, such as Ethereum
1
and
Hyperledger Fabric
2
, give the possibility of devel-
oping decentralized applications with complex busi-
ness logic. They introduce the concept of “smart
contracts” which is defined as a collection of pro-
gramming code and data that runs on the blockchain
and it is executed to fulfill a given task when some
events and rules happened. By using smart contracts,
blockchain can be used not only to exchange cryp-
tocurrencies but also can be applied on several in-
dustry sectors such as property management, assur-
1
https://ethereum.org/en/
2
https://www.hyperledger.org/
ance and health (Fekih and Lahami, 2020). Although
the blockchain technology presents a novel approach
to develop decentralized applications between several
stakeholders, this technology may suffer from several
vulnerabilities and bugs within smart contracts. Due
to the decentralized nature of blockchain involving
anonymous nodes, malicious attackers may extract
digital assets from a contract, cause damage by lead-
ing a smart contract into a deadlock, which prevents
account owners from withdrawing or spending their
assets (Praitheeshan et al., 2019). For instance, 50
million dollars were stolen in the well-known ”DAO
attack”, which used a combination of smart-contract
vulnerabilities (Finley, 2016). Therefore, applying
testing techniques, before the deployment of smart
contracts, becomes a crucial task with the aim of in-
creasing confidence in blockchain oriented software
and avoiding attacks and asset losses.
In this direction, several efforts were made to
detect such vulnerabilities in smart contracts. Re-
searchers have used different approaches including
theorem proving, model checking (Nelaturu et al.,
2020) and software testing (Antonino and Roscoe,
2020). Although testing blockchain oriented software
(BoS) has gained the attention of many researchers,
we have not found either recent surveys or systematic
mapping studies that provide in depth an overview
of the considered research field, and identify exhaus-
tively the amount and quality of available research re-
sults related to it. To cope with this lack, this system-
atic literature review was conducted in which 20 stud-
Lahami, M., Maalej, A., Krichen, M. and Hammami, M.
A Comprehensive Review of Testing Blockchain Oriented Software.
DOI: 10.5220/0011042800003176
In Proceedings of the 17th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE 2022), pages 355-362
ISBN: 978-989-758-568-5; ISSN: 2184-4895
Copyright
c
2022 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
355
ies dealing with dynamic testing of Blockchain ori-
ented Software (BoS), dated from 2017 to November
2021, were deeply discussed. Moreover, we identi-
fied several challenges and future research guidelines
related to this topic.
The organization of this survey is as follows:
Section 2 provides background information about
blockchain technology, smart contracts, and the soft-
ware testing techniques. In Section 3, related surveys
are discussed. The state-of-the-art testing approaches
and techniques for performing different types of test-
ing on smart contracts are provided in detail in Sec-
tion 4. Section 5 includes a discussion based on
blockchain testing challenges and some future direc-
tions. Finally, section 6 concludes the paper.
2 PRELIMINARIES
2.1 Blockchain
The concept of Blockchain appeared for the first time
in the white paper of Nakamoto et al. (Nakamoto
et al., 2008) as the technology underlying Bitcoin.
Now, it garnered a lot of attention and is used in sev-
eral platforms such as Ethereum and Hyperledger. It
can be defined as a distributed ledger maintained over
a peer-to-peer network.
As illustrated in Fig. 1, BC is composed of a
growing list of blocks. Each block contains essen-
tially a given number of transactions that have oc-
curred within the network. The transaction can be
seen as token transfers or any manner of data ex-
change. Each block is composed of two parts: the
header and the body. Within the body of the block,
transactions are stored while the header contains sev-
eral fields, particularly a timestamp of when the block
was produced and the identifier of the previous block.
The latter is obtained by executing a cryptographic
hash function (e.g., SHA256, KECCAK256, etc.). In
this way, blocks are connected to each other like a
linked list (Ali et al., 2018).
Block 1
Header
Block 1 Body
Transactions are stored here
Hash of the previous
block header
Block 2
Header
Block 2 Body
Transactions are stored here
Hash of the previous
block header
Block 3
Header
Block 3 Body
Transactions are stored here
Hash of the previous
block header
Figure 1: Representation of a blockchain general scheme
(Ali et al., 2018).
2.2 Smart Contracts
One of the interesting features of recent blockchain
platforms like Ethereum and Hyperledger is the possi-
bility of attaching business logic code to transactions,
called Smart Contracts (SC). A SC is seen as a piece
of code stored in the blockchain, that is capable of self
execution when some conditions are met, without the
need of centralized authorizing entities.
For the case of the Ethereum platform, smart con-
tracts are written in a Turing complete language
3
,
called Solidity
4
and then they are compiled to the
Ethereum Virtual Machine (EVM) bytecode. Like
JavaScript language, Solidity supports features like li-
braries, inheritance and user-defined types. The most
relevant feature within smart contracts is their im-
mutability. Once deployed on the blockchain, they
cannot be altered or changed. Therefore, it is highly
required to ensure their correctness and security be-
fore their deployment on the blockchain platform.
2.3 Blockchain Testing Techniques
One of the most important activities for proper soft-
ware development is testing activity. In fact, it is de-
fined in (Freedman, 1991) as the process of validat-
ing and ensuring the quality of a System Under Test
(SUT). It is usually performed with the aim of assess-
ing the compliance of a system to its intended speci-
fications. Furthermore, software testing can be static
or dynamic. Static testing does not involve software
execution, but examines or analyses the source code
structure, syntax and data-flow, and is usually called
Static analysis
5
.
In contrast with static testing, dynamic testing in-
volves the software execution while feeding inputs
and producing outputs. Test cases are typically devel-
oped by specifying test inputs and expected outputs.
The purpose of testing here is to check whether the ac-
tual outputs correspond to the expected ones. For in-
stance, test cases can be designed to validate whether
the observed behavior of the tested software conforms
to its specification or not. This is mostly referred to in
the literature as Conformance testing. Dynamic Test-
ing can be used to verify different properties either
functional or non-functional.
Functional Testing: can be classified into three
categories:
3
It is worthy noting that Ethereum supports several pro-
gramming languages (e.g., Vyper) and compilers.
4
https://solidity.readthedocs.io/
5
Static analysis can be used with formal verification
methods with the aim to prove the correctness of a given
software.
ENASE 2022 - 17th International Conference on Evaluation of Novel Approaches to Software Engineering
356
Black-box testing technique: it focuses on ex-
amining the software functionalities without
any knowledge of source code. It requires
the system specification and it is usually per-
formed at the integration and system level. Typ-
ical black-box testing techniques include essen-
tially model-based testing (MBT) and fuzz test-
ing
6
(Klees et al., 2018).
White-box testing technique: it is the detailed
investigation of internal logic and structure of
the code. This technique requires a full knowl-
edge of source code and it is usually applied at
unit testing level.
Grey-box testing technique: it provides com-
bined benefits of black-box and white-box test-
ing techniques. It requires a partial knowledge
of internal structure of the application.
Non-functional Testing: aim to check non-
functional requirements, such as reliability, per-
formance and security.
In the case of testing blockchain oriented software, we
identify in the literature several kinds of testing tech-
niques that are performed with the aim of increasing
confidence and trustworthiness of BoS. For instance,
we cite:
Smart contract testing: It consists in performing
detailed functional testing of business logic and
process.
Performance testing: It verifies the performance
and the latency within the Blockchain network.
Security testing: It performs security validation at
transaction, block, and network levels.
Node testing: It aims at checking the integrity of
the network and the overall ledger by verifying
that all newly added transactions are saved ade-
quately.
3 RELATED SURVEYS
Verification and validation of blockchain oriented
software have gained the attention of many re-
searchers. In this direction, we have identified re-
cent comprehensive surveys and systematic literature
reviews that focus on verification of smart contracts.
For example, authors in (Liu and Liu, 2019) propose
an in-depth survey in which they study papers dealing
with the security assurance and papers related to the
correctness verification (i.e., using formal techniques
6
Fuzzing is an automated testing technique that consists
in generating random data as inputs to a software.
such as model checking and theorem proving). More-
over, they summarize research done in this area and
point out new research directions in smart contract se-
curity and correctness.
Similarly, another recent survey, dated on 2020
(Tolmach et al., 2020), analyses and classifies exist-
ing approaches to formal modeling, specification, and
verification of smart contracts. First, this work out-
lines various types of properties from different smart
contract domains (e.g., functional correctness and se-
curity). Second, it discusses the current approaches,
tools and frameworks used in verifying such prop-
erty specifications. It covers a wide range of pa-
pers dealing with model checking, theorem proving,
program verification, symbolic execution and test-
ing. From testing perspective, it analyses only four
fuzzing-based approaches. Also, we notice that its
focus is on smart contract functional behavior and
does not discuss problems related to other blockchain-
related execution aspects, such as performance, scal-
ability, consensus, etc.
Authors in (S
´
anchez-G
´
omez et al., 2020) de-
velop a systematic literature review carried out in
the field of smart contract development life cycle.
They review essentially the state of art of formal
smart contract modeling and code generation. A re-
duced number of studies dealing with testing BoS
were briefly discussed. We have identified another
study (Koul, 2018) which underlines several chal-
lenges faced when testing BoS. It also acknowledges
the need to reuse specialized tools and techniques for
blockchain-oriented software testing with the aim of
ensuring high levels of quality. Nevertheless, the au-
thors do not address nor analyze the current research
in this topic. To the best of our knowledge, we have
not found recent surveys that provide comprehensive
studies related to software testing topic and identify
exhaustively the amount and quality of available re-
search results related to it. Therefore, our proposal
targets recent research work by analyzing and clas-
sifying approaches and tools in the area of software
testing of BoS, discussing their challenges as well as
the open research directions for future work.
4 TESTING BLOCKCHAIN
ORIENTED SOFTWARE
Although testing blockchain oriented software should
cover several layers : application layer (e.g., DApps),
smart contract layer, blockchain layer (e.g., blocks,
transactions), consensus layer and network layer, test-
ing efforts are concentrated essentially on testing
smart contracts and ensuring their correctness.
A Comprehensive Review of Testing Blockchain Oriented Software
357
A wide range of papers focus on security testing
and on detecting vulnerabilities with respect to the so-
lidity language. . Up to our knowledge, only (Yuan
and Zhu, 2020) proposes a novel approach that deals
with performance testing of blockchain system. The
proposed test tool measures several test metrics such
as delay, transaction execution success rate and re-
source consumption (i.e., CPU usage, memory usage,
etc.). Moreover, existing work on testing BoS can be
categorized into static testing approaches like and dy-
namic testing ones. Due to space limitation, we focus
in this review on studying only dynamic testing ap-
proaches.
4.1 White-box Testing Approaches
This research line is based on the investigation of in-
ternal logic and structure of the smart contract code.
Up to our knowledge, the majority of studied papers
deal with testing solidity source code and some of
them claim that their proposals can be extended to
other languages and platforms. For example, the au-
thors in (Driessen et al., 2021) present and partially
validate AGSOLT, a tool that creates test suites aim-
ing to achieve branch coverage for Solidity smart con-
tract unit testing. AGSOLT works with both a ran-
dom testing and a guided-search approaches. It was
tested on a set of real-world smart contracts from
GitHub. However, AGSOLT was not validated for
permissionned/commercial smart contracts.
A wide range of papers focus on mutation testing
(Li et al., 2019; Ivanova and Khritankov, 2020; Wang
et al., 2019b; Akca et al., 2019; Andesta et al., 2020;
Hartel and Schumi, 2020) and show that this testing
technique has a good effect on smart contract quality.
In fact, mutation testing is considered as a fault-based
software testing technique generally used to evaluate
the adequacy of test cases and their fault detection ca-
pabilities. In other words, its purpose is to help testers
in identifying limitations of the test suites.
In this direction, a well established approach is
proposed in (Li et al., 2019) providing a mutation test-
ing tool for Ethereum smart contracts called, MuSC.
This proposal takes as input a smart contract under
test and transforms its source files to Abstract Syn-
tax Tree (AST) version. Next, it generates various
mutants that implement traditional mutation opera-
tors and new ones according to the characteristics
of solidity language. The obtained mutants are then
transformed back to solidity source files with injected
faults for compilation, execution and testing purposes.
It also provides user-friendly interface to create test
nets and to display test reports. The latter include
execution results for each mutant (i.e., pass or fail)
and the total mutation score. Similarly, authors in
(Ivanova and Khritankov, 2020) developed a Regu-
larMutator tool for mutation analysis. Its major goal
is to improve the test suites in order to find defects
as well as to increase the effectiveness and the fault
detection capabilities of test suites. Taken as input a
Truffle project, RegularMutator generates mutants for
each source file in the project. Once mutant files are
generated, it substitutes the original files with the mu-
tant ones, executes project test suites, and then, the
test output is analysed. The main problem within this
approach is its high computational cost of executing a
set of tests when generating numerous mutants.
In (Wang et al., 2019b), the authors deal with the
problem of Ethereum smart contracts test generation,
targeting three main objectives, minimizing uncov-
ered branch coverage, time cost, and gas cost. They
also proposed a random based and an NSGA-II based
multi-objective approach to seek cost-effective test-
suites. The authors lead their empirical study on a set
of smart contracts in the most widely used Ethereum
decentralized applications and verified that their pro-
posed approaches could significantly reduce both the
gas cost and the time cost while retaining the abil-
ity to cover branches. To ameliorate their work, the
authors need to conduct more experiments and to em-
ploy more optimization objectives as maximizing mu-
tation killed ability during the test-suite generation.
The authors present in (Akca et al., 2019) a fully
automated technique, SolAnalyser, for vulnerability
detection over Solidity smart contracts that uses both
static and dynamic analysis. They also implemented
a tool that injects different types of vulnerabilities in
smart contracts and used the mutated contracts for as-
sessing the effectiveness of different analysis tools.
The main contributions in this paper concern static
checks, developing of a test generator and a fault
seeding tool, run-time monitoring and empirical eval-
uation. The authors intend to generate inputs that pro-
vide control flow coverage within functions and trans-
actions in Solidity contracts. This tool is considered
effective in terms of execution time due to the use
of AST code representation. In contrast to Regular-
Mutator (Ivanova and Khritankov, 2020), it did not
compare mutation operators for their effectiveness in
detection vulnerabilities and did not explore the pos-
sibility of using the results to improve tests.
In (Andesta et al., 2020), the authors propose a
testing mechanism for Solidity smart contracts based
on mutation testing. They analyzed a comprehen-
sive list of known bugs in such smart contracts and
designed classes of mutation operators inspired by
the real faults. In this work, the authors extended
the Universal Mutator tool with their proposed mu-
ENASE 2022 - 17th International Conference on Evaluation of Novel Approaches to Software Engineering
358
tation operators to automatically generate mutants
for smart contracts written in Solidity. In (Hartel
and Schumi, 2020), the authors evaluate the quality
of smart contract mutation testing at scale using re-
play tests that were automatically generated from the
Ethereum blockchain. They propose a set of muta-
tion operators on the basis of existing works and eval-
uate these operators at scale. The authors introduce
a novel killing condition based gas limits for smart
contract transactions aiming to improve the mutation
score. A first limitation of this work concerns a threat
to the validity of the proposed evaluation in the way
that the authors only consider a replay test suite that
is less powerful than other testing techniques, which
might get a higher mutation score. Another limita-
tion regarding the validity of the proposed approach
might be that it is not wise to kill a mutant only based
on a different gas usage since it could still be seman-
tically equal. Even it is widely considered as an ef-
fective method of enhancing the adequacy of testing,
mutation testing is challenging and hard to be used in
industry due to its high cost. It requires a high under-
standing of the semantics of the smart contract lan-
guage to generate only useful mutants.
Another promising research line is identified
that consists in testing Decentralized Applications
(DApp). A DApp is a Web application composed of
two parts: the front-end and the back-end. The iden-
tified approaches (Gao et al., 2019; Wu et al., 2020)
touch several research areas including smart contract
analysis and automated Web application testing. They
overcome the lack of effective methods and tools for
testing DApps since the existing ones either focus on
testing front-end code or back-end programs but they
ignore the interaction between them. For instance, au-
thors in (Gao et al., 2019) propose a tool called Sun-
gari that employs first random events to infer the in-
teraction between browser-side code and smart con-
tracts. Second, the proposed technique generates test
cases and orders them based on a data flow graph
of smart contracts. In the same direction, the Kaya
framework cited in (Wu et al., 2020) reduces the com-
plexity of testing DApps by providing GUI and CLI
tool. It consists of three modules: a Kaya Graph-
ical User Interface (KGUI), Core Function Module
(CFM) and Log Analyzer (LA). Within Kaya, test en-
gineers may produce easily test cases with a simple
setting and then Kaya will execute them in an auto-
matic manner. This is done by simulating front-end
events, setting Blockchain pre-states (i.e., runtime en-
vironment of test cases) and finally running smart
contracts. The Log Analyzer compares the initial val-
ues of parameters in SCs and the generated outputs,
then, it produces a report illustrating such analysis.
4.2 Black-box Testing Approaches
As already mentioned, Black-box testing includes
several kinds of testing techniques that apply testing
activities without having any knowledge of the inter-
nal structure of BoS. The most used ones in the stud-
ied context are fuzz testing and model-based testing
(Krichen, 2007; Krichen, 2018).
From fuzz testing perspective, we identify the
Fuse project (Chan and Jiang, 2018) which is a fuzz
testing service for smart contracts and Dapp testing.
It assists developers for test diagnosis via test sce-
nario visualization. The first prototype developed in
the context of Fuse project is ContractFuzzer (Jiang
et al., 2018) that detects seven security vulnerabili-
ties of Ethereum smart contracts. The proposed ap-
proach generates fuzzing inputs from the ABI
7
spec-
ification of the smart contract. It also defines test or-
acles for detecting the supported real world vulner-
abilities within smart contracts. ContractFuzzer was
performed on 6991 real-world Ethereum SCs showed
that it has identified 459 SCs vulnerabilities, includ-
ing the DAO and Parity Wallet attacks.
A closely approach to ContractFuzzer is sFuzz,
an adaptive fuzzing engine for EVM smart contracts
(Nguyen et al., 2020). sFuzz is composed of three
components: runner that manages test case execu-
tion, liboracles that supports eight oracles inspired
by the previous researches (Jiang et al., 2018; Luu
et al., 2016) and libfuzzer which implements the test
suite generation algorithm. The latter is based on
a feedback-guided fuzzing technique which trans-
forms the test generation problem into an optimiza-
tion problem and uses feedbacks as an objective func-
tion in solving the optimization problem. This pro-
posal is based on adaptive strategy since it is possible
to change the objective function adaptively based on
the feedback to evolve the test suite with the aim of
improving its branch coverage. Due to its effective-
ness and its reliability, sFuzz has already gained in-
terest from multiple companies and research organi-
zations. However, fuzz-based approaches may suffer
from false positive detection as a reported vulnerabil-
ity may be a false positive
8
.
From the model-based testing perspective, authors
in (Liu et al., 2020) introduce an MBT tool, namely
ModCon, that uses an explicit abstract model of the
target SC in order to derive tests automatically. Mod-
Con shows its effectiveness specifically for enter-
prise SC applications written in Solidity from permis-
sioned/consortium blockchains. It allows SC devel-
7
Application Binary Interface
8
Some test cases fail but there is no bug and the program
is working correctly.
A Comprehensive Review of Testing Blockchain Oriented Software
359
opers to input their test model for the SC under test.
Certainly, the efficiency of MBT depends both on the
input test model and fault model (or test hypotheses).
A promising research challenge could be to investi-
gate SCs and their faults to determine suitable fault
models or test assumptions. In the same direction,
authors in (S
´
anchez-G
´
omez. et al., 2019) propose a
model driven approach that generates smart contract
code from UML diagrams (i.e., Use Cases and Ac-
tivity diagrams). They also point out the necessity of
applying testing technique in the early stage of Soft-
ware Development Life Cycle (SDLC), especially in
the context of blockchain oriented software. It out-
lines the necessity of applying model-based testing
process to generate test scenarios from requirements
(UML use cases) and test cases from UML activity
diagrams. However, this approach is still immature
since no transformation rules for the smart contract
generation code are given and no real tool implemen-
tation for the discussed ideas were presented. Simi-
larly, the work in (Kakadiya, 2017) proposes a com-
plete software testing life cycle to test BoS projects.
The proposal is composed of four phases including
system overview, test design, test planning and test
execution.
4.3 Grey-box Testing Approaches
Another research line is identified. It consists in ap-
plying testing techniques with a partial knowledge of
internal BoS structure. In this direction, the work
of Wang et al. (Wang et al., 2019a) combines fuzz
and mutation testing in order to detect several smart
contract vulnerabilities triggered during transactions.
The proposed tool called ContraMaster is a grey-box
fuzzing approach that dynamically executes transac-
tions and observes their actual effects on the contract
states in order to detect vulnerabilities.
ContraMaster includes a novel feedback mech-
anism to guide efficiently the fuzzing process, by
taking into consideration the data-flow information,
control-flow information and states of smart con-
tracts. It also monitors the executions of target smart
contracts, and validates the obtained results against a
general purpose semantic test oracle to detect vulner-
abilities. Compared to traditional fuzzing approaches
like ContractFuzz (Jiang et al., 2018), this proposal
has not shown any false positives, and it easily gener-
alizes to unknown types of vulnerabilities.
HARVEY (W
¨
ustholz and Christakis, 2019) is an-
other grey-box Fuzzer for smart contracts, which is
the first one used in industry by one of the largest
blockchain-security consulting companies. It is based
on a novel technique that systematically forecasts new
inputs for the smart contract under test in order to in-
crease the performance and effectiveness of the grey-
box fuzzing approach. In contrast to existing works
based on grey-box fuzzing, this proposal suggests
concrete input values based on data from past execu-
tions, instead of performing arbitrary mutations.
Similar to ContraMaster tool, HARVEY gener-
ates, executes, and fuzzes sequences of transactions,
which invokes the contract’s functions. To allevi-
ate the exploration of the search space of all possi-
ble sequences, authors use a demand-driven sequence
fuzzing technique, avoiding the generation of transac-
tion sequences that cannot further enhance the cover-
age.
Another promising study was proposed in (Liao
et al., 2019). This work introduces SoliAudit tool that
combines machine learning and Fuzz testing for smart
contract vulnerability assessment. Indeed, it em-
ploys machine learning using Solidity machine code
as learning features to verify 13 kinds of vulnerabili-
ties. The latter are listed as Top 10 threats by an open
security organization
9
.
Moreover, the tool includes a grey-box fuzz test-
ing mechanism, composed of a fuzzer contract and a
simulated blockchain environment for on-line trans-
action verification. Similar to (Wang et al., 2019a),
SoliAudit can detect vulnerabilities without prede-
fined patterns. The source code is only required if the
user wants to identify the vulnerability location in the
source code. It only needs the ABI and the bytecode
to deploy on the blockchain.
Yet another potential work to explore is the HF-
ContractFuzzer tool (Ding et al., 2021). Compared to
the majority of cited studies that focus on Ethereum
smart contracts written in solidity language, this tool
is specific to a Hyperledger Fabric smart contract
written in Go language. Indeed, authors deal with
the security vulnerability detection while combining
Fuzzing technology and Go language testing tool,
called go-fuzz.
To do so, HFContractFuzzer takes as input smart
contracts and test cases. Then, it automatically gen-
erates test results. In order to improve the efficiency
of smart contract fuzzing, this proposal gives two op-
timization methods of the go-fuzz tool : optimizing
the generation of the initial Fuzzing corpus and op-
timizing the mutation process. However, it requires
more effort to evaluate its vulnerability detection and
to measure its performance.
9
DASP Top 10: https://dasp.co/
ENASE 2022 - 17th International Conference on Evaluation of Novel Approaches to Software Engineering
360
5 CHALLENGES AND FUTURE
DIRECTIONS
5.1 Challenges
1. Smart Contract Testing still a hard task for devel-
opers: Despite the growing interest generated by
its use in a wide range of applications, the use and
the validation of smart contract technology is still
considered as tough mission for many developers,
mainly because of its innovative design.
2. Formal methods for smart contract testing still
overlooked: Formal testing is regarded as a crucial
part of the software development process. Formal
testing, on the other hand, has been largely dis-
regarded in the smart contract development cycle.
Although a shift of formal testing from standard-
ised software to smart contract-based software is
quite conceivable.
3. Performance tools still not enough evaluated: De-
spite the fact that several techniques for evalu-
ating Blockchain performance have been devel-
oped, only a few of them have been thoroughly ex-
amined due to the lack of standardised interfaces
while running workloads.
4. Choice of performance metrics: Many developers
utilize various types of coverage to demonstrate
the effectiveness of testing. Although this is an
intuitive indicator, there is no inescapable link be-
tween increased coverage and improved vulnera-
bility detection efficiency.
5.2 Future Directions
1. Making Smart Contract Testing easier: The most
important thing regarding this point is to define
some specific techniques which may enable de-
velopers to establish a testing procedure dedicated
for smart contracts.
2. Formal methods for smart contract testing: At this
level too, there is a real need to make formal test-
ing techniques more accessible for smart contracts
developers by providing more sophisticated tools
which are both efficient and easy to use at the
same time.
3. Performance tools: these tools must evolve in or-
der to meet the need of developers and to meet
high-quality levels. In particular, they need to be
equipped with standardised interfaces which may
be used while running workloads.
4. Choice of performance metrics: At this level, new
criteria and performance metrics have to be de-
fined in order to improve the efficiency of vulner-
ability detection.
6 CONCLUSION
The main motivation for this study was to investigate
the state of art in testing Blockchain oriented Soft-
ware. This review was conducted to determine what
issues have been studied and by what means, also to
provide a guide for researchers in this emergent re-
search area. Up to our best knowledge, the existing
surveys covered essentially static testing and analysis.
Dynamic testing approaches were rarely addressed.
Therefore, this paper is a significant contribution in
the literature which includes over 20 research stud-
ies published from 2017 to 2021. A classification of
these approaches was reported while providing their
strengths and weaknesses. In summary, the obtained
results of this study indicated that most research pa-
pers are from conference proceedings, suggesting that
the area of research is still young. The proposed re-
search direction may lead to advances in three rela-
tively open fields of software engineering research:
(1) Model based testing of BoS, (2) Performance test-
ing of blockchain networks and (3) Testing consensus
algorithms. We believe that to improve the current
state of art in these three lines, further work needs to
be done and more exploration is required.
REFERENCES
Akca, S., Rajan, A., and Peng, C. (2019). Solanalyser: A
framework for analysing and testing smart contracts.
In 2019 26th Asia-Pacific Software Engineering Con-
ference (APSEC), pages 482–489.
Ali, M. S., Vecchio, M., Pincheira, M., Dolui, K., An-
tonelli, F., and Rehmani, M. H. (2018). Applications
of blockchains in the internet of things: A comprehen-
sive survey. IEEE Communications Surveys & Tutori-
als, 21(2):1676–1717.
Andesta, E., Faghih, F., and Fooladgar, M. (2020). Testing
smart contracts gets smarter. In 2020 10th Interna-
tional Conference on Computer and Knowledge Engi-
neering (ICCKE), pages 405–412.
Antonino, P. and Roscoe, A. (2020). Formalising and veri-
fying smart contracts with solidifier: a bounded model
checker for solidity. arXiv preprint arXiv:2002.02710.
Chan, W. and Jiang, B. (2018). Fuse: An architecture for
smart contract fuzz testing service. In The 25th Asia-
Pacific Software Engineering Conference (APSEC),
pages 707–708.
Ding, M., Li, P., Li, S., and Zhang, H. (2021). Hfcontract-
fuzzer: Fuzzing hyperledger fabric smart contracts for
vulnerability detection. CoRR, abs/2106.11210.
A Comprehensive Review of Testing Blockchain Oriented Software
361
Driessen, S., Nucci, D. D., Monsieur, G., and van den
Heuvel, W.-J. (2021). Agsolt: a tool for automated
test-case generation for solidity smart contracts.
Fekih, R. B. and Lahami, M. (2020). Application of
blockchain technology in healthcare: A comprehen-
sive study. In The Impact of Digital Technologies on
Public Health in Developed and Developing Coun-
tries - 18th International Conference, ICOST 2020,
Hammamet, Tunisia, June 24-26, 2020, Proceedings,
pages 268–276.
Finley, K. (2016). A $50 million hack just showed that the
dao was all too human.
Freedman, R. (1991). Testability of Software Compo-
nents. IEEE Transactions on Software Engineering,
17(6):553 –564.
Gao, J., Liu, H., Li, Y., Liu, C., Yang, Z., Li, Q., Guan,
Z., and Chen, Z. (2019). Towards automated test-
ing of blockchain-based decentralized applications. In
IEEE/ACM 27th Int. Conf. on Program Comprehen-
sion (ICPC), pages 294–299.
Hartel, P. and Schumi, R. (2020). Mutation testing of smart
contracts at scale. In Ahrendt, W. and Wehrheim, H.,
editors, Tests and Proofs, pages 23–42, Cham.
Ivanova, Y. and Khritankov, A. (2020). Regularmutator:
A mutation testing tool for solidity smart contracts.
Procedia Computer Science, 178:75–83.
Jabbar, R., Dhib, E., ben Said, A., Krichen, M., Fetais, N.,
Zaidan, E., and Barkaoui, K. (2022). Blockchain tech-
nology for intelligent transportation systems: A sys-
tematic literature review. IEEE Access.
Jiang, B., Liu, Y., and Chan, W. K. (2018). Contractfuzzer:
Fuzzing smart contracts for vulnerability detection.
ASE 2018, page 259–269.
Kakadiya, A. (2017). Block-chain oriented software test-
ing approach. International Research Journal of En-
gineering and Technology (IRJET).
Klees, G., Ruef, A., Cooper, B., Wei, S., and Hicks, M.
(2018). Evaluating fuzz testing. In Proceedings of
the 2018 ACM SIGSAC Conference on Computer and
Communications Security, page 2123–2138.
Koul, R. (2018). Blockchain oriented software testing -
challenges and approaches. In 3rd International Con-
ference for Convergence in Technology (I2CT), pages
1–6.
Krichen, M. (2007). Model-based testing for real-time sys-
tems. PhD thesis, PhD thesis, PhD thesis, Universit
Joseph Fourier (December 2007).
Krichen, M. (2018). Contributions to model-based test-
ing of dynamic and distributed real-time systems.
PhD thesis,
´
Ecole Nationale d’Ing
´
enieurs de Sfax
(Tunisie).
Li, Z., Wu, H., Xu, J., Wang, X., Zhang, L., and Chen, Z.
(2019). Musc: A tool for mutation testing of ethereum
smart contract. In 2019 34th IEEE/ACM Interna-
tional Conference on Automated Software Engineer-
ing (ASE), pages 1198–1201.
Liao, J.-W., Tsai, T.-T., He, C.-K., and Tien, C.-W. (2019).
Soliaudit: Smart contract vulnerability assessment
based on machine learning and fuzz testing. In
Sixth International Conference on Internet of Things:
Systems, Management and Security (IOTSMS), pages
458–465.
Liu, J. and Liu, Z. (2019). A survey on security verification
of blockchain smart contracts. IEEE Access, 7:77894–
77904.
Liu, Y., Li, Y., Lin, S.-W., and Yan, Q. (2020). Modcon: A
model-based testing platform for smart contracts. In
Proceedings of the 28th ACM Joint Meeting on Eu-
ropean Software Engineering Conference and Sym-
posium on the Foundations of Software Engineering,
page 1601–1605.
Luu, L., Chu, D.-H., Olickel, H., Saxena, P., and Hobor, A.
(2016). Making smart contracts smarter. In Proceed-
ings of the 2016 ACM SIGSAC Conference on Com-
puter and Communications Security, page 254–269.
Nakamoto, S. et al. (2008). Bitcoin: A peer-to-peer elec-
tronic cash system.
Nelaturu, K., Mavridou, A., Veneris, A., and Laszka, A.
(2020). Verified development and deployment of
multiple interacting smart contracts with verisolid.
In Proc. of the 2nd IEEE International Conf. on
Blockchain and Cryptocurrency (ICBC).
Nguyen, T. D., Pham, L. H., Sun, J., Lin, Y., and Minh, Q. T.
(2020). Sfuzz: An efficient adaptive fuzzer for solid-
ity smart contracts. In Proceedings of the ACM/IEEE
42nd International Conference on Software Engineer-
ing, page 778–788.
Praitheeshan, P., Pan, L., Yu, J., Liu, J. K., and Doss,
R. (2019). Security analysis methods on ethereum
smart contract vulnerabilities: A survey. CoRR,
abs/1908.08605.
S
´
anchez-G
´
omez., N., Morales-Trujillo., L., and Torres-
Valderrama., J. (2019). Towards an approach for ap-
plying early testing to smart contracts. In Proceedings
of the 15th International Conference on Web Infor-
mation Systems and Technologies - APMDWE,, pages
445–453.
S
´
anchez-G
´
omez, N., Torres-Valderrama, J., Garc
´
ıa-Garc
´
ıa,
J. A., Guti
´
errez, J. J., and Escalona, M. J.
(2020). Model-based software design and testing in
blockchain smart contracts: A systematic literature re-
view. IEEE Access, 8:164556–164569.
Tolmach, P., Li, Y., Lin, S., Liu, Y., and Li, Z. (2020). A
survey of smart contract formal specification and ver-
ification. CoRR, abs/2008.02712.
Wang, H., Li, Y., Lin, S.-W., Artho, C., Ma, L., and Liu,
Y. (2019a). Oracle-supported dynamic exploit gener-
ation for smart contracts.
Wang, X., Wu, H., Sun, W., and Zhao, Y. (2019b). To-
wards generating cost-effective test-suite for ethereum
smart contract. In IEEE 26th International Confer-
ence on Software Analysis, Evolution and Reengineer-
ing (SANER), pages 549–553.
Wu, Z., Zhang, J., Gao, J., Li, Y., Li, Q., Guan, Z., and
Chen, Z. (2020). Kaya: A testing framework for
blockchain-based decentralized applications. In IEEE
International Conference on Software Maintenance
and Evolution (ICSME), pages 826–829.
W
¨
ustholz, V. and Christakis, M. (2019). Harvey: A greybox
fuzzer for smart contracts. CoRR, abs/1905.06944.
Yuan, C. and Zhu, J. (2020). A new performance testing
scheme for blockchain system. In Zhang, J., Dres-
ner, M., Zhang, R., Hua, G., and Shang, X., editors,
LISS2019, pages 757–773.
ENASE 2022 - 17th International Conference on Evaluation of Novel Approaches to Software Engineering
362