A Preliminary Mapping Study of Software Metrics Thresholds

Elisabetta Ronchieri and Marco Canaparo

INFN, CNAF, Viale Berti Pichat 6/2, 40126, Bologna, Italy

Keywords:

Software Metrics, Thresholds, Mapping Study.

Abstract:

Many papers deal with the topic of thresholds in software metrics to determine the quality level of a software

project. This paper aims to identify the status of inﬂuential software metrics thresholds papers. We use search

facilities in the SCOPUS Web tool to establish the cited papers published from 1970 to 2015. We classiﬁed

the selected papers according to different factors, such as the main topic and the general type. The cited papers

were more frequently on journals than conference proceedings. We observed three main problems: an unclear

explanation of the method for selecting the technique that calculates thresholds; a direct application of the

metric threshold values to different code context; a lack of objective analysis for the calculated thresholds. To

our knowledge, this paper is the only one that performs this kind of study. It can provide baselines to assist

new research and development efforts. Due to the page limit, this paper contains a summary of the results.

1 INTRODUCTION

Different organizations have proposed software met-

rics to measure code characteristics. For instance,

the IEEE deﬁnes software metrics as the quantitative

measure of the degree to which a system, component

or process possesses a given software attribute

Metrics have been used successfully for quantiﬁ-

cation, whereas they have generally failed to support

subsequent decision-making (Fenton and Neil, 2000).

The use of metrics entails thresholds to determine if

a certain value is normal or anomalous. Thresholds

give semantics to metrics enabling them to become

a decision-making tool (Lorentz and Kidd, 1994).

However, the determination of suitable threshold val-

ues is arduous as detailed in the following studies:

Nagappan et al. (Nagappan et al., 2006) have shown

that thresholds obtained by performing a correlation

analysis are only valid for a limited set of similar soft-

ware systems; Zhang et al. (Zhang et al., 2013) have

claimed that thresholds cannot be generalized and de-

pend on speciﬁc domains (such as aerospace and stu-

dent exercises) and programming language character-

istics.

Organizations and computer scientists have given

many deﬁnitions of software quality over time: the

IEEE deﬁnes quality as the degree to which a system,

component, or process meets speciﬁed requirements

IEEE Standard Glossary of Software Engineering Termi-

nology. IEEE Std 610.12-1990.

or customer or user needs or expectations

. How-

ever, a good deﬁnition must lead us to measure qual-

ity meaningfully. According to Fenton and Bieman

(Fenton and Bieman, 2014), measurement is the pro-

cess by which numbers or symbols are assigned to at-

tributes of entities in the real world in such a way as

to describe them accordingto clearly deﬁned rules. In

our study, we found many papers that talk about the

topic of thresholds, but none of them is able to pro-

vide objective rules to use them effectively: they usu-

ally use threshold directly without explaining clearly

the techniques used to calculate them nor the context

in which they are derived. Therefore, this paper aims

at identifying the inﬂuential software metrics thresh-

olds papers with the purpose of leading developers

and scientists to improve their knowledge in this ﬁeld.

Our methodology leverages the evidence-based soft-

ware engineering (Kitchenham et al., 2004), which

(Kitchenham and Charters, 2007): relies on empiri-

cal software engineering research; suggests collecting

best available evidence on speciﬁc topic; uses sec-

ondary studies such as systematic literature reviews

and mapping studies. This paper introduces a pre-

liminary mapping study of software metrics thresh-

olds research. It intends to identify and categorize in-

ﬂuential software metrics thresholds research. Thus,

it addresses two research questions: 1. What pa-

pers are currently most important in the software met-

IEEE Standard Glossary of Software Engineering Termi-

nology. IEEE Std 610.12-1990.

232

Ronchieri, E. and Canaparo, M.

A Preliminary Mapping Study of Software Metrics Thresholds.

DOI: 10.5220/0005988402320240

In Proceedings of the 11th International Joint Conference on Software Technologies (ICSOFT 2016) - Volume 1: ICSOFT-EA, pages 232-240

ISBN: 978-989-758-194-6

Table 1: Search queries.

Goal Search String

papers 2015 ( TITLE-ABS-KEY ( software ) AND TITLE-ABS-KEY ( metrics ) AND TITLE-ABS-KEY ( thresholds ) )

AND SUBJAREA ( mult OR ceng OR CHEM OR comp OR eart OR ener OR engi OR envi OR mate OR math

OR phys ) AND PUBYEAR = 2015

papers 2015 ( TITLE-ABS-KEY ( metrics ) AND TITLE-ABS-KEY ( for ) ANDTITLE-ABS-KEY ( object-oriented )) AND

SUBJAREA ( mult OR ceng OR CHEM OR comp OR eart OR ener OR engi OR envi OR mate OR math OR

phys ) AND PUBYEAR = 2015

papers 1970-2014 ( TITLE-ABS-KEY ( software ) AND TITLE-ABS-KEY ( metrics ) AND TITLE-ABS-KEY ( thresholds ) )

AND SUBJAREA ( mult OR ceng OR CHEM OR comp OR eart OR ener OR engi OR envi OR mate OR math

OR phys ) AND PUBYEAR > 1969 AND PUBYEAR < 2015

papers 1970-2014 ( TITLE-ABS-KEY ( metrics ) AND TITLE-ABS-KEY ( for ) AND TITLE-ABS-KEY ( object-oriented ) AND

TITLE-ABS-KEY ( software ) ) AND SUBJAREA ( mult OR ceng OR CHEM OR comp OR eart OR ener OR

engi OR envi OR mate OR math OR phys ) AND PUBYEAR > 1969 AND PUBYEAR < 2015

rics thresholds research community? 2. Can soft-

ware metrics thresholds papers be meaningfully ag-

gregated? We refer to this as a preliminary study, be-

cause we make no claim for completeness. We have

focused on the years 1970-2015: to the best of our

knowledge, there are no surveys in the thresholds’

ﬁeld. We have used the SCOPUS tool

to search

for relevant peer-reviewed papers and the AlmaDL

service

of the University of Bologna to download

their complete text. So far we have decided to leave

other tools (such as ACM, IEEE and CiteSeer digital

libraries) for further studies.

2 BACKGROUND

Software metrics thresholds represent a way to deter-

mine the quality of code through a quantitative cri-

terion. Metrics can measure some features of the

code such as the size of the ﬁnal program, the com-

plexity of the software design, the modularity of a

class and the quality characteristics of the software.

Size metrics quantify code size. They are estima-

tors of software cost and effort. Complexity met-

rics measure the relative simplicity of the system

design. Object-oriented metrics (Chidamber and

Kamerer, 1994) measure complexity, cohesion, cou-

pling and inheritance (Brito e Abreu and Carapuca,

1994). Quality metrics also compute the length of

time between occurrences of defects (mean time be-

tween failures) or defects density (e.g., defects per

size). Let us start illustrating criteria for thresholds

identiﬁcation. We brieﬂy summarize the various ap-

proachesto determine metrics thresholds (Alves et al.,

2010). Personal experience of software quality

experts is very common. Many authors (McCabe,

Scopus, https://www.elsevier.com/solutions/scopus

Libraries, digital resources and study rooms,

http://www.unibo.it/en/services-and-opportunities/

libraries-digital-resources-and-study-rooms/

1976) (Nejmeh, 1988) have deﬁned thresholds ac-

cording to their know-how. This complicates the re-

production or generalization of their results and leads

to arguments about their values. Descriptive statis-

tics, such as average and standard deviation, provide

reasonable thresholds when the data distribution is

Gaussian. Other techniques, such as error informa-

tion and cluster analysis, are recently exploited by re-

searchers.

3 RESEARCH METHODS

For the preliminary mapping study we deﬁned all the

steps that compose the methodology we decided to

adopt.

Let us start detailing how we have identiﬁed rel-

evant papers. We used SCOPUS to search for soft-

ware metrics thresholds papers published from 1970

up to 2015. SCOPUS is a general indexing system

that includes publishers, such as IEEE, ACM, Else-

vier, Wiley and Springer Lecture Notes publications.

The search process has been split in two phases ended

in January 2016: the ﬁrst one considered studies up to

2014; the second one only regarded year 2015. The

papers were separated in two categories because we

included number of citations only amongst the ﬁlter-

ing criteria of the ﬁrst group. Table 1 shows the search

queries whose identiﬁer is Search

with Id = 1, ..., 4.

Due to the high number of papers found by the dif-

ferent searches, we decided to ﬁlter them according

to various criteria, as explained in the following para-

graphs.

Papers 2015 - Search

found 24 articles many

of which were irrelevant, for example papers that

reported in the title the following words: biodiver-

sity, watermaking, MOSFET, dielettric, FFT, routing,

spectrum, dosimetry, reverberation, MPSoCs, MIC

and analog. We removed all the papers with the

words above in the title and one paper with no au-

A Preliminary Mapping Study of Software Metrics Thresholds

233

Table 2: Main topic of thresholds and general type of paper.

Main Topic of thresholds

Category Meaning

Development The paper is about a speciﬁcation of a new tech-

nique for calculating thresholds.

Assessment The paper is about the assessment of existing

thresholds or techniques.

Analysis The papers discuss and or illustrate methods for

analyzing software metrics thresholds.

Framework The paper is about general or automated process

by which thresholds are deﬁned, extracted and

analyzed.

Literature survey The paper summarizes the literature on some as-

pect of thresholds.

Application The paper is only an application of existing or

calculated values of thresholds.

General type of paper

Category Meaning

Empirical The paper assesses existing thresholds or a tech-

nique for calculating them.

Theoretical The paper discusses some issues about software

engineering and may consider some theoretical

aspects of software metrics thresholds.

Both The paper is a mixed theoretical and empirical

paper.

thors: as a consequence, there were 10 records left

with no citations that we call Search

. Search

found

77 articles many of which were relevant. We removed

one paper with the mobile word and 3 papers with no

authors: as a consequence, there were 73 records left

(only one contains a citation) that we call Search

After the ﬁltering operations, we found that

and Search

have two relevant papers in

common. We identify this set with Search

and we

decided to exclude them from the set of Search

and

that we call Search

and Search

respec-

tively. A more detailed review of the abstract and

text in the papers belonging to Search

found 4 pa-

pers no relevant to the topic of this paper. Therefore,

there are 4 papers left named Search

. For what con-

cerns Search

, 44 papers are not relevant, therefore

27 records left named Search

. The total number

of relevant papers are 33 obtained adding together

+ Search

. Among them there

are 9 papers requested to their authors.

Papers 1970-2014 - Search

found 213 articles

that becomes 128 after having removed those with no

citations. Many left papers were irrelevant, for exam-

ple those reported in the title the following words: for-

est, vehicle, voting, landscape, healths, social, UMLS,

neuronal, organs, PAM, genes, clinical, cloud anoma-

lies, breast, LC-MS/MS, proteomic, ECG, gyrokinetic,

tensor, mammography, satellite, macro-invertebrates,

water-making, hue, car hood, routing, network, cir-

cuit, cellular, MOSFET, ionospheric, cortical, ur-

Table 3: Citations per publication type for year 1970-2014.

Set #Citations #Books #Journals #Proceedings Total

< 10 0 5 4 9

10-19 0 1 0 1

20-29 0 2 0 2

30-39 0 0 0 0

40-49 0 1 0 1

50+ 0 0 0 0

Total 0 9 4 13

Set #Citations #Books #Journals #Proceedings Total

< 10 0 15 16 31

10-19 0 2 3 5

20-29 0 2 1 3

30-39 0 0 0 0

40-49 0 0 1 1

50+ 0 1 1 2

Total 0 20 22 42

Set #Citations #Books #Journals #Proceedings Total

< 10 2 78 77 157

10-19 1 21 17 39

20-29 0 12 4 16

30-39 0 10 1 11

40-49 0 3 3 6

50+ 0 24 7 31

Total 3 148 109 260

Table 4: Topic per paper type.

Year 2015

Topic Empirical Theoretical Both Total

Development 0 0 1 1

Assessment 2 1 3 6

Analysis 0 0 1 1

Framework 0 0 0 0

Application 2 0 0 2

Literature

Survey

0 0 0 0

Total 4 1 5 10

Years 1970-2014

Topic Empirical Theoretical Both Total

Development 30 20 49 99

Assessment 20 13 3 36

Analysis 10 0 1 11

Framework 11 23 34 68

Application 2 6 0 8

Literature

Survey

1 1 0 2

Total 74 63 88 225

ban, multi antenna, wireless, MLSDA, diagnostically,

DIBL, agriculture, vestibular, radio, history sensitive,

laser optics, fog events, SIFT, WLAN, WAAS, ultra-

sound, trafﬁc, spatial vision, ﬂood attack, portfolio

and ﬁltering. We removed all the papers with the

words above in the title; as a consequence, there were

72 records left that we call Search

. Search

found

1403 articles that becomes 832 after having removed

those with no citations. We removed one paper with

ICSOFT-EA 2016 - 11th International Conference on Software Engineering and Applications

234

Figure 1: The number of publications per year: books, proceedings and journals.

Table 5: Source of conference and journal papers for year 2015.

Conferences #papers Journals #papers

ACM Symposium on Applied Computing 1 Expert Systems 1

ACM International Conference Proceeding Series 3 Advances in Computers 1

Ibero-American Conference on Software Engineering, CIbSE

2015

1 Procedia Computer Science 1

International Conference on Knowledge and Smart Technology,

KST 2015

1 Software and Systems Modeling 1

International Workshop on Software Architecture and Metrics,

SAM 2015

1 Journal of Systems and Software 4

IEEE International Conference on Software Testing, Veriﬁcation

and Validation Workshops, ICSTW 2015

1 Information and Software Technology 1

International Conference on Advances in Computing, Commu-

nications and Informatics, ICACCI 2015

1 Lecture Notes in Electrical Engineering 3

IEEE International Conference on Emerging Technologies and

Factory Automation, ETFA

1 Journal of Software: Evolution and Process 1

IEEE International Conference on Computational Intelligence

and Communication Technology, CICT 2015

1 Advances in Intelligent Systems and Computing 2

Frontiers in Artiﬁcial Intelligence and Applications 1

International Journal of Applied Engineering Research 2

International Journal of Software Engineering and its Applica-

tions

International Journal of Software Engineering and Knowledge

Engineering

Total 11 Total 22

the satellite word in the title; as a consequence, there

were 831 records left that we call Search

After the ﬁltering operations, we found that

and Search

have 15 papers in common,

two of which are not relevant being out of topic. We

identify this set with Search

and we decided to ex-

clude them from the set of Search

and Search

that

we call Search

and Search

respectively. A more

detailed review of the abstract and text in the papers

belonging to Search

found 14 papers no relevant to

the topic of this paper. In addition, we removed 1

paper that we were unable to ﬁnd. Therefore, there

are 42 papers left named Search

. For what con-

cerns Search

, 556 papers are not relevant, therefore

260 records left named Search

. The total number

of relevant papers are 314 obtained adding together

+ Search

. Among them there

are 60 papers requested to their authors.

Let us explain how we have extracted data. Start-

ing from the relevant papers, we collected some stan-

dard information about all papers, such as the authors,

the full reference, whether the paper was related to a

conference or a journal, the total number of citations.

So far we have not observed for the selected papers

changes in the number of citations. We aim to clas-

sify all the papers according to the following criteria:

A Preliminary Mapping Study of Software Metrics Thresholds

235

Table 6: Source of conference papers for years 1970-2014.

Conferences #papers Conferences #papers

International Conference and Exhibition on Technology of Object-Oriented Languages and Systems, TOOLS 39 1 International Symposium on Empirical Software Engineering and Measurement, ESEM 2009 1

Central and Eastern European Software Engineering Conference in Russia, CEE-SECR 2009 1 International Conference on Signal Processing Systems, ICSPS 2009 1

World Congress on Software Engineering, WCSE 2009 1 International Conference on Research Challenges in Information Science, RCIS 2010 1

Malaysian Conference in Software Engineering, MySEC 2011 1 International Workshop on Emerging Trends in Software Metrics, WETSoM 2012 1

International Conference on Innovations in Information Technology, IIT 2012 1 International Conference on Computing, Management and Telecommunications, ComManTel 2013 1

IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering, CSMR-WCRE 2014 1 International Conference on Aspect-Oriented Software Development 1

ACS/IEEE International Conference on Computer Systems and Applications, 2005 1 Annual Conference of the North American Fuzzy Information Processing Society, NAFIPS 1

Canadian Conference on Electrical and Computer Engineering 1 Conference on Software Maintenance 4

Genetic and Evolutionary Computation Conference, GECCO 2006 1 International Conference on Software and Data Technologies, ICSOFT 2010 1

IEEE Aerospace Applications Conference Proceedings 1 IEEE International Conference on Automation Science and Engineering 1

IEEE International Conference on Neural Networks 1 IEEE International Conference on Program Comprehension 5

IEEE International Conference on Software Maintenance, ICSM 9 IEEE International Working Conference on Mining Software Repositories 2

IEEE/ACM International Conference on Computer-Aided Design, Digest of Technical Papers, ICCAD 1 International Software Metrics Symposium 4

International Symposium on Software Testing and Analysis, ISSTA’08 1 International Conference on Information and Communications Technology ICICT 2005 1

International Conference on Information Technology: New Generations, ITNG 2009 1 International Joint Conference on Computer Science and Software Engineering, JCSSE 2012 1

Conference on Power Electronics and Intelligent Transportation System, PEITS 2009 1 IEEE International Working Conference on Source Code Analysis and Manipulation, SCAM 2010 1

International Conference on Automated Software Engineering, ASE 2004 1 International Symposium on Search Based Software Engineering, SSBSE 2009 1

International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2010 1 IEEE International Conference on Granular Computing, GrC 2010 1

IEEE Software Engineering Workshop, SEW 2011 1 IEEE International Conference on Control System, Computing and Engineering, ICCSCE 2013 1

Brazilian Symposium on Software Engineering, SBES 2010 1 International Conference on Software Maintenance and Evolution, ICSME 2014 1

IEEE International Workshop on Source Code Analysis and Manipulation, SCAM 2003 1 IEEE/ACIS International Conference on Computer and Information Science, ICIS 2007; IEEE/ACIS Interna-

tional Workshop on e-Activity, IWEA 2007

European Dependable Computing Conference, EDCC-7 1 International Conference on the Quality of Information and Communications Technology, QUATIC 2010 1

IEEE International Working Conference on Source Code Analysis and Manipulation, SCAM 2008 1 ACM Computer Science Conference 1

Asia-Paciﬁc Software Engineering Conference, APSEC 3 Workshop on Interaction between Compilers and Computer Architectures, INTERACT-8 2004 1

International Conference on Quality Software, QSIC 2004 2 IEEE Computer Society’s International Computer Software and Applications Conference 1

International Computer Software and Applications Conference 1 International Conference on Advanced Information Networking and Applications, AINA 1

International Conference on Computer Science and Software Engineering, CSSE 2008 1 International Conference on Dependability of Computer Systems, DepCoS - RELCOMEX 2007 2

International Conference on Quality Software 1 International Conference on Software Engineering 4

International Conference on Tools with Artiﬁcial Intelligence, ICTAI 1 International Software Metrics Symposium 4

International Symposium on Software Reliability Engineering, ISSRE 2 Joint Conference of the 21st International Workshop on Software Measurement, IWSM 2011 and the 6th Inter-

national Conference on Software Process and Product Measurement, MENSURA 2011

International Conference onInformation Technology: New Generations, ITNG 2006 1 Working Conference on Reverse Engineering, WCRE 3

International Conference on Reliability, Maintainability and Safety, ICRMS 2009 1 IEEE International Symposium on High Assurance Systems Engineering 1

International Society for Optical Engineering, SPIE 2 IEEE International Requirements Engineering Conference, RE’08 1

International Conference on Software Testing, Veriﬁcation and Validation, ICST 2008 1 International Conference on Computational Aspects of Social Networks, CASoN 2012 1

Joint Working Conference on Software Architecture and 6th European Conference on Software Architecture,

WICSA/ECSA 2012

1 International Florida Artiﬁcial Intelligence Research Society, FLAIRS - 24 1

International Conference on Conﬂuence 2014: The Next Generation Information Technology Summit 1 International Conference on Generative Programming and Component Engineering, GPCE’06 1

International Conference on Applied Computer Science, ACS ’09 1 ACM Symposium on Applied Computing 2

Australian Software Engineering Conference, ASWEC 2 Conference on Object-Oriented Programming Systems, Languages, and Applications, OOPSLA 2

European Conference on Software Maintenance and Reengineering, CSMR 9 Hawaii International Conference on System Science 1

IASTED International Conference on Computational Intelligence 1 IASTED International Conference on Software Engineering 1

International Conference on Computer and Communication Engineering, ICCCE08 1 International Conference on Software Engineering Research and Practice, SERP’04 3

International Symposium on Software Reliability Engineering, ISSRE 1 International Conference on Software Engineering and Data Engineering, SEDE 2011 1

IEEE International Workshop on Visualizing Software for Understanding and Analysis, VISSOFT 2009 1 Working Conference on Reverse Engineering 1

International Conference on Software Engineering and Knowledge Engineering, SEKE 2011 2 Joint WOSP/SIPEW International Conference on Performance Engineering, WOSP/SIPEW’10 1

Total 67 Total 68

ICSOFT-EA 2016 - 11th International Conference on Software Engineering and Applications

236

Table 7: Source of journal papers for years 1970-2014.

Journals #papers Journals #papers

ACM Computing Surveys 1 ACM SIGPLAN Notices 1

Computer Journal 1 IBM Systems Journal 1

IET Software 4 Informatica (Ljubljana) 1

Information Sciences 2 Journal of Software 2

SpringerPlus 1 Neurocomputing 1

Pattern Recognition 1 Procedia Computer Science 1

Software Quality Journal 3 The Scientiﬁc World Journal 1

Ruan Jian Xue Bao/Journal of Software 3 Software - Practice and Experience 3

Software Process Improvement and Practice 1 Journal of Systems Architecture 1

Tien Tzu Hsueh Pao/Acta Electronica Sinica 1 Software Testing Veriﬁcation and Reliability 1

Wuhan University Journal of Natural Sciences 1 Information Processing Letters 2

Empirical Software Engineering 9 Expert Systems With Applications 2

Applied Artiﬁcial Intelligence 1 IEEE Transactions on Reliability 1

Journal of Electronic Imaging 1 Journal of Object Technology 5

Journal of Software Maintenance 1 Conference on Software Maintenance 4

Journal of Systems and Software 18 Lecture Notes in Computer Science 13

Information and Software Technology 18 Lecture Notes in Electrical Engineering 1

European Journal of Operational Research 1 Advances in Intelligent Systems and Computing 1

ACM International Conference Proceeding Series 1 IEEE Transactions on Software Engineering 19

IEICE Transactions on Information and Systems 4 Innovations in Systems and Software Engineering 2

International Review on Computers and Software 3 JOOP - Journal of Object-Oriented Programming 2

Journal of Zhejiang University: Science C 1 Journal of software: Evolution and Process 1

Journal of Computer Science and Technology 1 Journal of Information Processing Systems 1

Journal of Object-Oriented Programming 1 Journal of Software Maintenance and Evolution 6

Journal of Integrated Design and Process Science 1 Lecture Notes in Business Information Processing 1

Monthly Notices of the Royal Astronomical Society 1 Communications in Computer and Information Science 2

IEEE Transactions on Parallel and Distributed Systems 1 International Journal of Computers and Applications 1

World Academy of Science, Engineering and Technology 1 International Journal of Machine Learning and Cybernetics 1

ACM Transactions on Software Engineering and Method-

ology

3 International Journal on Artiﬁcial Intelligence Tools 1

International Journal of Computer Applications in Technol-

ogy

3 Journal of Computational Methods in Sciences and Engi-

neering

WSEAS Transactions on Information Science and Applica-

tions

1 International Journal of Digital Content Technology and its

Applications

International Journal of Software Engineering and Knowl-

edge Engineering

Total 97 Total 81

1. the main topic (see Table 2); 2. the type of pa-

per (i.e., empirical, theoretical or both (see Table 2);

3. the type of publication (i.e., proceedings, journal,

book); 4. the software licence (i.e., open source or

commercial software) of the analysed projects; 5. the

considered dataset of metrics (i.e., name, public or

private); 6. the programming languages of the anal-

ysed projects; 7. the type of metrics; 8. the type of

presented technique (e.g., statistical or artiﬁcial intel-

ligence based). Let us detail why we have aggregated

data. The main hindrance to the adoption of thresh-

olds is the lack of guidelines for their exploration and

exploitation. As a consequence, one of the purposes

of this paper is to deﬁne several classiﬁcation crite-

ria to acquire valid aggregations that can facilitate the

way scientists and developers search information re-

lated to thresholds. Since they have been always cal-

culated in speciﬁc applicability domain (characterized

by, e.g., a particular software license or programming

language), we established to track all the information

about the environment where thresholds were deter-

mined and used.

4 RESULTS

In this section we present some tabulations of the re-

sults of categorizing the identiﬁed papers. They con-

cern the following information: number of publica-

tions per year (see Figure 1); number of citations per

publication type for year 1970-2014 (see Table 3);

topic per paper type for year 2015 (see Table 4); topic

per paper type for years 1970-2014 (see Table 4);

source of conference papers for year 2015 (see Table

5); source of journal papers for year 2015 (see Table

5); source of conference papers for years 1970-2014

(see Table 6); source of journal papers for years 1970-

2014 (see Table 7); details of common (Search

and

A Preliminary Mapping Study of Software Metrics Thresholds

237

Table 8: Common pubblications in detail.

Reference Main Topic Empirical/Theoretical Number of SW

projects

Open Source

SW project

Public Data Set Programming

Language

Type of Metrics Statistical/Artiﬁcial

Intelli-

gence/other

Threshold

Values

(Alves et al., 2010) Development Empirical 100 projects Proprietary and

open source

Not public Java, C# Size, Complexity Statistical McCabe <= 6

low risk, ]6,8]

moderate risk,

]8,15] high risk,

> 15 very high

risk

(Al Dallal, 2012) Assessment Empirical and Theo-

retical

6 projects: Art

of illusion, Free-

Mind, etc

Open source Not public Java Size, coupling

and cohesion

Statistical

(Aman et al., 2005) Development Empirical 5 projects: Jboss,

Relaxer, etc

Open source Not used Java Statements Statistical Statements = 113

(class)

(Ampatzoglou et al., 2011) Application Empirical and Theo-

retical

Not used Not used Not used Not used Size, Complexity,

C& K

Not used Table 5 reports

data for 5 imagi-

nary case studies

(Barkmann et al., 2009) Assessment Empirical 146 projects Open source Not public Java Size, C& K Statistical

(El Emam et al., 2002) Application Empirical and Theo-

retical

3 systems Industrial com-

mercial and open

source

Not public C++, Java Size and some

C& K

Statistical

(Ferreira et al., 2012) Development Empirical 40 projects from

sourceforge

Open source Not public Java Size, C&K Statistical Different thresh-

olds calculated by

different applica-

tion domain, soft-

ware types, soft-

ware size.

(Mihancea and Marinescu,

2005)

Development Empirical 7 projects Open source Not public Java, C++ Size and metrics

derived from

C&K

Genetic Algo-

rithm

WMC > 31, TCC

< 0.33, WOC <

0.95

(Rodriguez et al., 2013) Development Empirical Not used Not used Yes (Promise

Repo)

C, C++, Java Size, Complexity,

C&K

Artiﬁcial intelli-

gence

Different thresh-

olds for individ-

ual datasets

(Shatnawi, 2010) Development Empirical and Theo-

retical

Mozilla and

Rhino systems,

Eclipse version

2.0 and 2.1

Yes Not used Java A subset of C&K

metric suites

Statistical CBO = 9, RFC =

40, WMC = 20

(Shatnawi et al., 2009) Development Empirical 3 release of

Eclipse

Open source Not public Java C&K, LK Statistical, ROC

analysis

CBO = 13, RFC

= 44, WMC = 24,

CTM = 33, NOC

= 9 to identify

low, medium and

high risk

(Sodiya, 2012) Application Empirical and Theo-

retical

Not used Not used Not used No McCabe Cyclo-

matic Complex-

ity, Size

Not used Used thresholds

listed by Olague

et al., 2006 and

Rosenberg, 1998

(Malhotra and Bansal, 2015) Evaluation Empirical NASA software,

Apace Ivy, JEdit

All open source

except NASA

software

Open DataSet,

KC1

Java, C++ C&K metrics Statistical Different thresh-

olds for individ-

ual datasets

) publications 1970-2015 (see Table 8). As

regards this table we list here the meaning of the met-

rics’ acronyms used in the ”threshold values” column:

WMC (Weighted Method Count), TCC (Tight Class

Cohesion), WOC (Weight of a Class), CBO (Class

Between Objects), RFC (Response for Class), CTM

ICSOFT-EA 2016 - 11th International Conference on Software Engineering and Applications

238

(Coupling Through Message passing), NOC (Number

of Child classes), C&K (Chidamber and Kemerer),

LK (Lorentz and Kidd).

For the years 1970-2014 we identiﬁed 3 books

whose titles are: Model-Driven Software Devel-

opment: Integrating Quality Assurance; Object-

Oriented Design Knowledge: Principles, Heuristics

and Best Practices; Theory and Practice of Object

Systems. We have hitherto received 4 papers out of

the 69 requested to the authors.

5 CONCLUSIONS

We believe this study is useful, in spite of its limita-

tions, because it may act as the starting point for more

detailed work. This paper: identiﬁes the most inﬂuen-

tial papers, in terms of citations, published from 1970

to 2014; collects some standard information about all

the papers in the range 1970-2015 in terms of whether

the paper was related to a conference or journal, the

total number of citations; establishes some criteria ac-

cording to which papers can be classiﬁed (such as

main topic, whether the paper is theoretical or empir-

ical, the programming language of the software em-

ployed for thresholds extraction and validation, the

type of software license, the type of metrics used, the

type of the presented technique and some threshold

values); starts a meaningful aggregation of all the ma-

terial based on the established criteria, in order to fa-

cilitate the selection of papers;

We may extend this work by taking into consid-

eration other criteria to provide further classiﬁcation

results. Furthermore, we could use other tools (such

as ACM, IEEE and CiteSeer digital libraries) in or-

der to have a comparison with the results obtained by

SCOPUS.

To fully analyze the current status in the ﬁeld

of thresholds, we will ﬁrstly undertake the mapping

study (Kitchenham & Charters, 2007) and secondly

the systematic review (Cronin et al, 2008). The for-

mer allows identifying the set of primary works high-

lighting their gaps according to the established ques-

tion. The latter provides a list as complete as possible

of all the published and unpublished studies relating

to a particular subject area.

ACKNOWLEDGEMENTS

This study was funded by INFN CNAF.

REFERENCES

Al Dallal, J. (2012). Constructing models for predicting ex-

tract subclass refactoring opportunities using object-

oriented quality metrics. Information and Software

Technology, 54:1125–1141.

Alves, T. L., Ypma, C., and Visser, J. (2010). Deriving met-

rics thresholds from benchmark data. In The IEEE

International Conference on Software Maintenance,

pages 1–10.

Aman, H., Mochiduki, N., Yamada, H., and Noda, M.-T.

(2005). A simple predictive method for discriminating

costly classes using class size metric. IEICE Transac-

tion on Information and Systems, E88-D(6).

Ampatzoglou, A., Frantzeskou, G., and Stamelos, I. (2011).

A methodology to assess the impact of design patterns

on software quality. Information and Software Tech-

nology, 54:331–346.

Barkmann, H., Lincke, R., and Love, W. (2009). Quanti-

tative evaluation of software quality metrics in open-

source projects. In International Conference on

Advanced Information Networking and Applications

Workshops, pages 1067 – 1072. IEEE.

Brito e Abreu, F. and Carapuca, R. (1994). Object-oriented

software engineering: Measuring and controlling the

development process. In 4th Int. Conf. on Software

Quality QSIC.

Chidamber, S. R. and Kamerer, C. F. (1994). A metrics

suite for object-oriented design. IEEE Trans. Software

Engineering, SE-20(6):476–493.

El Emam, K., Benlarbi, S., Goel, N., Melo, W., Lounis, H.,

and Rai, S. (2002). The optimal class size for object-

oriented software. IEEE Transactions on Software En-

gineering, 28.

Fenton, N. and Bieman, J. (2014). Software Metrics: A Rig-

orous and Practical Approach, Third Edition. CRC

Press.

Fenton, N. E. and Neil, M. (2000). Software metrics:

roadmap. ICSE - Future of SE Track, pages 357–370.

Ferreira, K. A. M., Bigonha, M. A. S., Bigonha, R. S.,

Mendes, L. F. O., and Almeida, H. C. (2012). Identi-

fying thresholds for object-oriented software metrics.

Journal of Systems and Software, 85(2):244–257.

Kitchenham, B. A. and Charters, S. M. (2007). Guide-

lines for performing systematic literature reviews in

software engineering. EBSE Technical Report EBSE-

2007-01, Keele University and Durham University.

Kitchenham, B. A., Dyba, T., and Jorgensen, M. (2004).

Evidence-based software engineering. In The 26th

International Conference on Software Engineering

(ICSE’04).

Lorentz, M. and Kidd, J. (1994). Object-oriented software

metrics:a practical guide. Prentice-Hall, Inc.

Malhotra, R. and Bansal, A. J. (2015). Fault prediction con-

sidering threshold effects of object-oriented metrics.

Expert Systems, 32(2):203–219.

McCabe, T. J. (1976). A complexity measure. IEEE Trans-

actions on Software Engineering, SE-2(4):308–320.

Mihancea, P. and Marinescu, R. (2005). Towards the op-

timization of automatic detection of design ﬂaws in

A Preliminary Mapping Study of Software Metrics Thresholds

239

object-oriented software systems. In Ninth European

Conference on Software Maintenance and Reengi-

neering, pages 92 – 101. IEEE.

Nagappan, N., Ball, T., and Zeller, A. (2006). Mining met-

rics to predict component failures. In The 28th in-

ternational conference on Software engineering (ICSE

’06), pages 452–461, New York, NY, USA,. ACM.

Nejmeh, B. A. (1988). Npath: A measure of execution path

complexity and its applications. Communications of

the ACM, 31(2):188–200.

Rodriguez, D., Ruiz, R., Riquelme, J., and Harrison, R.

(2013). A study of subgroup discovery approach for

defect prediction. Information and Software Technol-

ogy, 55:18101822.

Shatnawi, R. (2010). A quantitative investigation of the ac-

ceptable risk levels of object-oriented metrics in open-

source systems. IEEE Transactions on Software En-

gineering, 36:216–225.

Shatnawi, R., Li, W., Swain, J., and Newman, T. (2009).

Finding software metrics threshold values using roc

curves. Journal of Software Maintenance and Evolu-

tion: Research and Practice, 22(1):1–16.

Sodiya, A. S. (2012). A survability model for object-

oriented software systems. In Fourth International

Conference on Computational Aspects of Social Net-

works (CASoN), pages 283 – 290. IEEE.

Zhang, F., Mockus, A., Zou, Y., Khomh, F., and Hassan,

A. E. (2013). How does context affect the distribution

of software maintainability metrics? In The 29th IEEE

International Conference on Software Maintainability

(ICSM’13), pages 350–359, Eindhoven. IEEE.

ICSOFT-EA 2016 - 11th International Conference on Software Engineering and Applications

240