AN OVERVIEW OF SOFTWARE PROCESS QUALITY

INSTRUMENTS ADOPTION AT BRAZIL

Preliminary Results of a Survey

Rodrigo Santos de Espindola

, Edimara Mezzomo Luciano

Faculty of Informatics,

Faculty of Administration, Accountancy and Economy, PUCRS

Av. Ipiranga 6681, Porto Alegre – RS, Brazil

Jorge Luis Nicolas Audy

Faculty of Informatics, PUCRS, Av. Ipiranga 6681, Porto Alegre – RS, Brazil

Keywords: Software Quality, Survey, Statistical Analysis.

Abstract: This paper presents results from a quantitative research that was conducted among 260 participants of a

software quality event in Brazil. This research aims the understanding the occurrence of several IT problems

in the analyzed organizations and the impact of the quality instruments adoption on those problems. The

main contribution is the identification of the respondent’s perception about the relationship of the quality

instruments adoption and their impact on several IT problems throughout statistical analysis of the data.

1 INTRODUCTION

This paper presents the preliminary results of a

quantitative research conducted in Brazil in order to

explore and understand the relationship among some

quality instruments, the organizations and the usual

Information Technology (IT) problems. It was

carried out statistical analisys in the collected data

aiming to understand the impact of quality

instruments adoption in the usual IT problems.

This paper is organized in five sections. This

introduction is the section 1 and it is followed by the

literature review in the section 2. The section 3

presents the research method. The section 4 presents

the data analysis and the section 5 presents our

conclusions.

2 LITERATURE

The ITIL has been pointed out by both academy and

industry as a good way to manage IT services

(ITSMF, 2003). The ISO 9000 standards were

created to deal with quality management (ISO

2000). The PNQ (a Portuguese acronym to National

Quality Prize) was developed and it is maintained by

the Brazilian National Quality Foundation (FNQ

2004). CMMI was created to help organizations on

their process area capability and maturity

assessments, and on establishing and implementing

the improvements prioritization (SEI, 2004). IT

Governance Institute has designed and created the

COBIT (Control Objectives for Information and

related Technology) (IT Governance Institute,

2005).

3 RESEARCH METHOD

This work has an exploratory nature, proper to be

applied when the goal is evaluate a theme or an

investigation problem not widely studied or that has

no previous study (SAMPIERI, COLLADO,

LUCIO, 1991 and YIN, 2001). The research strategy

adopted was a survey.

The data was collected by applying a form that

was filled out by the respondents whom have

attended a software and IT services quality event in

2006. 240 out of 260 participants had answered the

questions. 34 out of 240 forms were not considered

as they had a lot of non answered questions.

347

Santos de Espindola R., Mezzomo Luciano E. and Luis Nicolas Audy J. (2008).

AN OVERVIEW OF SOFTWARE PROCESS QUALITY INSTRUMENTS ADOPTION AT BRAZIL - Preliminary Results of a Survey.

In Proceedings of the Tenth International Conference on Enterprise Information Systems - ISAS, pages 347-350

DOI: 10.5220/0001714403470350

 SciTePress

4 DATA ANALYSIS

The first question analyzed is about the IT problems

faced by the organizations. A list of 13 usual IT

problems was proposed and the respondents were

asked to choose one option in the following scale:

Totally disagree, disagree, agree and totally agree.

The Table 1 presents the relative frequency of

answers about the IT problems faced by the

organizations. In this table the IT problems were

numbered from Q1.1 to Q1.13. This same

numbering schema will be used in the

Table 3, but

the statement of the IT problem won’t be repeated.

The second question aims to measure

relationship between the organizations and a list of

quality instruments. The respondents were asked to

indicate the current step in adoption of each quality

instruments.

The Table 2 presents the frequencies and the

quantity of answers by each quality instrument.

Table 2: Quality instrument adoption.

Quality

Won't be

implemented

Plannin

Implementin

instrument (%) (%) (%) Answers

PNQ 61.4 25.1 13.5 171

ISO9001 36.6 19.3 44.1 161

COBIT 47.6 37.3 15.1 166

eSCM 70.3 27.7 1.9 155

People

CMM

52.1 35.6 12.3 163

ITIL 23.2 48.2 28.6 168

CMMI 6.5 32.4 61.2 170

OPM3 55.7 38.6 5.7 158

Six Sigma 41.5 39.6 18.9 164

These results show that the CMMI is the most

adopted quality instrument in these organizations,

followed by ISO9001 and ITIL. It also shows that

the majority of the organizations have no plan to

adopt eSCM, PNQ or OPM3.

The analysis of IT problems faced by the

organization and the degree of adoption of quality

instruments leads to the analysis of the association

between these two variables. The hypothesis is the

possibility of a negative association between quality

instruments adoption and the IT problems, i.e. the

adoption of some quality instrument reducing the

frequency of some IT problem. Therefore, and due

to the nominal nature of the variables, we tested the

hypothesis using a chi-squared statistical test.

A chi-squared test is a statistical tool to verify

the association between two nominal variables. It

can be used to calc two values. The first one, the p-

value, provides the statistical significance of the test

and allows us rejecting the null hypothesis of no

association between the analyzed variables. The

second one, the contingency coefficient C, is a

number between 0 and 1 that express the degree of

association between the two analyzed variables.

The

Table 3 presents several statistically

significant associations. Since we are looking for

negative associations, then we need analyze the

contingency tables to verify if the adoption of each

quality instrument is increasing or decreasing the

frequency of the IT problems. Only five associations

are analyzed in this paper due to their highest

contingency coefficient C and statistical

significance.

The

Table 4 presents the contingency table

between PNQ adoption and the question 1.13. This

Table 1: Relative frequency of IT problems and quantity of answers.

Question

Totally

Disagree

(%)

Disagree

(%)

Agree

(%)

Totally

Agree

(%)

Answers

Q1.1 - Lack of formal decision making process 17.3 43.1 31.0 8.6 197

Q1.2 - Lack of clear alignment among IT actions and strategic goals of the

organization

17.0 42.0 31.5 9.5 200

Q1.3 - Lack of quantitative knowledge about performance 6.4 29.7 40.6 23.3 202

Q1.4 - Quality and performance of suppliers below the expectations 5.4 30.1 52.2 12.4 186

Q1.5 - Frequent IT project delays 6.0 25.9 47.8 20.4 201

Q1.6 - Poor quality of our IT products 11.4 51.7 28.4 8.5 201

Q1.7 - High cost of IT area 2.1 36.3 51.6 10.0 190

Q1.8 - IT area isn't treated as service provider internally 16.2 39.1 36.0 8.6 197

Q1.9 - Low availability of IT systems and applications 18.2 57.6 21.2 3.0 198

Q1.10 - Lack of customer focus 15.3 47.0 29.7 7.9 202

Q1.11 - High turnover of IT professionals 10.1 42.4 32.8 14.6 198

Q1.12 – Not motivated teams and professionals or not aligned with strategic goals 9.4 42.9 36.5 11.3 203

Q1.13 - Lack of companies and professionals with integrated comprehension of

problems and solutions

4.6 31.0 53.8 10.7 197

ICEIS 2008 - International Conference on Enterprise Information Systems

348

association has p-value of 5.40x10

-4

, allowing us to

reject the null hypothesis of no association between

the variables. However, we can see that neither the

frequency of respondents that agree with this IT

problem change significantly in the group of those

are implementing PNQ nor the frequency of

respondents that disagree change significantly in the

group of those are implementing PNQ. Then, we

cannot affirm that there is an association between

the adoption of PNQ and the decrease of this IT

problem.

Table 4: Association between PNQ adoption and the lack

of companies and professionals with integrated

comprehension of problems and solutions (Q1.13).

PNQ Impl. Planning

Won't

implement

Total

Q1.13 (%) (%) (%) (%) Answers

Agree 63.6 40.5 74.5 64.5 107

Disagree 36.4 59.5 25.5 35.5 59

100.0 100.0 100.0 100.0 166

Table 5: Association between ISO 9001 adoption and the

lack of quantitative knowledge about performance (Q1.3).

ISO9001 Impl. Planning

Won't

implement Total

Q1.3 (%) (%) (%) (%) Answers

Agree 57.1 56.7 83.1 66.7 106

Disagree 42.9 43.3 16.9 33.3 53

100.0 100.0 100.0 100.0 159

The

Table 5 presents the contingency table

between ISO 9001 adoption and the question 1.3.

This association has p-value of 3.46x10

-3

, allowing

us to reject the null hypothesis of no association

between the variables.

We can see in the

Table 5 a negative association

between the adoption of ISO 9001 and the lack of

quantitative knowledge about performance. The

frequency of respondents that agree with this IT

problem decreases from 66.7% to 57.1% in the

group of those are implementing ISO 9001. On the

other hand, the frequency of respondents that

disagree with this IT problem rises from 33.3% to

42.9% in the group of those that are implementing

ISO 9001. Then, we can affirm that, at least in the

analyzed sample, the adoption of ISO 9001 tends to

decrease the lack of quantitative knowledge about

performance.

The

Table 6 presents the contingency table

between ITIL adoption and the question 1.2. This

association has p-value of 4.42x10

-3

, allowing us to

reject the null hypothesis of no association between

the variables.

Table 6: Association between ITIL adoption and the lack

of clear alignment among IT actions and strategic goals of

the organization (Q1.2).

ITIL Impl. Planning

Won't

implement

Total

Q1.2 (%) (%) (%) (%) Answers

Agree 21.3 50.6 44.4 40.9 67

Disagree 78.7 49.4 55.6 59.1 97

100.0 100.0 100.0 100.0 164

In this case, there is also a negative association

between ITIL adoption and the lack of clear

alignment among IT actions and strategic goals of

the organization (Q1.2). The frequency of

Table 3: Contingency coefficient C among quality instruments and IT problems.

problems

PNQ ISO9001 COBIT eSCM

People

CMM

ITIL CMMI OPM3

Six

Sigma

Q1.1 0.037 0.119 0.067 0.163 0.092 0.247** 0.056 0.132 0.099

Q1.2 0.104 0.156 0.112 0.151 0.030 0.249** 0.102 0.089 0.116

Q1.3 0.186* 0.258** 0.158 0.173 0.166 0.097 0.035 0.094 0.204*

Q1.4 0.033 0.075 0.120 0.107 0.141 0.212* 0.090 0.077 0.070

Q1.5 0.076 0.108 0.085 0.142 0.246** 0.109 0.150 0.107 0.047

Q1.6 0.202* 0.155 0.071 0.094 0.097 0.083 0.171 0.102 0.156

Q1.7 0.209* 0.219* 0.065 0.122 0.167 0.048 0.046 0.121 0.193

Q1.8 0.215* 0.043 0.069 0.128 0.062 0.087 0.080 0.067 0.196*

Q1.9 0.033 0.131 0.124 0.113 0.112 0.180 0.047 0.139 0.127

Q1.10 0.190* 0.061 0.051 0.101 0.078 0.111 0.127 0.050 0.115

Q1.11 0.108 0.153 0.074 0.180 0.104 0.079 0.113 0.172 0.196*

Q1.12 0.080 0.147 0.132 0.054 0.151 0.071 0.022 0.080 0.118

Q1.13 0.288*** 0.207* 0.038 0.031 0.131 0.067 0.142 0.104 0.148

statistically significant at: * 5% level, ** 1% level and *** 0.1% level

AN OVERVIEW OF SOFTWARE PROCESS QUALITY INSTRUMENTS ADOPTION AT BRAZIL

349

New Text

AN OVERVIEW OF SOFTWARE PROCESS QUALITY INSTRUMENTS ADOPTION AT BRAZIL - Preliminary

Results of a Survey

respondents that agree with this IT problem

decreases from 40.9% to 21.3% in the group of those

are implementing ITIL. On the other hand, the

frequency of respondents that disagree with this IT

problem rises from 59.1% to 78.7% in the group of

those that are implementing ITIL. Then, we can

affirm that, at least in the analyzed sample, the

adoption of ITIL tends to decrease the lack of clear

alignment among IT actions and strategic goals of

the organization.

The

Table 7 presents another association about

ITIL. It shows the contingency table between ITIL

adoption and the question 1.1, which is about the

lack of formal decision making process. This

association has p-value of 4.79x10

-3

, allowing us to

reject the null hypothesis of no association between

the variables.

Table 7: Association between ITIL adoption and the lack

of formal decision making process (Q1.1).

ITIL Impl. Planning

Won't

implement

Total

Q1.1 (%) (%) (%) (%) Answers

Agree 25.5 44.3 60.5 42.7 70

Disagree 74.5 55.7 39.5 57.3 94

100.0 100.0 100.0 100.0 164

In this case, there is also a negative association

between ITIL adoption and the lack of formal

decision making process (Q1.1). The frequency of

respondents that agree with this IT problem

decreases from 42.7% to 25.5% in the group of those

are implementing ITIL. On the other hand, the

frequency of respondents that disagree with this IT

problem rises from 57.3% to 74.5% in the group of

those that are implementing ITIL. Then, we can

affirm that, at least in the analyzed sample, the

adoption of ITIL tends to decrease the lack of formal

decision making process.

The

Table 8 presents the contingency table

between People CMM adoption and the question

1.5. This association has p-value of 5.84x10

-3

allowing us to reject the null hypothesis of no

association between the variables.

Table 8: Association between People CMM adoption and

frequent IT project delays (Q1.5).

People

CMM

Impl. Planning

Won't

implement

Total

Q1.5 (%) (%) (%) (%) Answers

Agree 40.0 70.9 76.5 70.0 112

Disagree 60.0 29.1 23.5 30.0 48

100.0 100.0 100.0 100.0 160

In this case, there is also a negative association

between People CMM adoption and frequent IT

project delays (Q1.5). The frequency of respondents

that agree with this IT problem decreases from 70%

to 40% in the group of those are implementing

People CMM. On the other hand, the frequency of

respondents that disagree with this IT problem rises

from 30% to 60% in the group of those that are

implementing People CMM. Then, we can affirm

that, at least in the analyzed sample, the adoption of

People CMM tends to decrease the frequency of IT

project delays.

5 CONCLUSIONS

This paper presents results from a quantitative

research that was conducted among 260 participants

of a software quality event in Brazil. The main

contribution is the identification of the respondent’s

vision about the relationship of the quality

instruments adoption and the several IT problems.

The results obtained through this survey

provided an overview of the impact of the analyzed

IT problems in the organizations. Through the

appliance of statistical tests, we also obtained

understanding about the impact of the quality

instruments in some IT problems.

As future work, other dimensions of the survey

will be analyzed and a framework to quality

instruments integration will be developed.

REFERENCES

CMM - Capability Maturity Model. Guidelines for

Improving the Software Process. SEI Series, SEI

(Software Engineering Institute), 2004.

Fitzsimmons, J. A.; Fitzsimmons, M. J. Administração de

Serviços: Operações, Estratégia e Tecnologia da

Informação. 2. ed. Porto Alegre: Bookman, 2000

FNQ - National Quality Foundation. São Paulo, 2004.

ISO – International Organization for Standardization.

Switzerland, 2000.

IT Governance Institute. “CobiT 4.0”. EUA, Ilinois: IT

Governance Institute, 2005, 207pp.

ITSMF-INTERNATIONAL. IT Service Management: An

Introduction. Holland: Van Haren Publishing, 2002.

Sampieri, Roberto H.; Collado, Carlos F.; Lucio, Pilar B.

Metodología de la investigación. México: McGraw-

Hill, 1991.

Yin, R. K. Estudo de Caso - Planejamento e Métodos. 2.

ed. São Paulo: Bookman, 2001.

ICEIS 2008 - International Conference on Enterprise Information Systems

350