Validity of Critical Factors Scale for SMEs Readiness in

Implementing Quality Management System

Darmawan Napitupulu

1,3

, Krisna Adiyarta

and Zulvia Khalid

Fakultas Teknologi Informasi, Universitas Budi Luhur, Petukangan, Jakarta, Indonesia

Fakultas Ekonomi dan Bisnis, Univesitas Budi Luhur, Petukangan, Jakarta, Indonesia

Indonesian Institute of Sciences, Tangerang Selatan, Banten, Indonesia

Keywords: Validity, Reliability, Critical Factors, ISO 9001, SMEs

Abstract: To be able to survive amid intense organizational competition, Small Medium Entrepreneurs (SMEs)

require strategies to improve the quality of products/services offered. Quality improvement is clearly needed

so that companies have high competitiveness. Good product/services can be produced by good internal

processes. ISO 9001 Quality Management System is a framework that has been used extensively by SMEs

to ensure the quality of the process. But not a few SMEs have successfully implemented and obtained QMS

certification. Many factors influence the process of implementing ISO 9001 QMS, especially in the context

of SMEs that have many limitations. This study aims to test the validity and reliability of scale for

implementing ISO 9001’s Critical Factors in SMEs level. The method used is a quantitative survey of four

SMEs that have successfully implemented and obtained ISO 9001 certification. Data from the survey were

analysed using the Aiken approach to show the level of validity and reliability. The results showed that of

the 20 items tested, only 19 items were met the criteria. Item of Employee Acceptance was eliminated from

scale because the implementation of ISO 9001 is mandatory for all stakeholders so the factors could be

ignored.

1 INTRODUCTION

To be able to survive in the midst of tight

organizational competition, every organization needs

a strategy to improve the quality of products /

services offered. Quality improvement is clearly

needed so that companies have high

competitiveness. Good product / services can be

produced by good internal processes. ISO 9001

Quality Management System is a framework that has

been used widely by various organizations to

improve performance and competitiveness.

Quality Management System (QMS) as a

framework certainly requires large resources and a

number of procedures that must be implemented

correctly so that the running process in the

organization is carried out consistently according to

agreed standards, where the objectives to be

achieved are the quality and productivity of the

organization (Maranhao, 2005) Based on the above

understanding, QMS does not only focus on

organizational structures that describe the duties and

responsibilities of personnel, but also explains how

and what each personnel must do to achieve

organizational goals. This is closely related to

process management in organizations (Moura,

2003). In other words, process management is

important to optimize all available resources and

QMS helps organizations to manage business

processes that run better (Psomas, 2010). The

process is standardized so that there is harmony

between process objectives and customer needs

(Conti, 2004; Miguel, 2001). The success of an

organization is not related to the choice of certain

standards but rather to how the standard

implementation process is carried out properly so

that it can finally obtain recognition in the form of

certification. Among the existing standards, QMS

ISO 9001 has been adopted globally and recognized

by various organizations in the world. The QMS ISO

9001 is designed based on a process model that can

be applied to various business or organizational

models. The ISO 9001 standard is written in general

(not technical) language so that it can be understood

by various parties (Pearch & Kitka, 2010) This can

be shown by the growth of ISO 9001 QMS

penetration every year where the system is adopted

with motivation that is improving efficiency,

competitiveness, satisfaction customers (Psomas,

Napitupulu, D., Adiyarta, K. and Khalid, Z.

Validity of Critical Factors Scale for SMEs Readiness in Implementing Quality Management System.

DOI: 10.5220/0009491400130018

In Proceedings of the 1st Unimed International Conference on Economics Education and Social Science (UNICEES 2018), pages 13-18

ISBN: 978-989-758-432-9

2010; Magd, 2008). In the early years, ISO 9001

QMS was widely applied to industrial scale, but

currently almost all sectors have implemented this

standard including the SMEs sector (Boiral, 2003).

Figure 1 below illustrates the growth of ISO 9001

QMS penetration in recent years as follows:

Figure 1: Growth of ISO 9001 Penetration (ISO, 2014)

The implementation of ISO 9001 QMS correctly can

provide benefits exceeding the costs incurred

because QMS can improve performance and build

sustainable competitive advantage (Lin & Jang,

2008; Augustyn & Pheby, 2000). The advantage of

implementing QMS according to some literature is

increasing awareness of the importance of quality

and customer satisfaction, reducing costs and

customer complaints, standardizing work procedures

and improving communication and increasing

market share (Cebeci & Beskese, 2002; Dwyer,

2002; Herasm 2002; Arauz & Suzuki , 2004).

However, it should be noted that the implementation

of QMS in an organization will not produce optimal

results if not done correctly (Psomas, 2010).

Undeniably, many organizations are implementing

ISO 9001 QMS just to get certification so that the

quality of the process is ignored (Claver & Molina,

2003). Certification is often the demand of

customers or organizations that must be fulfilled as

one of the business requirements. As a result, many

organizations prioritize certification and override

product / service quality improvements that should

be the focus of the organization. Although this

certification is actually much criticized, because this

is not a riskless job.

In fact, many SMEs have difficulties in

implementing QMS 9001 to obtain certification

(Gustafsson, 2001; Trust, 2006; Yuwono, 2012).

Many obstacles faced by SMEs range from limited

resources, expertise and skills to rejection from

internal stakeholders (Nwankwo, 2000; Hudson,

2001; Garengo, 2007). In order to ensure the

deployment of an ISO 9001 effective QMS, it is

intended to identify and validate some critical

factors to ensure the successful implementation of

ISO 9001 QMS. Researchers have already found

twenty (20) Critical Factors from literature-based

and in this research, The validity of all factors has

been certified in QMS ISO 9001.

2 METHODS

Validity in research is a very important problem

because it involves the accuracy of the measuring

instrument used. It can be interpreted that an

improper instrument will have implications for the

validity of the results of the research itself. Validity

determines the extent to which a measuring

instrument actually describes what is being

measured. In practice, psychometrics experts have

developed various ways, techniques and methods to

improve the validity of items on instruments, one of

which is through content validity which is the first

step in assessing the suitability of the scale items

used. This evaluated the significance of critical

factors of ISO 9001 implementation for SMEs

context is proposed through its content validity.

Content validity reflects the representation and

relevance of a set of items used to measure a concept

that is carried out through a rational analysis of the

content of the test through an expert panel

assessment. Content validity refers to the accuracy

of measurements based on instrument content to

ensure that the scale items used have fulfilled the

entire contents of the concept or the suitability of the

items.

Content validity is the validity estimated through

testing of the feasibility or relevance of the test

content through rational analysis by a competent

panel or through expert judgment. Content validity

ensures that measurements include a sufficient and

representative set of items that reveal the concept.

The more the item scale reflects all the concepts

measured, the greater the validity of the contents. In

other words, content validity is a function of how

well the dimensions and elements of a concept have

been described (Sekaran, 2006). Content validity is

done to ascertain whether the contents of the

questionnaire are appropriate and relevant to the

purpose of the study. The validity of the content

shows the contents reflecting the complete range of

attributes under study (Devon et al, 2007). Estimates

of the validity of the content of the tests are obtained

thoroughly and systematically in examining test

items to determine the extent to which they reflect

and do not reflect the content domain (Kowsalya,

2012). Thus, content validity shows the high and

UNICEES 2018 - Unimed International Conference on Economics Education and Social Science

low agreement among experts who assess the

feasibility of a measurement scale (Azwar, 2012).

As said, content validity is the representation of

content that should be to evaluate (Napitupulu,

2005). Content validity should be conducted in the

instrument development stage and also judgmental

one (Burns, N., & Grove, 1993). Developing

instrument had the purpose of understanding the

construct that is being measured. The constructs

could be obtained from qualitative ways such as

literature reviews, interviews, and focus groups.

Selecting the domain of constructs will bring

undoubtedly research variables, scope and elements

of the subject could be obtained.

In the other hand, the instrument judgment stage

is based on expert opinion that surveyed with a

questionnaire in quantitative ways (Yaghmaie,

2003). In this case, during testing, the validity of an

item, an expert gave opinion or agreement according

to the measurement item that is being assessed.

Content validity is determined by professional or

expert and assessed through an expert agreement,

not by the researcher (Yang, 2011). In another word,

content validity shows the extent of expert

agreement toward item or construct in the instrument

being assessed. Involvement of at least five to ten

experts in the same domain was valuable to evaluate

each item or construct of the instrument (Yaghmaie,

2003). Conducting the content validity become the

main activity in testing instrument had been

designed. Therefore, this study used quantitative

survey to explore expert agreement or opinion about

critical factors of ISO 9001 implementation as items

proposed in the instrument. Based on expert

comment or rating quantitatively, the statistical

approach used to measure the extent of instrument

validity. In this research, we tested the validity using

Aiken approach (1980, 1985) that widely used to

validate a scale (Yu, 1993; Lai & Chang, 2007;

Aiken, 1980; Aiken, 1985). The extent of agreement

between experts indicated the significance of items,

and it was calculated symbolized by V coefficient.

Content validity coefficient (V) has a value

between 0 to 1 where the higher the value of the

coefficient V, the higher the validity of the contents.

In other words, if the content validity coefficient (V)

is large and reaches the standard of significance

specified for a number of experts (expert) it can be

said that the item tested has good content validity.

The value of the coefficient V depends on the rating

category or scale (c) which has a range from 2 to 7

and the number of items (m) or experts as an

appraiser (n) that has a range of 2 to 25 (Aiken,

1985). V coefficient based on Aiken was formulated

(Aiken, 1980; Aiken, 1985):

V = ∑ S / [ n (c-1) ] (1)

S = r – lo (2)

With n is the number of experts. The coefficient of

V had a value from lowest 0 to maximum of 1.

According to Aiken Table (Aiken, 1980; Aiken,

1985), content validity index (V) required of the

item is significant if above the cut off value 0.75

(V>0.75). It means if the validity index of the item

below 0.78 (V>0.75). It means if the validity index

of the item below 0.70 (V<0.75), an item doesn't

have a good content validity or not significant (Yu,

1993, Lai & Chang, 2007; Aiken, 1985; Cohen,

1960).

In the other hand, V Coefficient was confirmed by

the extent of consistency of items namely

homogeneity reliability. Homogeneity reliability is

used to test how much consistency among experts in

giving the same inter-rater agreement to a

measurement item. Homogeneity reliability was

calculated symboliezed by H coefficient. H

coefficient based on Aiken was formulated (Yu,

1993, Lai & Chang, 2007; Aiken, 1985; Azwar,

2012):

H = 1 - S / (c-1) (n

-k) (3)

With n is some experts.

The reliability coefficient of homogeneity (H) also

has a range of values between 0 to 1 where the

higher the value of the H coefficient, the higher the

reliability of the homogeneity. In other words, if the

reliability coefficient of homogeneity (H) is large

and reaches a standard of significance determined

for a number of experts, it can be said that the tested

item has good internal reliability or consistency. The

value of the H coefficient also depends on the rating

category or scale (c) which has a range from 2 to 7

and the number of items (m) or experts as an

appraiser (n) that has a range of 2 to 25.

Another approach to testing content validity is

the CVR (content validity ratio) proposed by

Lawshe (1975). The weakness of the CVR approach

proposed by Lawshe (1975) is that the coefficient

value generated by this approach can be negative

because it ranges from -1 to +1. In addition to the

Lawshe approach, there is no assessment standard

that shows the coefficient can be considered

significant or not. In other words the standard of

Validity of Critical Factors Scale for SMEs Readiness in Implementing Quality Management System

significance is not explicitly defined so that the

justification of the importance of an item in

measurement becomes unclear (Yu, 1993). In

another side, some researchers proposed Cohen’s

kappa introduced by Jacob Cohen (1960), as a robust

method for nominal scale collecting data comparing

to Aiken (Cohen, 1960). However, we use Aiken

approach since we collect and analyze data based on

five points scale yaitu 1 (strongly disagree), 2

(disagree), 3 (netral), 4 (agree) and 5 (strongly

agree) for each item asked.

Instruments containing 20 items using a Likert

scale were distributed to five SMEs who had

implemented and managed to get international

recognition in the form of ISO 9001 certification. Of

the four SMEs there were 8 people surveyed and

asked for their level of approval of the list of critical

factors submitted. All 8 people are owners and

managers who understand the application of ISO

9001 in their organizations.

3 RESULT & DISCUSSIONS

At this stage, each item is validated in the form of

critical factors in the instrument that has been

obtained with the help of expert judgment. A total of

8 experts were involved in the instrument validation

activities using the Delphi technique. The Delphi

technique has been widely used and accepted to

achieve convergence of opinions about real-world

knowledge requested from experts in certain topics.

Delphi technique is designed as a group

communication process that conducts detailed

examinations and discussion of specific issues aimed

at setting goals, research policies or predictions of

future events (Hsu & Sanford, 2007). Unlike the

Focus Group Discussion (FGD), the experts were

not met face-to-face, and the identities of each

expert were hidden so that each expert did not know

the identity of the other expert. This aims to avoid

the domination of other experts and can minimize

biased opinions (Boar, 2001) as often happens in

FGDs (Afiyanti, 2008).

To reach an agreement regarding critical factors

that are considered important by experts, the Delphi

process is carried out up to 2 rounds. The results of

the instrument validity test with Aiken approach in

round 1 can be shown as follows:

Table 1: Instrument Validity Test (Round 1

)

Critical Factors of

ISO 9001 in SMEs

Validity (V)

Aiken

Customer Focus

0.81

Top Management Support

0.84

Information & Analysis

0.90

Strategy & Planning

0.87

Training & Development

0.87

Design Quality

0.84

Process Control

0.81

Continuous Improvement

0.84

Motivation

0.87

10.

Enough Funding

0.87

11.

Supplier Quality

Management

0.87

12.

Infrastructure &

Technology

0.87

13.

Team Involvement

0.72

14.

Clear Job Responsibility

0.81

15.

Awareness of ISO 9001

0.78

16.

Quality Oriented Culture

0.78

17.

Employee Acceptance

0.66

18.

Cooperation &

Teamwork

0.69

19.

Communication

0.72

20.

Audit

0.72

In Table 1, the validity of the instrument has been

tested, especially each item in the form of critical

factors ISO 9001 for SMEs. The test results with the

Aiken approach show the validity coefficient (V) has

a range from 0.66 to 0.90. Based on Aiken's

minimum significance requirement, for 8 experts

with 5 rating scales, each item must have a greater

coefficient value of 0.75 (V> 0.75). Of the 20 items,

five invalid items were found based on expert

opinions, namely the Involvement Team (0.72),

Employee Acceptance (0.66), Cooperation &

Teamwork (0.69), Communication (0.72) and Audit

(0.72). This indicates that the coefficient V does not

exceed 0.75 (V <0.75) according to the minimum

requirements. In other words, the five items are not

considered significant by expert opinion so the

consequences can be eliminated from Table 1 above.

However, after seeing that not all experts have the

same opinion or reach agreement on these five

factors, considering this matter should be carried out

by Dephi round 2 to obtain consensus. Validity test

results of round 2 instruments can be shown in Table

2 especially for the five items that have not reached

an expert agreement.

Table 2: Instrument Validity Test (Round 2

)

Critical Factors of

ISO 9001 in SMEs

Validity (V)

Aiken

Team Involvement

0.82

Employee Acceptance

0.50

Cooperation & Teamwork

0.81

Communication

0.78

UNICEES 2018 - Unimed International Conference on Economics Education and Social Science

Audit

0.78

Based on Table 2 the instrument validity testing

process was carried out on some invalid items in the

first round. After being re-confirmed to the expert

with an explanation, finally an agreement or

consensus can be found where only the Employee

Acceptance item is invalid with a coefficient of V

0.50 (V <0.50). This is in the opinion of experts that

the application of ISO 9001 is an obligation that

must be carried out by all stakeholders. Because it is

mandatory or mandatory, the Employee Acceptance

factor is no longer important. Even more important

is preparing employees through training and

socialization so that they are capable of carrying out

tasks related to ISO 9001. Furthermore, the next

stage will also be tested the level of reliability or

consistency of experts. Instrument reliability test

results can be shown in Table 3 as follows:

Table 3: Instrument Reliability Test

Critical Factors of

ISO 9001 in SMEs

Reliability

(H) Aiken

Customer Focus

0.89

Top Management Support

0.89

Information & Analysis

0.88

Strategy & Planning

0.88

Training & Development

0.88

Design Quality

0.89

Process Control

0.89

Continuous Improvement

0.89

Motivation

0.88

10.

Enough Funding

0.88

11.

Supplier Quality Management

0.88

12.

Infrastructure & Technology

0.88

13.

Team Involvement

0.90

14.

Clear Job Responsibility

0.89

15.

Awareness of ISO 9001

0.90

16.

Quality Oriented Culture

0.90

17.

Employee Acceptance

0.93

18.

Cooperation & Teamwork

0.91

19.

Communication

0.90

20.

Audit

0.90

The reliability coefficient value in Aiken is called

the homogeneity-reliabilty coefficient (H). Based on

Table 3 above, it can be shown that the reliability

coefficient has a range between 0.88 to 0.90. Based

on Aiken's minimum significance requirement, for 8

experts with 5 rating scales, each item must have a

greater coefficient value of 0.67 (H> 0.67). In Table

3 it can be seen that the entire item has met the

threshold value of the coefficient that is considered

significant, namely H> 0.67. But for Employee

Acceptance items with a high coefficient value of

0.93. This shows that all experts consistently agree

to issue the item on the proposed scale. Thus it can

be said that the instrument has been tested for

reliability and shows good internal consistency. The

total of 19 items obtained in this study have met the

required validity and reliability criteria so that this

paper contributes. The importance of this research

can provide information regarding what key factors

should be considered by SMEs in order to support

the successful implementation of ISO 9001 in their

organizations.

4 CONCLUSIONS

This study contributes to critical factors in

implementing ISO 9001 in SMEs. The results of the

validity and reliability test show that there are 19

critical factors that meet the minimum criteria or

requirements. Other contributions are the use of

Delphi techniques and Aiken's approach in testing

the validity and reliability that had not previously

been used especially in critical factors studies on the

application of ISO 9001 in SMEs. Further research

is in the form of elaborating operational definitions

and measurement indicators because critical factors

are generally still latent so that instruments can be

used to evaluate the success rate of SMEs in

implementing ISO 9001.

REFERENCES

Maranhao, M. (2005). ISO Serie 9000 (versao 2000) –

Manual de Implementacao. Rio de Janeiro:

Qualitymark. !

Moura L. R. (2003). Qualidade Simplesmente Total.

Qualitymark Editora, 2, 208.

Psomas, E. L., Fotopoulos, C. V., & Dimitrios, P. K.

(2010). Critical factors for effective implementation

!of ISO 9001 in SME service companies. Managing

Service Quality, 20(5), 440-457. !

Conti, T. (2004). How to conceptually harmonize ISO

9000 certification, levels of excellence recognition

!and real improvement. Total Quality Management

and Business Excellence, 15(5/6), 665-677. !

Miguel, P. C. (2001). Qualidade: enfoques e ferramentas.

Sao Paulo: Artliber. !

Pearch, C., & Kitka, J. (2010). Answering Questions

About ISO 9000:2000,”. Retrieved from:

http://www.pcimag.com/CDA/Archives/bcd09fb6a36a

7010VgnVCM100000f932a8c0 !

Magd, H. (2008). ISO 9001:2000 in the Egyptian

manufacturing sector: perceptions and perspectives.

Validity of Critical Factors Scale for SMEs Readiness in Implementing Quality Management System

International Journal of Quality & Reliability

Management, 25(2), 173-200.

Boiral, O. (2003). ISO 9000, outside the iron cage.

Organization Science, 14(6), 720-737. !

ISO. (2014). ISO 9001 Survey. Retrieved from:

http://www.iso.org/iso/survey2012.pdf

Lin, C. I., & Jang, W. Y. (2008). Successful ISO 9000

implementation in Taiwan: How can we achieve it,

!and what does it mean?. International Journal of

Productivity and Performance Management, 57(8),

600- 622. !

Augustyn, M. M., & Pheby, J. D. (2000). ISO 9000 and

performance of small tourism enterprises: a focus !on

Westons Cider Company. Managing Service Quality,

10(6), 374-388. !

Cebeci, U., & Beskese, A. (2002). An approach to the

evaluation of quality performance of the companies

!in Turkey. Managerial Auditing Journal, 17(1/2), 92-

100. !

Dwyer, G. (2002). Business excellence versus ISO 9000 in

Irish context – which delivers? Managerial !Auditing

Journal, 17(4), 404-412. !

Heras, I., Casadeses, M., & Dick, G.M. (2002). ISO 9000

certification and the bottom line: a !comparative study

of the profitability of Basque region companies.

Managerial Auditing Journal, 17(1/2), 72-78. !

Arauz, R., & Suzuki, H. (2004). ISO 9000 performance in

Japanese industries. Total Quality !Management and

Business Excellence, 15(1), 3-33. !

Claver, E., & Molina, T. J. (2003). Critical factors and

results of quality management: an empirical !study.

Total Quality Management, 14, 91-118.

Gustafsson, R., Klefsjo, B., Berggren, E., & Granfors-

Wellemets, U. (2001). Experiences from

implementing ISO 9000 in small enterprises – a study

of Swedish organizations. The TQM Magazine, 13(4),

232-46.

Trust. (2016). Usaha Kecil Menengah dalam menerapkan

ISO 9001. Retrieved from: http://trustmandiri.com/iso-

9001-usaha-kecil-menengah-ukm/

Yuwono, B., Zakaria, F.R., & Panjaitan, N.K. (2012).

Faktor-faktor yang mempengaruhi penerapan cara

produksi yang baik dan standar prosedur operasi

sanitasi pengolahan fillet ikan di Jawa. Manajemen

IKM, 7(1), 10-19.

Nwankwo, S. (2000). Quality assurance in small business

organisations: myths and realities. International

Journal of Quality & Reliability Management, 17(1),

82-99.

Hudson, M., Smart, A., & Bourne, M. (2001). Theory and

practice in SME performance measurement system.

International Journal of Operations & Production

Management, 21(8), 1096-1115.

Garengo, P., Biazzo, S., & Bernardi, G. (2007). Design

and implementing a performance measurement system

in SMEs: a bottom up approach. International Annual

EurOMA Conference, 14(1).

Sekaran, U. (2006). Metode riset bisnis. Jakarta: Salemba

Empat.

DeVon, H. A., et al. (2007). Psychometric toolbox for

testing validity and reliability. Journal of Nursing

scholarship, 39(2), 155-164.

Kowsalya, D.N., et al. (2012). Development and

validation of a scale to assess self-concept in mild

intellectually disabled children. International J. Soc.

Sci. & Education, 2(4), 699-709.

Azwar, S. (2012). Reliabilitas and validity, 4.0 edition,

Pustaka Pelajar.

Napitupulu, D. (2015). Kajian Empiris Faktor Sukses

Implementasi e-Government Studi kasus: Pemerintah

Kota Bogor J. Sisfo, 5, 229-36.

Burns, N., & Grove, S. (1993). The practice of nursing

research: conduct, critique, and utilization (2nd ed).

W.B. Saunders: Philadelphia, Pennsylvania, USA.

Yaghmaie, F. (2003). Content validity and its estimation.

J. Med. Educ, 3, 25–27.

Yang, W. (2011). Applying content validity coefficient

and homogeneity reliability coefficient to investigate

the experiential marketing scale for leisure farms. J.

Global Business Management, 7.

Yu, N. (1993). The computation of content validity

coefficient and homogeneity reliability coefficient in

ordinal data. Psychological, 40, 199-214.

Lai, K., & Chang, M. (2007). The Application of aiken

content validity coefficient and homogeneity

reliability coefficient in enterprise business risks scale

for hot-spring hotel industries. Journal of Management

Practices and Principles, 1, 64-75.

Aiken, L. R. (1985). Three coefficients for analyzing the

reliability, and validity of ratings. Educational and

Psychological Measurement, 45, 131-142.

Aiken, L. R. (1980). Content Validity and Reliability of

Single Items or Questionnaires. Educational and

Psychological Measurement, 40, 55-59.

Cohen, J. (1960). A coefficient of agreement for nominal

scales. Educ. Psycol. Meas., 20, 37-46.

UNICEES 2018 - Unimed International Conference on Economics Education and Social Science