MEASURING EFFECTIVENESS OF COMPUTING FACILITIES

IN ACADEMIC INSTITUTES A NEW SOLUTION FOR A

DIFFICULT PROBLEM

Smriti Sharma

Department of Industrial & Management Engineering, Indian Institute of Technology, Kanpur, U.P., India

Veena Bansal

Department of Industrial & Management Engineering, Indian Institute of Technology, Kanpur, U.P., India

Keywords: Computing Facilities, Usability, Functional Utility, User Satisfaction, Individual and Organizational Impact.

Abstract: There h

as been a constant effort to evaluate the success of Information Technology in organizations. This

kind of investment is extremely hard to evaluate because of difficulty in identifying tangible benefits, as

well as high uncertainty about achieving the expected value. Though a lot of research has taken place in this

direction, but not much is written about evaluating IT in non-profit organizations like educational

institutions. Measures for evaluating success of IT in such kind of institutes are markedly different from that

of business organizations. The purpose of this paper is to build further upon the existing body of research by

proposing a new model for measuring effectiveness of computing facilities in academic institutes. As a

baseline, Delone & McLean’s model for measuring the success of Information System (DeLone & McLean

1992,DeLone & McLean 2003) is used, as it is the most pioneering model in this regard.

1 INTRODUCTION

Given the crucial role of education in development

and the expansion of Information and

Communication technology in the global economy,

the role of IT in education cannot be ignored. Of late

there has been a major surge in the use of IT in the

territory of education. This, at the same time, has

raised the questions- How effective is IT in

academic institutions? How to measure the

effectiveness/ success of IT in educational

institutions? Effectiveness is concerned about the

impact of the information provided in helping users

do their job. It is important to evaluate the impact of

the IT on the organization as a whole rather than

looking at the quality of the system, user satisfaction

or by looking at a narrow financial perspective of the

evaluation.

The difficulties in effectively evaluating the

pact of information systems are widely

acknowledged in the IS literature (DeLone et al

1992, Willcocks & Lester 1996, Willcocks 1996).

Evidence suggests that poo

r performance of the

IS function is a serious inhibitor to good business

performance (Carlson & McNurlin 1992b). Better

use of information, both internal and external, relates

positively to profitability (Strassman 1990).

A lot of research has been undertaken in this

rega

rd to develop frameworks for measurement of

Information Systems’ success. Economic and

quantitative measures for the success of IS, however,

are difficult to obtain. Researchers and practitioners

alike often rely on subjective assessment and

surrogate measures, such as end-user computing

satisfaction (EUCS) instrument.

Saunders and Jones (1992) developed the "IS

nction Performance Evaluation Model" which

was used to describe how measures should be

selected from the multiple dimensions of the IS

function relative to specific organizational factors

and based on the perspective of the evaluator.

The model proposed by Delone et al (1992,2003)

to m

easure the effectiveness of Information System

is the most pioneering work in this regard. DeLone

and McLean Information Systems (IS) Success

Sharma S. and Bansal V. (2006).

MEASURING EFFECTIVENESS OF COMPUTING FACILITIES IN ACADEMIC INSTITUTES A NEW SOLUTION FOR A DIFFICULT PROBLEM.

In Proceedings of the First International Conference on Software and Data Technologies, pages 46-51

DOI: 10.5220/0001313900460051

 SciTePress

Model is a framework and model for measuring the

complex-dependent variable in IS research. It

concludes with a model of "temporal and causal"

interdependencies between their six categories of IS

success- Information Quality, System Quality, Use,

User Satisfaction, Individual Impact, and

Organizational Impact.

Their model depicts the relationships of the 6 IS

success dimensions. They contend that

System Quality and Information Quality singularly

and jointly affect both Use and User Satisfaction.

Additionally, the amount of Use can affect the

degree of User satisfaction. Use and User

Satisfaction are direct antecedents of Individual

Impact; and lastly, this impact on individual

performance should eventually have

some Organizational Impact. This model was later

on validated by many researchers including Seddon

and Kiew (1994), who tested the causal structure of

the model.

Inspite of being the most complete and a better

known model some shortcomings have been sighted

in this model by researchers. It does not take into

consideration the effect of extraneous variables both

internal and external to the organization. They

themselves accept that it is necessary to include the

organization type and its environment into context

before applying this model.

In the light of the above argument, we have made

an attempt to modify Delone and McLean’s model

to make it relevant for measuring the effectiveness

of computing facilities in academic institutes.

Information Quality and System Quality have been

replaced by Usability and Functional Utility. Use

construct is omitted from the proposed model.

Measures for evaluating success of IT in such kind

of institutes are markedly different from that of

business organizations. Therefore, for capturing

Individual Impact and Organizational Impact

measures suitable in the context of academic

institutes have been introduced.

2 PROPOSED MODEL

Following modifications have been proposed in the

Delone & McLean’s model

Replacing System Quality and Information Quality:

We are concerned with measuring effectiveness of

all the computing facilities of an academic institute

unlike (DeLone et al 1992) where focus is on an

individual Information System. Therefore, System

Quality and Information Quality have been replaced

by Usability and Functional Utility.

Omission of Use construct:

A main criticism of Delone and McLean has

centered on the Use construct. It is considered to be

an inappropriate measure of IS success. Its

implication is that if a system is used, it must be

useful, and therefore successful. Take the example

of an expensive design software, which is used only

by handful of students. If this software helps these

students to produce some excellent research work, it

will be considered as an asset for the institute,

irrespective of the number of students using it.

Hence, Use construct was considered as

inappropriate in this context.

Taking the points mentioned above into

consideration, the proposed model includes the

following five constructs- Usability, Functional

Utility, User Satisfaction, Individual Impact and

Organizational Impact. The relationship between the

constructs is as shown in Fig. 1.

Usability

Functional

Utility

User

Satisfaction

Individual

Impact

Organizational

Impact

Figure 1: Proposed model.

MEASURING EFFECTIVENESS OF COMPUTING FACILITIES IN ACADEMIC INSTITUTES A NEW SOLUTION

FOR A DIFFICULT PROBLEM

This model shows the interdependent nature of

success categories used.

Usability measures the extent to which the

computing facilities match user characteristics and

the skills for the tasks concerned. Functional Utility

focuses on how well the computing facilities meet

the requirements of the users. It also measures the

availability, accuracy and up-to–datedness of the

information obtained from the use of computing

facilities. User satisfaction is the most extensively

used single measure for IS evaluation (Delone et al

1992). End-user’s feelings of satisfaction arise when

he or she combines his or her perception of and

valuation of discrepancy regarding desires and

expectations from the use of computing facilities.

Individual Impact and Organizational Impact

indicate the impact of computing facilities on

individual performance and organizational

performance, respectively. Measures used for

Individual Impact are concerned with evaluating the

impact of computing facilities on an individual in

learning, course work, research work, planning and

decision making, communication and overall

productivity. Likewise, Measures of Organizational

Impact evaluate the impact of comporting facilities

on the organizational as a whole in the following

respects- innovation, research quality, pass

rate/grades, decision making, image of the institute,

capacity in terms of students, and overall

productivity of the institute.

3 MODEL VALIDATION

Aim of testing this model was to provide an

empirical evidence for the relationships between the

five constructs used in the proposed model. We

conducted a self-administered survey to collect the

primary data from the target population, which

consisted of students and faculty of five academic

institutes.

For the survey, a questionnaire was designed

based on discussions with students and faculty and

literature. Respondents were asked to fill the

questionnaire in the context of computing facilities

used in their institutes.

Questionnaire contained five sets of questions to

measure the five constructs of the model.

Questions were framed by discussions with

students and faculty of various academic institutes

and available literature. To evaluate the first

construct Usability, a set of four questions was used.

For measuring Functional Utility six questions were

framed. Four questions on Overall Satisfaction were

from Seddon and Yip (1992). To measure Individual

Impact and Organizational Usability measures the

extent to which the computing facilities match user

characteristics and Impact group of five and six

questions were used, respectively.

Likert scale was used for measurement in which

respondents indicate a degree of agreement or

disagreement with each of a series of statements

about the stimulus objects. Each statement has been

assigned seven response categories, ranging from 1

to 7. One signifies strong agreement, and seven

means strong disagreement.

3.1 Data Collection

Questionnaires were administered personally to the

students and faculty of the aforementioned institutes.

Total of 500 Questionnaires were distributed, out of

which, 411 completed questionnaires were returned

by the respondents. After screening of

questionnaires to identify illegible, incomplete, or

ambiguous responses, 31 questionnaires were

rejected. Total, 380 questionnaires were found

suitable for data analysis. Treatment of missing

values was done by substituting a neutral value.

3.2 Data Analysis and Results

To establish the model, three regression models have

been used

• Multiple regression model with Usability

and Functional Utility as independent

variables and User Satisfaction as

dependent variable.

• Simple regression model with User

Satisfaction as independent variable and

Individual Impact as dependent variable.

• Simple regression model with Individual

Impact as independent variable and

Organizational Impact as dependent

variable.

Using the abbreviations

= Usability

= Functional Utility

= User Satisfaction

= Individual Impact

= Organizational Impact

the following linear regressions are considered

= b

3.12

+ b

31.2

+ b

32.1

(1)

ICSOFT 2006 - INTERNATIONAL CONFERENCE ON SOFTWARE AND DATA TECHNOLOGIES

= b

4.3

+ b

(2)

= b

5.4

+ b

(3)

The eq. (1) represents a multiple linear

regression and (2) and (3) are simple linear

regressions, hereafter called Simple Regression 1

and Simple Regression 2 respectively.

Here b

3.12

, b

4.3

and b

5.4

are constants; b

and

are regression coefficients ,b

31..2

and b

32.1

are

partial regression coefficients. The suffix after the

dot refers to the variable held constant.

3.3 Hypotheses

The hypotheses to be tested are as follows:

H1: The partial regression coefficient b

31..2

> 0

It is assumed that if the user finds the computing

facilities easy to use, perceived usefulness of the

system will increase for him. This subsequently, will

result into increased User Satisfaction.

H2: The partial regression coefficient b

32..1

> 0

Increase in Functional Utility will result into

increased usefulness for the user and hence

increased satisfaction. The more the facilities meet

the requirements of the user the more will be the

User Satisfaction

H3: The regression coefficient b

> 0

This hypothesis states that if a student is more

satisfied with the computing facilities then it will

have a more positive Individual Impact e.g. better

learning or communication with students/faculty.

H4: The regression coefficient b

> 0

Higher Individual Impact will result into higher

Organizational Impact e.g. a positive effect of

computing facilities on learning of individual

students will result into overall improvement in pass

rate/ grades of the institute.

Data analysis was done using SPSS.

Table 1: Cronbach’s alpha.

No. of Items Cronbach alpha

Usability(X

)

4 .6790

Functional

Utility(X

)

6 .8479

User

Satisfaction(X

)

4 .8497

Individual

Impact(X

)

6 .8772

Organizational

Impact(X

)

7 .8796

High Cronbach’s alpha for all the variables in

Table 1, except for Usability, which is marginally

less, is an indication of high internal consistency.

Low value for Usability can be attributed to lower

number of items used to measure it.

Table 2: Pearson Correlation matrix.

Usability(X

)

Functional

Utility(X

)

.562 1

User

Satisfaction(X

)

.602 .815 1

Individual

Impact(X

)

.551 .774 .817 1

Organizational

Impact(X

)

.537 .722 .769 .812 1

Table 2 shows the Pearson Coefficient of

Correlation between all the variables. Pearson's

correlation coefficient (r) is a measure of the

strength of the association between the two

variables.

The coefficient of correlation between the

constructs Usability and Functional Utility is low,

which indicates their independence. The coefficients

of correlation are high for the constructs Functional

Utility and User Satisfaction; User Satisfaction and

Individual Impact; Individual Impact and

Organizational Impact as suggested by the model.

However, it is on the lower side for the constructs

Usability and User Satisfaction, which suggests that

dependence of User Satisfaction is higher on

Functional Utility as compared to Usability.

Table 3.

Adjusted R

(p-value)

Multiple

Regression

.695 .693 428.747

(0)

Simple

Regression 1

.667 .666 757.880

(0)

Simple

Regression 2

.659 .658 729.581

(0)

MEASURING EFFECTIVENESS OF COMPUTING FACILITIES IN ACADEMIC INSTITUTES A NEW SOLUTION

FOR A DIFFICULT PROBLEM

The high values of t and F- statistic in all the

cases strongly support the rejection of the Null

hypotheses, that the regression coefficients are zero.

The regression coefficients except for b

31.2

have high

positive values. Also the 95% confidence intervals

are small. The coefficients of determination show

reasonably good fit. All the above results tend to

validate the model and support all the four

hypotheses.

4 CONCLUSIONS

Results obtained from path analysis of the survey

data provide considerable empirical evidence for the

model. Results show strong dependence of User

Satisfaction on Usability and Functional Utility;

Individual Impact on User Satisfaction and

Organizational Impact on Individual Impact. All the

four Hypotheses assumed in the beginning of the

research are found be true.

An implication of the model is that because of

the causal nature of these dimensions, Usability,

Functional Utility and User Satisfaction are

sufficient to measure the effectiveness of computing

facilities.

On the basis of the small piece of work done in

this thesis, it is strongly recommended that every

academic institution should undergo through this

self screening or self assessment process. This

model can be used by academic institutes to get

regular feedbacks about their computing facilities,

which will help them in continuous improvements.

An attempt has been made to include all the

suitable measures of each construct. However, there

is a scope of including new measures for each of the

constructs. More questions can be added to the

questionnaire to measure each of these constructs,

including both positive and negative statements to

check the consistency of the respondents. Finally,

inclusion of other constructs in the model can be

investigated.

REFERENCES

Carlson, W.M., & McNurlin, B.C. (1992b). Do you

measure UP?, Computerworld, 26 (49), pp. 95-98.

Delone, W.H. & McLean, E.R. (1992). Information

systems success: the quest for dependent variable,

Information Systems Research 3, March, pp. 60-95.

DeLone, H.W. & McLean, R.E. (2003). The DeLone and

McLean model of information systems success : A

ten-year update, Journal of Management Information

Systems, 19 (4), pp. 9-30.

Saunders, C.S., & Jones, J.W. (1992). Measuring

performance of the information systems function,

Journal of Management Information Systems, 8 (4),

pp. 63-82.

Seddon, P.B. & Yip, S.K. (1992). An empirical evaluation

of user information satisfaction UIS, measures for use

with general ledger accounting software, Journal of

Information Systems, pp.75-92.

Seddon, Peter B. & Kiew, Min-Yen (1994). A partial test

and development of Delone and Mclean’s model of IS

success. Paper presented at the Fifteenth Annual

International Conference of Information Systems

(ICIS).

Strassman, P.A. (1990). The Business value of Computers:

An executive Guide, New Canaan, CI: Information

Economic Press.

Willcocks, Leslie & Lester, Stephanie (1996). Beyond the

IT Productivity Paradox, European Management

Journal 14, No. 3, pp. 279-290.

Willcocks, L. (1996). Investing in Information Systems:

Evaluation and Management, published by Chapman

& Hall, first edition.

Table 4.

Path Unstandardized

Coeff.

Std. Coeff. t (p-

value)

95%

Conf. Bounds

from to

Coeff Std. Er. Lower Upper

H1 Usability User Satisfaction .228 .037 .211 6.126 (0) .155 .302

H2 Functional Utility User Satisfaction .737 .036 .696 20.234

(0)

.666 .809

H3 User Satisfaction Individual Impact .779 .028 .817 27.530

(0)

.723 .835

H4 Individual Impact Organizational Impact .798 .030 .812 27.011

(0)

.740 .856

ICSOFT 2006 - INTERNATIONAL CONFERENCE ON SOFTWARE AND DATA TECHNOLOGIES

APPENDIX: SURVEY ON

COMPUTING FACILITIES IN

ACADEMIC INSTITUTES

This questionnaire uses a seven-point scale. The

scale represents a spectrum. 1 signifies that you

strongly agree with the given statement, and 7 means

you strongly disagree. For each question tick the

number that reflects what you think about each

statement. Computing facilities refer to computer

hardware, software and network of your institute.

PART A: Usability

1 Computing facilities are easy to use. 1 2 3 4 5 6 7

2 Computing facilities are user friendly. 1 2 3 4 5 6 7

3 It is easy to acquire skills for using the Computing facilities. 1 2 3 4 5 6 7

4 It requires lot of effort to use the Computing facilities 1 2 3 4 5 6 7

PART B: Functional Utility

1 Computing facilities meet most of your requirements. 1 2 3 4 5 6 7

2 The content of information obtained with the help of computing facilities meets your

requirements.

1 2 3 4 5 6 7

3 Computing facilities are available whenever required. 1 2 3 4 5 6 7

4 You can get in touch with sufficient sources of information by using computing facilities.

1 2 3 4 5 6 7

5 Computing facilities enable you to obtain accurate information. 1 2 3 4 5 6 7

6 Computing facilities enable you to obtain up-to-date information. 1 2 3 4 5 6 7

PART C: User Satisfaction

1 Computing facilities meet your information processing and computational needs. 1 2 3 4 5 6 7

2 Computing facilities are fast enough. 1 2 3 4 5 6 7

3 Computational facilities are effective. 1 2 3 4 5 6 7

4 Overall,you are satisfied with the computing facilities. 1 2 3 4 5 6 7

PART D: Individual Impact

1 Computing facilities help you in learning. 1 2 3 4 5 6 7

2 Computing facilities help you in course work. 1 2 3 4 5 6 7

3 Computing facilities help you in research work. 1 2 3 4 5 6 7

4 Computing facilities help you in planning and decision making. 1 2 3 4 5 6 7

5 Computing facilities help you in communication with teachers and students. 1 2 3 4 5 6 7

6 Computing facilities help you in improving your overall productivity. 1 2 3 4 5 6 7

PART E: Organizational Impact

1 Computing facilities help in encouraging innovation. 1 2 3 4 5 6 7

2 Computing facilities help in improving research quality. 1 2 3 4 5 6 7

3 Computing facilities help in improving overall pass rate/grades. 1 2 3 4 5 6 7

4 Computing facilities help in better decision making. 1 2 3 4 5 6 7

5 Computing facilities help in improving the image of the institute. 1 2 3 4 5 6 7

6 Computing facilities help you in increasing capacity in terms of students. 1 2 3 4 5 6 7

7 Computing facilities help in improving overall productivity of the institute. 1 2 3 4 5 6 7

MEASURING EFFECTIVENESS OF COMPUTING FACILITIES IN ACADEMIC INSTITUTES A NEW SOLUTION

FOR A DIFFICULT PROBLEM