EVALUATION OF IT SERVICE MANAGEMENT FROM AN
ORGANISATIONAL PERSPECTIVE
Blake E. McNaughton, Pradeep K. Ray
School of Information Systems Technology and Management, University of New South Wales, Sydney, Australia.
Keywords: IT Service Management (ITSM), IT Infrastructure Library (ITIL), IT Evaluation, IT Benefits Realisation,
ITSM Evaluation Framework.
Abstract: Applying IT Service Management (ITSM) is now a key issue in the management of the Information
Technology function. The Information Technology Infrastructure Library (ITIL) is perhaps the most popular
and influential framework consisting of a set of ‘best practice’ processes for performing ITSM. With
adoption growing globally it is important to understand the benefits that ITIL processes can bring an
organisation. To date the benefits of implementing and using the ITIL processes have merely been inferred
or assumed with very little academic research and only minimal anecdotal evidence available. Many
organisations are planning to, or have already started implementing ITIL processes into their IT function
with what would seem to be a lack of clear understanding into the tangible and more importantly intangible
benefits that these processes can bring. Since organisations are investing millions of dollars over several
years implementing ITIL it is important to evaluate the effectiveness of ITIL from the perspective of the
businesses needs. This paper presents an overview of various approaches and evaluation methodologies
with a view to develop an improved, holistic evaluation methodology for IT service management from an
organisational perspective.
1 INTRODUCTION
Applying IT Service Management (ITSM) is now a
key issue in the management of the Information
Technology function. Transforming the traditional
role of IT operations, help desk and network services
into a well governed, integrated, efficient, reliable,
user and business oriented, service delivery unit
within the organisation is of primary concern for
CIO’s and IT Management today. ITSM is by no
means a new phenomenon as it has gradually
evolved from early beginnings in the 1980’s.
Essentially “IT Service Management puts the
services delivered by IT at the centre of IT
management and is commonly defined as a set of
processes that cooperate to ensure the quality of live
IT services, according to the levels of service agreed
to by the customer. It is superimposed on
management domains such as systems management,
network management, systems development and on
many process domains like change management,
asset management and problem management.” (HP
Laboratories 2004).
Frameworks for applying ITSM such as the
popular Information Technology Infrastructure
Library (ITIL) are now in existence and being
implemented at an ever increasing rate. Research
and an understanding into the range of different
benefits that such best practice can bring, however,
is far behind this wave of implementation. To avoid
organisations spending resources and blindly
embracing ITIL we need more solid research into
the benefits of ITIL, not just financial but from a
range of perspectives such as how it effects the user,
technological efficiency and even the IT employees
– those working with the processes. Some evaluation
methodologies and their subsequent tools do exist
such as the IT balanced score card and the IT
Service Capability Maturity model, to name a few,
however there are still many gaps that need to be
filled in this area. This paper provides a discussion
of these and several other evaluation methods and
tools which can be applied to an ITIL
implementation from an evaluation and benefits
realisation lens. It also provides insight on
directions for further research into developing better
evaluation methods for this purpose.
This paper is organised as follows: First an
overview of ITSM and ITIL is provided to bring the
reader up to speed on what the best practice
140
E. McNaughton B. and K. Ray P. (2006).
EVALUATION OF IT SERVICE MANAGEMENT FROM AN ORGANISATIONAL PERSPECTIVE.
In Proceedings of WEBIST 2006 - Second International Conference on Web Information Systems and Technologies - Society, e-Business and
e-Government / e-Learning, pages 140-152
DOI: 10.5220/0001258401400152
Copyright
c
 SciTePress

framework is all about. Next it talks about research
into ITIL and why more is needed. Then a
discussion and analysis of existing assessment tools
and methods is provided and finally it concludes
with where this area of research is headed and what
we plan to achieve.
Many Internet service providers and online
services require you to manually enter information,
such as your user name and password, to establish a
connection. With Scripting support for Dial-Up
Networking, you can write a script to automate this
process.
2 OVERVIEW OF ITSM & ITIL
Throughout the last two decades a variety of ITSM
related frameworks have been developed to help
technology managers and employees better
understand the IT function and perform ITSM.
Predominantly these frameworks have been
proprietary in nature with Microsoft (Microsoft
Operations Framework), IBM (IBM Systems
Management Solutions Lifecycle) and HP (Hewlett-
Packard IT Service Management Reference Model)
all developing their own frameworks over the years.
The Information Technology Infrastructure Library
(ITIL), however, is probably the most popular and
influential ITSM framework. It is a collection of
defined and published best practice processes for IT
Service Management. ITIL was developed in 1989
by the British government in their Central Computer
Telecommunications Agency – now the Office of
Government Commerce (OGC) in an effort to
improve the management and efficiency of
Information Technology.
The IT Service Management Forum (itSMF), a
non profit association, now owns and supports the
ITIL framework with chapters in many countries
around the world. ITIL is essentially a ‘best
practice’ process framework for performing ITSM.
“The ITIL philosophy adopts a process driven
approach which is scaleable to fit both large and
small IT organisations. It considers Service
Management to consist of a number of closely
related and highly integrated processes.” (itSMF,
2001). With no defined international standards for
ITSM, ITIL has become the de facto industry
standard for ITSM and is currently in its 2
nd
version
with a 3
rd
on its way. Predictions are that the ITIL
process framework will become an ISO standard by
the end of the decade. ITIL as a process framework
can be seen below in Figure 1 at its highest level
view with the seven key processes modules sitting
between ‘The Technology’ and ‘The Business’.
The core of ITIL is Service management made
up of two key process modules: Service Support and
Service Delivery. These are the two most
commonly implemented areas of the ITIL
framework and like the other modules break down
further into their own set of defined processes.
These processes are interrelated linking together to
form a complete process map for the module. The
processes within Service Support module are shown
in figure 2 on the next page.
Each of the processes (shown in ovals) that can
be seen within the service support module have
specific goals, sub processes, workflows, routines,
inputs and outputs. Each process also has defined
roles and responsibilities associated with it and
should have an assigned process owner. Sub-
processes exist within each main process and all of
this more specific information needed to properly
understand and implement the ITIL framework is
contained in detail within the ITIL books like a
catalogue of best practice operational processes.
Existing as a process framework ITIL has been
designed such that it can be applied in any
organisation regardless of size, industry or location.
3 ITIL IN INDUSTY &
RESEARCH
Although IT Service Management has received
some attention in the IS research literature over the
years, ITIL has received very little, with only a
handful of papers specifically published on the topic
in the last 10 years. Much of this ITIL related
academic research literature comes out of Europe as
this continent has been to date the primary adopter.
Figure 1: ITIL Framework. Source: (itSMF 2004, page 1).
EVALUATION OF IT SERVICE MANAGEMENT FROM AN ORGANISATIONAL PERSPECTIVE
141

Industry world wide on the other hand has been far
more responsive to ITIL with a variety of companies
now selling the ITIL books, running training
courses, providing certification, and consulting. As
organisations push for better IT governance and
more efficient IT functions the ITIL framework
invariably comes up as a potential solution. Popular
help desk or IT Service Desk software vendors such
as Heat, Remedy, and HP have already had their
software products certified as being ITIL compliant
and many more of these software products (around
20) exist in the market place already compliant.
In terms of ITIL around the world, organisations
from the nations of Europe have been the primary
adopters with countries such as the UK and the
Netherlands leading the way. Germany and parts of
Scandinavia have also been implementing ITIL as
well. The USA, Australia, New Zealand and South
Africa have all been initially slow adopters however
more and more you hear or read about organisations
from these countries who have already embraced
ITIL or are in the process of implementing ITIL. To
date there are 50,000 ITIL certified professionals
world wide and around 24.1% of IT manager’s at
large organisations are familiar with the ITIL
standards (HP Laboratories, 2004).
4 WHAT ARE THE BENEFITS OF
ITIL?
With adoption growing globally it is important to
understand the benefits that ITIL processes can bring
an organisation choosing to embrace this best
practice. Like a lot of ‘best practice’ out there for
organisations to use, ITIL at first glance looks like
an ideal framework of processes to help improve IT
Service Management within an organisation.
Without research targeted specifically at the benefits
of implementing ITIL, however, it will risk the
possibility of remaining an industry buzz word or
fad implemented by management because its is the
latest trend in IT Management demanded by
stakeholders and discussed at conferences. Is it a
silver bullet solution to improve the management,
control and operations of IT within an organisation
or is it more trouble than it’s worth? These are
questions organisations and researchers should be
asking.
The objectives of implementing ITIL and hence
Service Management are many but perhaps the key
benefits an organisation would expect are; alignment
of IT services with current and future business
needs, improved quality of delivered IT services and
a reduction in the long term costs of service
provision. The itSMF states that one of the key
reasons to implement ITIL is to “improve IT
efficiency and effectiveness whilst improving the
overall quality of service to the business within
imposed cost constraints” (itSMF, 2004). These
claimed benefits, however have not yet been
demonstrated through practical research.
Basic survey research has been conducted in
regard to ITIL by research companies such as
TechRepublic, Forrester and Gartner, but as
mentioned very little has been done in relation to
organisations implementing ITIL and the benefits
and value they have measured and gained. Only a
handful of notable research papers seem to exist
(Barafort, et al. 2002, Potgieter, et al. 2004,
Hochstein, et al. 2005) on the topic of ITIL and its
benefits, two of which are yet to be published in
academic journals. It seems Industry has a
reasonable understanding of ITIL in regard to
implementation as evident by the amount of training,
certification and consulting available, yet a lack of
knowledge exists both in industry and academic
circles as to the researched, measured, quantifiable
benefits that the ITIL framework has brought
organisations who have adopted it.
To date the benefits of implementing and using
the ITIL processes have merely been inferred or
assumed (Potgieter, et al. 2004, Hochstein, et al.
2005) Regardless of the fact that ITIL has been
Figure 2: ITIL Service Support ModuleSource: (itSMF
2004, page 16).
WEBIST 2006 - SOCIETY, E-BUSINESS AND E-GOVERNMENT
142

around for the last 15 years, is now in its second
version, may become an ISO standard by the end of
the decade, and the existence of a complete
supporting industry, (i.e. Training, Publication,
Consulting, Websites and Software) little formal
academic research into ITIL exists. Despite
anecdotal evidence and positive claims made by
organisations who have adopted ITIL practices there
still seems to be a minimum of quantifiable data on
the benefits and impact that ITIL will have on an
organisation choosing to embrace the process.
Small snippets of information such as how Procter &
Gamble reduced help desk calls by 10% and the
Ontario Justice Department cut support costs by
40% do little to create a holistic picture of what ITIL
is doing for organisations, what about dimensions
such as intangible benefits, technological efficiency
and service quality?
Given that some organisations have already
made the change to this best practice framework for
IT Service Management, the apparent lack of
research and understanding into the benefits and
value of ITIL is somewhat surprising. Forrester
Research reports that 2005 will be the year when
ITIL goes mainstream and widespread adoption of
ITIL best practices by internal IT departments will
follow through to 2008 (Mendel, 2004). Many
organisations therefore are planning to, or have
already started implementing some or all of the ITIL
processes into their IT function with what would
seem to be a lack of clear awareness into the
tangible and more importantly intangible benefits
that these processes can bring. With some ITIL
implementations taking a few years to fully
implement and budgets running into the millions for
large organisations wouldn’t it make sense to avoid
misunderstanding and unreal expectations? The
need for evaluating organisational IT/IS investments
is widely accepted as well as the reasons for
undertaking a sound benefits realisation program to
gauge the value of the organisational change,
however “The tracking and harvesting of benefits is
a weakness and a crucial priority for most
organisations” (Pink Elephant, 2004). The
importance and the management of evaluation for
ITIL implementations should therefore be no
different. But knowing how best to do this is the
hard part.
ITIL’s espoused benefits are far and wide
ranging both long term and short term, tangible and
intangible. Traditional single dimension measures
such as Return on Investment (ROI) calculations
will not suffice in completing the picture. An
organisation may see little financial impact from
implementing ITIL processes yet dramatically
improve the quality of service provided to IT users
and see increased morale in IT staff who are now
working to a standard framework and speaking a
common language.
What is therefore needed is a prescriptive,
multiple dimension framework with both objective
and subjective means of capturing all the benefits of
ITIL from a variety of perspectives or dimensions.
As far as we are aware, no such evaluation
framework exists but there are several good starting
points in the form of existing assessment tools which
have been designed for the ITSM domain or can be
easily applied to this domain. We explore several of
these next and discuss their advantages and
disadvantages.
5 POSSIBLE EVALUATION
METHODOLOGIES
Here existing evaluation methodologies from
different spheres of literature encompassing ITIL, IT
Service Management, Service Quality, IT functional
assessment, IT evaluation and Benefits Realisation
are first described. They each look at or can be
applied to ITSM differently and therefore cover a
variety of perspectives or ways of looking at the
problem. Their potential use in an organisational
context is also discussed through the introduction of
business scenarios and demonstration against these.
Finally their suitability as an evaluation tool for ITIL
is analysed in terms of what exactly they assess,
their ability to be applied to ITIL, relative
complexity as used in an organisational context by
management, how prescriptive they are, e.g. high
level and abstract or detailed and specific, and
finally the dimensions or perspectives that they
assess on. A summary of this analysis is found in
table 1.
Before we begin it is necessary to clarify the use
of terms and definitions. A methodology is an
encompassing guideline made up of models,
processes, techniques and tools. A tool therefore is a
mechanism to implement a methodology.
5.1 itSMF/OGC Self Assessment
Tool
Designed specifically for ITIL, the self assessment
tool aims at giving an organisation an idea of how
well they are performing compared to ITIL best
practice. It is a simple interactive questionnaire
available for anyone to use online. It has a scoring
system which determines what level of ability or
assessment the organisation is currently performing
at in terms of its ITSM processes compared to what
ITIL prescribes. The goal of the self-assessment
EVALUATION OF IT SERVICE MANAGEMENT FROM AN ORGANISATIONAL PERSPECTIVE
143

questionnaire however is not to test whether there is
complete conformance with ITIL, rather the degree
to which the self-assessing organisation conforms in
the hope of guiding future decisions. The
questionnaire also aims to create awareness of
management and control issues that may be
addressed to improve the overall process capability.
5.1.1 Use in an Organisational Context
An organisation wishes to implement ITIL, so the
relevant IT manager, CIO or head of IT services first
goes online and completes the ‘ITIL Service
Management Self Assessment’ questionnaire
available at the itSMF website. A series of Yes/No
statements are answered in relation to the processes
of the two core ITIL modules – Service Delivery and
Service Support, specifically targeted at several
assessment areas based on capability levels and
current operations. Results are generated for each
assessment area of each ITIL process and scores are
reported against a pass mark. See figure 3 for an
example of the results generated for Incident
Management
From this initial assessment the organisation
realises that it is ready for ITIL having the given
pre-requisites for ITIL and management intent as
well as having over half the process capability
already in place. This initial information and results
proves useful in guiding the ITIL implementation
effort. As part of an ongoing evaluation the same
individual is directed by management to complete
the self assessment every 2 months in the to
demonstrate any continual improvement and
attainment above and beyond pass marks for each
assessment area of each ITIL process.
5.1.2 Analysis
This self assessment is both easy to use since it is
offered free and available online. It is also relatively
low in complexity and well explained. It is highly
prescriptive as the organisation using this tool needs
not define or select anything, as this has all been
done for them. While this tool proves useful as a
simple means for assessing the current state of IT
operations in regard to ITIL prescriptions its
usefulness as part of a larger methodology for ITIL
benefits realisation is questionable. First the
questions are very subjective if only one manager or
individual completes the assessment it is likely to be
Figure 3: Sample results for itSMF/OGC ITIL Sel
f
Assessment Questionnaire – Incident Management.
Figure 4: Sample results for IS Adapted SERVQUAL.
WEBIST 2006 - SOCIETY, E-BUSINESS AND E-GOVERNMENT
144

somewhat bias. Second with only yes/no answers a
possibility there is no room for scaled responses.
This means incremental improvement will be hard to
demonstrate and gauging a yes/no response can
become difficult since level of agreement to a
statement can and does vary.
5.2 IS adapted SERVQUAL
SERVQUAL is a survey based instrument consisting
of 45 questions, developed in the field of marketing
to measure service quality through the constructs of
tangibles, reliability, responsiveness, assurance and
empathy. The IS adapted SERVQUAL is a tailored
questionnaire created specifically for IT service
contexts and has been shown by researchers (Pitt, et
al. 1995) to be a suitable instrument for measuring
user opinions of IS service quality. Essentially it
compares user expectations of service quality
against their perceptions and looks for correlation
against a final overall quality rating as given by the
respondent. Kettinger & Lee (1997) in response to
debate over the IS adapted SERVQUAL expectation
construct, its reliability and validity introduced the
concept of a ‘zone of tolerance’ and derived a new
short form 27 question survey over only 4
constructs; reliability, responsiveness, assurance &
empathy.
5.2.1 Use in an Organisational Context
An organisation who has already implemented some
ITIL processes wishes to evaluate the effect this
investment in new work processes and best practice
is having on the service quality given to users. After
reading some literature an IT manager develops a
survey based on the IS adapted SERVQUAL
questions. The survey is issued via email to all IT
users in the organisation asking them to rate aspects
of the IT services based on their minimum and
desired expectations and of course perceptions of
current services. Responses come back and the
method of a “Zone of Tolerance” is used to analyse
the data and generate meaningful information, see
figure 4. This survey is issued every 3 months to
track over time any changes in user’s opinions of
service quality as indicated by the 4 constructs of
reliability, responsiveness, assurance and empathy.
5.2.2 Analysis
The IS adapted SERVQUAL could easily be used to
help managers track and evaluate changes in user
opinions of service quality over the life of their ITIL
implementations providing valuable insights into the
more intangible benefits that ITIL can bring. It is
not a complex instrument; however there are
different ways to analyse and interpret the data
which need to be considered. The set of
SERVQUAL questions can easily be found in
literature and building a survey based on these is not
too difficult.
5.3 IT Service Capability Maturity
Model
The IT Service Capability Maturity Model (IT
Service CMM) put forward by Niessink & Vliet
(1998) is similar to the widely known Software
CMM for software process improvement in which
an organisations software process capability is
measured against a five-level ordinal scale being:
Initial, Repeatable, Defined, Managed or Optimized.
Each maturity level has its own characteristics and
processes and the goal is to eventually reach the
optimized level. The IT Service CMM thus extends
this original concept into the context of IT Services,
to be used in assessing the maturity of IT service
processes and identify directions for improvement.
It was designed with ITIL specifically in mind as
ITIL was seen to lack an overall approach to the
improvement of service processes. With its
classification of levels the IS Service CMM provides
organisations with a mechanism for step-wise
improvement.
5.3.1 Use in an Organisational Context
An organisation has adopted only a few of the ITIL
Fi
g
ure 5: The 5 levels of the IT Service CMM.
EVALUATION OF IT SERVICE MANAGEMENT FROM AN ORGANISATIONAL PERSPECTIVE
145

processes and wishes to implement more to help the
running of their IT operation’s function. An IT
manager comes across the IT Service CMM and
after reading all about it and the suggested criteria
which should be met at each level decides that her
organisation is currently at the 2
nd
maturity level –
Repeatable in terms of IT services. The IT Service
CMM is discussed with other managers and a goal is
set to reach level 3 within 12 months and level 4 –
managed within 24 months. This goal is to
complement the ITIL implementation. From
readings of literature they develop their own criteria
for each level based on what the key process areas of
each level recommend.
5.3.2 Analysis
The IT Service CMM is a useful tool to complement
ITSM improvement efforts. It is average in terms of
its complexity with a lack of specific guidance on
how exactly to use it. Again the problem of
objectively determining wether a maturity level has
been met arises. Setting quantifiable targets for the
process areas within each maturity level would be
required by the organisation. The IT Service CMM
also steers away from benefits realisation focusing
only on maturity of processes.
5.4 IT Balanced Scorecard
The IT Balanced Scorecard (IT BSC) is an extension
of the traditional Balanced Scorecard developed by
Kaplan and Norton back in 1995, being a
performance management system that enables
businesses to drive strategies based on measurement
and follow-up. The IT BSC is essentially a
business-oriented process that drives goal-oriented
investments in IT. It generally applies to the whole
IT function and its goal is to derive strategies, direct
funding and resources to always keep the 4
perspectives (User orientation, Business
Contribution, Operational Excellence and Future
Orientation) in balance. “Each of the perspectives
has to be translated into corresponding metrics and
measures that assess the current situation”
(Grembergen, & Saull. 2001) Repeated overtime
with regular reassessment the IT BSC can be a great
performance management system.
5.4.1 Use in an Organisational Context
A medium sized organisation wishes to improve
their IT service delivery and support. Currently they
rely primarily on summary financial indicators
which provide only a delayed ‘snapshot’ of
performance and are not proving very useful in the
overall management of IT. This organisation decides
to try and use the concepts of the IT balanced
scorecard to create a new performance management
system. They develop measures, objectives and
targets for each perspective monitoring and keeping
each perspective in balance over time as they try to
improve their IT service support and delivery over
time.
5.4.2 Analysis
The ideas of the IT BSC in analysing and balancing
multiple perspectives may be a useful technique to
be applied to ITIL evaluation. It provides a more
thorough and holistic approach. There is a lack of
guidance on how exactly to use it as it is not very
prescriptive. Metrics and measures must be
developed by the organisation using it - examples do
exist, yet there is little in the literature on what
works best.
5.5 Stakeholder Process Approach
Adelakun and Jennex (2002) put forward a process
framework for evaluating Information Systems,
using the stakeholder approach. They stress the
need to “understand the various stakeholders’
expectations from the system at every stage of the
systems development process” (Adelakun & Jennex
2002). They propose a continuous process that has
the outcome of one phase serving as the input for the
next. This idea of incorporating all stakeholder
expectations into an evaluation is similar to the
multiple constituency approach of Chang and King
(2000), and is a useful idea for evaluating ITIL or
any ITSM improvement effort.
Figure 6: The 4 perspectives of the IT balanced scorecard.
WEBIST 2006 - SOCIETY, E-BUSINESS AND E-GOVERNMENT
146

5.5.1 Use in an Organisational Context
An organisation implementing ITIL best practice
wants a method for capturing the benefits to the
organisation. They comes across literature on
stakeholder process approach and conduct meetings
with various stakeholder groups such as users, IT
employees, help desk and support staff, process
owners, Management etc. From this, goals and
targets are created with associated means of
measurement for each stage of implementation and
the wheels are set in motion for benefits realisation.
5.5.2 Analysis
The usefulness of the stakeholder approach appears
high since success in satisfying multiple interests,
rather than meeting conventional economic and
financial criteria, would constitute the ultimate test
of corporate performance. The drawbacks are that
the literature on this method is very high level and
somewhat abstract with no direction on how best to
undergo this method. Again measures have to be
developed by the organisation.
5.6 Criteria Catalogue Based
Methodology
A Criteria Catalogue based Methodology for
Analysing Service Management Processes was put
forward by Brenner, Radisic and Schollmeyer in
2002 for the purpose of analysing existing ITSM
processes. Their goal was to develop an efficient
and practical technique for evaluating processes in
the area of IT service management. They focused
on deriving a single quality rating through
establishing a hierarchy of criteria or attributes,
assigning them weights, enacting a measurement
methodology and then compiling them into a
catalogue. As a proof of concept application, a
generic catalogue for analysing ITIL’s incident
management process (from the Service Support
Module) at the BMW User Service Centre was
created. A measurement methodology consisting of
a 0-3 rating scale and 1,2,4 weighting scheme is also
provided.
5.6.1 Use in an Organisational Context
An organisation implementing ITIL wanted a means
of evaluating the success of each process over time.
An IT manager came across the criteria catalogue
methodology. He liked the way criteria and sub
criteria could be defined, rated and weighted to
determine a final score for the performance of the
process. They went about establishing criteria
catalogues for each ITIL process. Figure 7 gives a
generic catalogue for incident management. Every
month the scores were calculated and this provided
an excellent means of assessing performance over
time and directing resources for future improvement.
5.6.2 Analysis
Many useful ideas and concepts can be drawn from
this work including the idea of weighted criteria to
establish a quality rating for a process and the use of
a hierarchy of assessment criteria. Essentially any
dimension or perspective can be assessed as criteria
are defined by the organisation and their simple
rating system provides for ease of use and weighted
calculations. It is low in complexity, being easy to
understand and use, and average in its level of
prescription since criteria must be defined by the
Figure 7: Generic Criteria Catalogue for Incident
Management. (Source: Brenner et al. 2002 p.150).
EVALUATION OF IT SERVICE MANAGEMENT FROM AN ORGANISATIONAL PERSPECTIVE
147

organisation how ever a rating scale and weighting
scheme is available.
5.7 Service Level Agreements
SLA’s are a principle tenant of ITIL and have risen
in both importance and use within organisations over
recent years. Essentially they are a contractual
obligation between service provider and customer
specifying agreed upon service levels and penalties
for when these service levels are not met. “The
purpose of a SLA has recently shifted from being a
financial contract towards an instrument for the
management of the customer’s expectations”
(Bouman, et al. 1999) SLA’s are more a tool for
ensuring adherence to given performance targets of
an IT Service yet can also provide input for analysis.
As Rautenstrauch & Scholz (2001) point out “a
sufficient specification of performance requirements
is the basis for a comprehensive performance
analysis of an information systems”. This important
idea regarding SLA’s is where organisations may
find them useful for evaluating the benefits of ITIL
processes over time.
5.7.1 Use in an Organisational Context
An IT department is planning to make some
improvements to ITSM. As part of this they
formulate a whole range of SLA’s with particular
user groups and management. Within each SLA is
defined metrics and targets which must be met and
minimum service levels which they must not go
under. Penalties are incurred for breaching these
agreements. As a means of evaluating ITSM
performance over time they use these very metrics
and measures and track the results over time on
graphs and charts.
5.7.2 Analysis
There is some guidance when it comes to defining
SLA’s, however almost anything can be
incorporated in them. Again, organisations must take
the time to define criteria and metrics within an
SLA. In addition having lots of metrics may
become cumbersome especially if they are all treated
differently with no way of knowing which are more
important.
5.8 Ramaswamy’s 8 Stage Model for
Service Process Design and
Management
Ramaswamy’s 1996 book – ‘Design and
Management of Service Processes’ provides great
insight into the management of services through 3
stages of his 8 stage model. Chapters on stage 6
measuring performance, stage 7 assessing
satisfaction and stage 8 improving performance all
contain valuable information on data collection,
defining metrics, analysis and reporting, assessing
customer/user satisfaction, statistical methods and
improving performance based on collected data for
any type of service. ITIL focuses on improving
service provision to the IT users and most of his
concepts as discussed throughout the pages of his
book can be well applied in this ITSM context of
evaluating the benefits of ITIL.
5.8.1 Use in an Organisational Context
A small-medium organisation has very much ad hoc
IT support. It therefore plans to adopt some of the
ITIL processes and is concerned with evaluating the
benefits that they bring the organisation. A manager
has a copy of Ramaswamy’s book and likes the
ideas expressed in the 8 stage model. Since ITIL is
already defined, stages 6-8 seem most useful. After
reading the appropriate chapters he has a clear
understanding of how to measure performance
through efficiency, effectiveness and capability
metrics, assess user satisfaction and work with this
information to set goals and ultimately improve
performance. A useful set of metrics to do with the
help desk e.g. calls per user for the month, first level
support resolution rate, and average time to close
calls are formulated as well as other more
technically oriented metrics and ITIL process
specific metrics. A user satisfaction survey is also
created and distributed every 3 months. Goals are
set based on baseline results and ITIL is
Figure 8: 8 stage service design and management model.
WEBIST 2006 - SOCIETY, E-BUSINESS AND E-GOVERNMENT
148

implemented with a simple way of measuring
performance, assessing satisfaction and constantly
improving performance.
5.8.2 Analysis
The concepts expressed by Ramaswamy are of
average complexity and fairly well understood.
Some experimentation would probably be required
to find out what works best for the organisation.
The level of prescription is again average with many
examples given yet room for organisational choice.
It’s applicability to ITIL comes from the fact that the
concepts can be applied to any service process not
just IT related.
5.9 Financial Measures & Methods
Financial measures and methods include more
traditional evaluations such as looking at budgets,
expenditures, Total Cost of Ownership (TCO),
Return on Investment (ROI) and Economic Value
Add (EVA) for investments and projects. Such
measures are commonly used to evaluate
IT function performance and also IT investments.
There is a lot of literature out there on these
financial assessment topics. Financial metrics
should be considered when evaluating the benefits of
ITIL as it is inferred that improved ITSM processes
should generate cost efficiencies. There is a
constant battle that exists between reducing TCO
and maximising IT function performance in
particular service levels. “Previous research showed
the reduction of IT operations costs is often
associated with a reduction in IT service levels”. Yet
it is argued that by exercising control in the form of
centralisation and standardisation TCO can be
minimized and IT service levels improved. This is
essentially the premise of ITIL. A configuration
management database helps in tracking and
standardizing the many configuration items of an
organisations IT infrastructure and a dedicated
Service Desk helps centralises service support and
service delivery. David et al. also discuss the many
cost factors for the IT function and metrics around
both financial costs and service levels, some of
which can be used for the purpose of evaluating
ITIL.
Organisational change efforts can never escape the
stranglehold of financial assessment and nor should
they as any new processes introduced to a business
especially best practice should bring some effect to
the bottom line whether it be positive or negative
and this needs to be measured. The trick is to not let
these be the only means of assessment.
5.9.1 Use in an Organisational Context
A large IT Department is planning to Implement
ITIL. Since this will be a substantial investment and
use of company resources this implementation effort
will be subject to financial assessment and formal
benefits realisation. This is in line with all projects
and initiatives costing over $50,000. A Financial
cost benefits analysis will be conducted as well as
calculating Return on Investment and the Pay Back
period. Other financial metrics such as cost of
support per user and cost of downtime will be
assessed before and after the adoption of the
processes. 3 years after ITIL processes were put in
place the organisation finally starts to see some
financial benefit from the ITSM improvement effort.
5.9.2 Analysis
Financial analysis is a common tool and is often
applied to ITSM improvement efforts. It has already
been applied to ITIL implementation with costs
being tracked and benefits expressed financially.
Managers have a good understanding of financial
metrics and they are not too complex to use provided
you have the right information. The important thing
is to ask are these the only benefits that ITIL can
bring.
6 DISCUSSION
With all these assessment tools and methods existing
out there, some more useful, complex, ITIL
appropriate and prescriptive than others the question
must be asked, which one is best when dealing with
the evaluation of ITIL implementations. Should IT
managers select one only, based on their specific
needs, or rather combine some of these? Would it
be best to create your own methods and tools for
evaluation, or contract costly consultants to do all
this for you? The answer will inevitably be one of
organisational choice. However simple guidance
should be noted. ITIL should be evaluated from
multiple perspectives with multiple methods to
capture the range of tangible and intangible benefits
that it can bring.
Since non of the discussed methodologies
completely cover every perspective that should be
looked at when evaluating ITIL it is suggested that
what needs to be done is to combine elements from
each, extend some, make them more prescriptive and
specific to ITIL, rather than starting from scratch to
create something new.
EVALUATION OF IT SERVICE MANAGEMENT FROM AN ORGANISATIONAL PERSPECTIVE
149

Table 1: Summary analysis of possible evaluation methodologies for ITSM methodologies.
Assessment
Tool/Method
What is Assessed
Applicability to
ITIL
Complexity
Level of
Prescription
Perspective/
Dimensions
itSMF/OGC ITIL
Assessment Tool
Level of ITIL
compliance. Gives the
organisation an idea
of how well it is
performing compared
to ITIL best practice.
High: Designed for
ITIL
Low: Based on a
framework and
series of capability
levels. Can be done
by anyone on
website
High: Set
questionnaire
available online
and offline in excel
spreadsheet format.
Pre-requisites
Management intent
Process Capability
Internal Integration
Products
Quality Control
Management
Information
External Integration
Customer Interface
IS Adapted
SERVQUAL
Users opinion of IS
service Quality
Medium: Useful
for assessing any
ITSM
improvement effort
Low: Only Survey
construction and
Analysis of data
required
Medium: Survey
Questions already
defined yet method
of analysis a
managerial choice
Users
Expectations and
Perceptions
IT Service
Capability Maturity
Model
Maturity of IT service
processes. Directions
for further
improvements of
service capability
High: Designed
with ITIL in mind
Medium: Lacking
guidance on how to
use it
Low: Requirements
of each Maturity
level well
explained but not
measurable
Management
Process Maturity
IT Balanced
Scorecard
4 perspectives
Balanced. Strategic
management and
measurement system
for improving
performance
Low: Originally for
wider
organisational
strategic purposes
only recently
tailored for IT
purposes
Medium: Detailed
yet Lacking
guidance on how
exactly to go about
using
Medium: Measures
are generally
defined by the
organisation
although examples
are prevalent
IT Value/Financial
User
IT Processes
Future Orientation
Stakeholder Process
Approach
Stakeholder
expectations in
relation to a system
and its evaluation
Medium:
Originally for
Systems
Evaluation can be
used for ITSM
improvement
efforts
Low: Ambiguous yet
principles easily
understood
Low: Very high
level concepts
Different Stakeholder
groups
Criteria Catalogue
for analysing
Service
Management
Processes
Existing processes
analysed based on
criteria for high
quality processes
High: As a Proof-
of-concept it was
tested against
ITIL’s Incident
Management
Low: Appears
simple enough to use
Medium: Criteria
and weights must
be self defined,
however a
weighting scheme
and rating scale is
provided.
Anything, all criteria
categorised and defined
by organisation
SLA Comparisons
Level of adherence to
SLA’s between IT
and the Business and
or improvement in
performance measures
within SLA
High: ITIL calls
for the use and
monitoring of
SLA’s
Medium: Definition
of SLA’s and
specification often a
lengthy procedure
Low: Organisations
must decide upon
and define SLA’s.
Measures with SLA
must also then be
defined
Anything as defined
within each SLA
Ramaswamy’s
8 Stage Service
Design and
Management model
Service Quality
Stage 6: Measuring
Performance
Stage 7: Assessing
Satisfaction
Stage 8: Improving
Performance
Medium: Concepts
can be applied to
any service process
Medium: Metrics
required, Satisfaction
measures required.
Descriptive and
inferential statistics
Medium:
Somewhat high
level, extensive
examples given
Effectiveness
Capability
Efficiency
Satisfaction
Financial
Standard Financial
Assessment and
Measures
Change in financial
metrics over time and
or the financial impact
of implementing
ITSM improvement
efforts. e.g. ROI,
TCO, EVA, Cost
tracking, Budgets
Medium: ITIL
brings many
intangible benefits
which will be
missed from this
approach
Low: Managers
seem to have a good
grasp of financial
measurement
Low: What to
measure is decided
by the organisation
Financial
WEBIST 2006 - SOCIETY, E-BUSINESS AND E-GOVERNMENT
150

7 CONCLUSIONS & FUTURE
WORK
We hope to lead research into this area of ITIL,
assisting managers and organisations alike by
developing a holistic framework for evaluating the
benefits of ITIL for any organisation implementing
these best practice processes helping to guide future
service improvements. Things that need to be
considered are from what perspectives organisations
need to consider benefits - Management,
Technology, User or even to look back internally at
the effect on IT employees. Any framework must be
easy to use, comprehensive, flexible and relatively
prescriptive. It should combine objective and
subjective means of assessment. It should consider
benefits in the long term and short term and at
multiple levels of operation such as the process level
and higher management level. It should also be
useful for employees working with the processes,
process owners, upper management and of course
specific to ITIL itself.
Frameworks and methodologies need to be drafted
and tested against real life ITIL case studies and
discussed with industry experts, until workable
solutions to this problem are created. Our research
will attempt to follow this journey to create a
methodology and such a tool.
ITIL is a complex piece of best practice and
managers need to be careful when deciding if they
want to invest in these processes to get them up and
running in their organisations and IT departments.
Stricter guidance on ways of evaluating ITIL
implementations, and gauging the subsequent
benefits can only help organisations now and in the
future. Hopefully a new evaluation framework will
be designed and tested in the not too distant future.
REFERENCES
Adelakun, O. & Jennex, M. 2002, ‘Stakeholder Process
Approach to Information Systems Evaluation’ in
Eighth Americas Conference on Information Systems,
pp. 1186 – 1194.
Barafort, B., Di Renzo, B. & Merlan, O. 2002, ‘Benefits
Resulting from the Combined Use of ISO/IEC 15504
with the Information Technology Infrastructure
Library (ITIL)’ Lecture Notes in Computer Science,
Vol 2559, pp. 314 – 325.
Bouman, J., Trienekens, J. & Zwan, M. 1999,
‘Specification of Service Level Agreements,clarifying
concepts on the basis of practical research’ In
Proceedings of the SoftwareTechnology and
Engineering Practice conference, Pittsburgh,
Pennsylvania, USA, August 30 - September 2, 1999.
Brenner, M., Radisic, I. & Schollmeyer, M. 2002, ‘A
Criteria Catalog based Methodology for Analyzing
Service Management Processes’ Proceedingsof the
13th IFIP/IEEE International Workshop on Distributed
Systems: Operations& Management (DSOM 2002),
Lecture Notes in Computer Science (LNCS) 2506,
pages 145–156, Montreal, Canada, October 2002.
IFIP/IEEE, Springer.
Chang, J. & King, W. 2000, ‘The Development of
Measures to Assess the Performance of the
Information Systems Function: A Multiple
Constituency Approach’ in Proceddings of the 21st
International Conference on Information Systems,
pp.640-646.
David, J., Schuff, D. & Louis, R. 2002, ‘Managing you IT
Total Cost of Ownership’, Communications of the
ACM, Vol. 45, No. 1, pp.101 – 105.
Grembergen, W. & Saull, R. 2001, ‘Aligning Business and
Information Technology through the Balanced
Scorecard at a Major Canadian Financial Group: its
Status Measured with an IT BSC Maturity Model’
Proceedings of the 34th Hawaii International
Conference on System Sciences (HICSS), Maui –
2001.
Hochstein, A., Zarnekow, R. & Brenner, W. 2005, ‘ITIL
as Common Practice Reference Model for IT Service
Management: Formal Assessment and Implications for
Practice’ in Proceedings of the 2005 IEEE
International Conference on e-Technology, e-
Commerce and e-Service (EEE'05) - Volume 00, pp.
704 – 710.
HP Laboratories 2004, IT Service Management and IT
Governance: Review, Comparative Analysis and their
Impact on Utility Computing, by M. Salle, Hewlett-
Packard Company, Palo Alto.
itSMF 2001, IT Service Management – A companion to
the IT Infrastructure Library, by I. Macfarlane, & C.
Rudd, itSMF Ltd, UK.
itSMF 2004, An Introductory Overview of ITIL – A high
level overview of the IT Infrastructure Library, by C.
Rudd, itSMF Ltd, UK.
Kettinger, W. & Lee, C. 1997, ‘Pragmatic Perspectives on
the Measurement of Information Systems Service
Quality’ Management Information Systems Quarterly,
vol.21, no.2 pp. 223 – 240.
Mendel, T. 2004, ‘ITIL’s Final Breakthrough: From what
to how’, Forrester Research URL:
http://www2.cio.com/analyst/report2843.html
(accessed 10 April 2005).
Niessink, F. & Vliet, H. 1998, ‘Towards Mature IT
Services’ Software Process Improvement and Practice,
vol.4, pp 55 – 71.
Pink Elephant, 2004. ‘The Benefits of ITIL White Paper’
URL: http://www.pinkelephant.com/
pink_site/Asset_Repository/folder/Files/Documents/B
enefits_of_ITIL.pdf (accessed 20 Feb 2005.
Pitt, L., Watson, R. & Kavan, B. 1995, ‘Service Quality: A
Measure of Information Systems Effectivness’,
Management Information Systems Quarterly, vol. 19,
no.2, pp. 173 – 185.
EVALUATION OF IT SERVICE MANAGEMENT FROM AN ORGANISATIONAL PERSPECTIVE
151

Potgieter, B., Botha, J. & Lew, C. 2004, ‘Evidence that
use of the ITIL framework is effective’, Paper
presented at the Seventeenth Annual Conference of the
National Advisory Committee on Computing
Qualifications (NACCQ), 6th - 9th July 2004, URL:
http://www.naccq.ac.nz/
conference05/proceedings_04/potg_itil.pdf, (accessed
20 April, 2005) pp. 160 – 167.
Ramaswamy, R. 1996 “Design and Manegement of
Service Processes – Keeping Customer for Life”
Massachusetts.
Rautenstrauch, C. & Scholz, A. 2001, ‘Performance
Engineering on the Basis of Performance Service
Levels’ Lecture Notes in Computer Science, vol. 2047,
pp. 68 -77.
WEBIST 2006 - SOCIETY, E-BUSINESS AND E-GOVERNMENT
152