IMPROVING USER SATISFACTION IN THE
POST-IMPLEMENTATION PHASE OF A LARGE-SCALE
INFORMATION SYSTEM
Colman Gantley
Athlone, Co. Westmeath, Ireland
Keywords: Usability, Post-Implementation, Information Systems, Satisfaction.
Abstract: This paper presents a framework for identifying and improving user satisfaction for implemented
information systems. Frequently, the post-implementation phase of a systems lifecycle is ignored and
problems go undetected, which leads to user dissatisfaction. A poorly designed system can become a barrier
for users and often they become reluctant to tolerate and use the system. If users resist working with the
technology, the potential for the system to generate significant organisational performance gains can be lost,
rendering the introduced system a costly mistake. This paper attempts to refine usability beyond just human-
computer interaction (HCI). It introduces the Post-Implementation, Usability Synergy (PIUS) framework,
which focuses on six different elements, including: training, functionality, reliability, working environment
and interfacing, all centred on the actual users of the system, with the aim to enhance user satisfaction
towards the new system.
1 INTRODUCTION
A large-scale information system can enable an
organisation to dramatically improve their business
model, change internal structures and over time
increase profit margins. Due to this, many
organisations around the world have invested a
considerable amount of money in setting up these
systems (Lucas, 2005; Hawking & Stein, 2002).
Although information systems (IS) can bring a
competitive advantage to organisations, the high
failure rate in deploying such systems is a major
worry. Many organisations have suffered from
ineffective IS, for instance: Whirlpool, Irish Health
Authority, Allied Waste Industry, Hershey Foods,
Boeing, Mobile Europe, Applied Materials, Waste
Management Inc, Kellogg’s, Irish Prison Service,
Irish League of Credit Unions and Nestlé (Weston,
2001; Yu, 2005). Past research on advanced IT
deployment identifies the post-implementation phase
as the critical period during which the new system
becomes embedded in the host organisation.
Conducting a post-implementation review (PIR) on
the system during this critical period can highlight
usability flaws and system weaknesses. Repairing
these flaws can ultimately limit the chances of a
large-scale system failure (Halpin, 2003; Kueng,
2002; Nielsen et al., 2001). Organisations have a
tendency to push people and process related issues to
the bottom of the list due to time and cost overruns,
many expecting users to find the new system easy to
learn. Unfortunately, this is usually an over
optimistic hypothesis. The problems causing large-
scale IS failure after a system has gone ‘live’ fall into
multiple categories, including inadequate user
training, poor system reliability, inaccurate business
functionality and shoddily designed user interfaces
(Carroll & Carrithers, 1984; Wixon et al., 1990). A
number of prominently publicised system failures
have highlighted the difficulties involved particularly
in the post-implementation phase of an IS. For
example, in 1997 the Irish Health Authority (IHA)
invested in the Personnel, Payroll and Related
Systems (PPARS) project to manage the
development and implementation of a fully
integrated Human Resource Management system.
Originally, the PPARS project was due to cost €8.8
million. However in 2007, the PPARS project has
yet to be fully integrated, largely because the system
was complicated to use, lacked general training and
failed to do what it was intended. It is estimated it
will cost over €231 million, 25 times the original
estimate to solve these problems (Hunter, 2005;
McGee, 2005; Kennedy, 2005, PPARS, 2005;
388
Gantley C. (2008).
IMPROVING USER SATISFACTION IN THE POST-IMPLEMENTATION PHASE OF A LARGE-SCALE INFORMATION SYSTEM.
In Proceedings of the Tenth International Conference on Enterprise Information Systems - HCI, pages 388-392
DOI: 10.5220/0001713703880392
Copyright
c
SciTePress
INBITE, 2005; Irish Health, 2007). Key reasons for
such failures is information systems development
(ISD) methodologies concern themselves primarily
with pre-implementation activities, rather than the
extended post-implementation phase and often
exclude any usability evaluations or users
involvement after the system goes ‘live’.
There is a growing need for research, which
endeavours to develop a framework that captures and
aggregates usability issues in the post-
implementation phase of large-scale IS. This paper
attempts to bridge a significant gap in ISD research
and endeavours to synthesise a framework that will
both examine and indicate specific usability issues
that may arise within the post-implementation phase
of an IS.
2 POST-IMPLEMENTATION
The post-implementation phase (PIP) occurs after the
system has been developed and has gone live. By
conducting an evaluation after the system during this
stage IT payoffs can be realised against their original
objectives and corrective action can be performed
where necessary. As a consequence of this
intervention, the PIP can serve extremely useful. It
can allow the IS managers gain an understanding of
the new system and can reduce the chances of
expensive failures. A well planned and executed PIR
can assist organisations to address needed changes in
the systems architecture and correct any errors in the
system that went unnoticed in the development stage,
such as an inadequately designed interface
(Nicolaou, 2003; Woodings & Everett, 1999).
Including the users in the evaluation of the system
throughout the PIP can determine whether the system
is easy to use, easy to manage and easy to learn.
Despite the huge importance of this stage, ISD
methodologies largely ignore this critical phase of
the development life-cycle. Past research proves that
very few organisations review their system after it
goes live often expecting users to find the new
system easy to learn. Regrettably, this is usually an
over optimistic assumption (Wixon et al., 1990;
Devaraj & Babu, 2004; Palvia et al., 2001; Nicolaou,
2004; Yu, 2005; Nicolaou, 2003; Woodings &
Everett, 1999; Tallon et al., 2001).
3 USABILITY
The International Standardization Organization
(ISO) established a usability standard that can be
defined as the extent to which a product can be used
by specified users to achieve specified goals with
effectiveness, efficiency and satisfaction (ISO-9241,
1998). Usability is multidimensional encompassing
user attitudes, learn-ability, effectiveness and
flexibility towards the system or product with the
aim to make users quality of life better (John, 1996;
Shneiderman, 1998). Systems with high usability
have ‘natural’ interfaces, are easy to learn, easy to
use and are associated with many positive outcomes,
such as a reduction in errors, a positive user attitude
towards the system and increased system use,
therefore gaining greater productivity for the
company (Fruhling et al., 2005; Shang & Seddon,
2000). While organisations have spent millions
implementing large-scale IS, research indicates that
potential users often unable to use the new system,
and often revert back to previous methods or use the
system incorrectly. Users of IS that have low or poor
usability often get to know and use 20–30% of the
available features, which leads to high productivity
slumps, huge loss of profits for the organisation, low
user morale and often an expensive, unused IS
(Nielsen & Coyle, 2001). Yet, organisations still
overlook the importance of usability particularly in
the PIP of the system development life cycle (Calisir
& Calisir, 2003).
4 CONCEPTUAL FRAMEWORK
A comprehensive conceptual framework is needed to
address a wide range of usability issues in the
extended PIP of a large-scale IS. Figure 1 is a
conceptual post-implementation, usability
framework developed by the author. This framework
is called The Post-Implementation Usability Synergy
(PIUS) framework. It is a working theoretical
framework that is to be informed by a future pilot
study. The key practical implication of this
framework is to enhance the user experience by
identifying any usability issues with the IS. The
framework contains six different parts but the
combination of all these elements will improve user
satisfaction with the new system. The PIUS
framework acts as a synergy. It contains six different
parts but operates as one. Each element has an
importance, but combining all the elements together,
similar to a gestalt, will be greater than the sum of
their individual effects. The PIUS framework was
designed to identify areas of user satisfaction with a
new large-scale IS, from the users point of view. The
framework comprises of the six components, these
include: user interface, business functionality, system
IMPROVING USER SATISFACTION IN THE POST-IMPLEMENTATION PHASE OF A LARGE-SCALE
INFORMATION SYSTEM
389
reliability work practices, and support and training.
All these components are built around the users of
the system. By overlooking one of these elements, it
would have a knock-on affect and reduce the user
satisfaction toward the system. If users were satisfied
with all of these elements, the usability synergy was
complete and user satisfaction would be created.
Fig. 1: The PIUS Framework.
End-Users. The purpose of the PIUS framework is to
help identify areas of user satisfaction or
dissatisfaction users are having with an IS. The best
way to achieve these areas is to know exactly what
the user wants and needs, this can be accomplished
by working closely with the user and establishing
which area(s) of the system s/he is having most
trouble with. The user is the person for who will be
working first-hand with the system. If the user is
having difficulties with a certain part of the system,
satisfaction with the system will inevitably be low. It
is essential that the system is reviewed in the post-
implementation phase from the user’s point of view,
to uncover any issues with the system. By not
considering the thoughts and opinions of the user in
the PIP it is impossible to establish whether the
system has an adequate level of reliability,
interfacing, functionality, or training supplied to
ensure the end- user can work effectively and
efficiently with the new system. For an organisation
trying to successfully implement an IS, the user
(employee) is the most important piece. If they are
having difficulties with any aspect of the system user
satisfaction will be low and the PIUS framework will
not able to piece together properly. The synergy will
be incomplete until all issues have been resolved.
Work Practices/Functionality. The business
functionality of an IS indicates the features and
capabilities the system comprises of, and can be
defined as ‘the requirements necessary to perform
the specific tasks which the users require it to do in
the operational situation’. The functionality of a
large-scale IS can be categorised into two different
components, these include:
Meeting business requirements
Meeting user’s requirements
Organisations need to have clear and unambiguous
understanding of the business objectives before the
system is designed, to ensure that the correct
functionality is obtained. Failing to meet the
requirements of the business can cause a great deal
frustration for the people trying to work with the
system. A system that is designed with limited
functionality or high inconsistency can cause
problems for the user and frequently is the main
cause for users rejecting to use the system entirely.
The IS should provide the right information to the
right person, at an accurate time. If the functionality
is inadequate, it does not matter how well the
interface is designed, ultimately the system will
create low user morale and a decrease in
productivity. If the functionality is inadequate,
regardless on how well the interface is designed or
how reliable the software is, eventually the system
will create low user confidence. If this occurs, the
PIUS framework will be incomplete and the
elements will not piece together. User satisfaction
will remain low until the functionality is fixed.
System Reliability.
The reliability of a product is
defined as ‘the measure of its ability to perform its
function when required, for a specified time, in a
particular environment’. A reliable IS must support
the needs of the user in a simple, fast and consistent
way. A reliable system must be able to protect
against unauthorized access to its physical and
logical components. The reliability of an IS can be
categorised into three different areas, these include:
The systems software
The systems hardware
The systems communications (Networks)
A reliable system is one that is capable of operating
without material error, fault, or failure during a
specified period in a specified environment. If a user
cannot access the system or cannot display data s/he
requires, it does not matter how well developed the
systems is users will reject using it. For user
satisfaction to be achieved IS reliability must be
developed effectively. If the system does not perform
as expected, it will cause problems for the person
who is trying to carry out their work. Users get very
frustrated when the system fails and they have to
redo any work they have already completed. A
system with poor reliability can reduce a user’s
satisfaction dramatically, and can affect the work
practices of the user. If this component of the PIUS
framework is missing, the user and the work
practices will be also be affected, resulting in a
dissatisfied user and a poor system.
ICEIS 2008 - International Conference on Enterprise Information Systems
390
User Interface. As far as the user is concerned, the
interface is the product, it is the part of the system,
which the user sees, hears and communicates with.
An interface should provide the user with an easy
and flexible interaction with the system therefore,
preventing users from becoming disorientated, and
assisting them carry out their working objectives
effectively.
The design of the interface must consider the
physical design of the workplace (ergonomics) and
the support documentation (training and support).
Since the functionality of systems is made available
through its user interface, its design has a huge
influence on the usability of the system. The
interface must be developed according to the needs
of the users and the business, and should be designed
after all the business process have been finalised. A
good interface requires deep understanding of the
work practices in the context of the tasks that the IS
will help carry out. It is irrelevant how well
engineered the software code or how sophisticated
the hardware is; a bad interface can ruin an otherwise
excellent system. Ultimately, a systems interface can
have a huge bearing on the training, the work
practices and the user. If this component is poor or
limited the overall PIUS framework will not be
complete and will affect the user’s ability to learn
(training) and to carry out their work appropriately.
Support and Training.
Effective support and training
is crucial when users are being introduced to a new
system. New users of high-function application
systems can become frustrated and confused by the
errors they make in the early stages of learning. Lack
of user training and a failure to fully comprehend
how the IS works, is a major reason why newly
implemented systems fail. A systems support is
made up of documentation, both online and physical
manuals and technical support, offered by
individuals who are experts in the system
architecture. If the support and training component is
missing the PIUS framework will be significantly be
hampered. Poor support and training will have a
knock on affect and can lead to an unproductive and
unsatisfied user. Training should provide the user
with the knowledge required to do their job, if
training is inadequate the user would not be able to
do their job correctly.
Working Environment / Ergonomics.
A large
number of factors play a role in ergonomics; these
include body posture and movement, environmental
factors, information and operations as well as tasks
and jobs. Ergonomics can contribute to the solution
of a large number of social problems, such as
comfort, health and safety and user performance
within the environment of the system. The goals of
ergonomics range from the basic aim of making
work safe through increasing human efficiency, to
the purpose of creating human well-being. A well-
designed environment can produce improved
productivity, efficiency, acceptance and contentment
for the user. The aim of ergonomics is to achieve an
increase in user satisfaction by making their working
life secure. Despite, the importance of human safety
in the workplace, organisations fail to recognise how
closely it relates to their success. Users who find the
environment they are working in too uncomfortable
will have low job satisfaction, and will have poor
productivity disregarding how easy the IS is to use or
how reliable it is.
5 CONCLUSIONS
The PIR and usability evaluations are critical
activities within the ISD life cycle, as far as
information evaluation is concerned. Forcing users to
use and accept systems only leads to failure or user
resistance (Nielsen & Coyle, 2001; Woodings &
Everett, 1999; Willcocks et al., 1997; Ardito et al.,
2004). This paper introduces a conceptual framework
that will ensure ultimate system usability for the
users it was designed to assist. The objective of the
PIUS framework is to achieve ultimate usability for
the user and the best way to produce a usable system
is to know exactly what the user wants and needs.
Working closely with the user and establishing
which area or areas of the system s/he is having
difficulty with can accomplish this. This framework
has been specially designed for systems that have
recently been introduced to the workplace and places
the user at the centre of the evaluation. The
framework should be incorporated into a post-
implementation evaluation of the new large-scale IS.
For user satisfaction to be achieved the organisation
must develop a suitable environment for each user.
Only when each piece of a jigsaw is fitted together
properly can a jigsaw be deemed finished. The PIUS
Framework is no different. For example, if the users
are having difficulties working with the interface, no
matter how reliable or functional the system is, the
PIUS framework will not piece together entirely.
Only when each component of the PIUS framework
is pieced together properly can the large-scale IS
achieve ultimate usability for the users. The PIUS
framework is based on current research in the area of
usability and post-implementation. However, as little
work has been done to validate these issues in a
IMPROVING USER SATISFACTION IN THE POST-IMPLEMENTATION PHASE OF A LARGE-SCALE
INFORMATION SYSTEM
391
holistic way, the next stage is to carry out field
research to verify the various elements of the PIUS
framework and to determine the relative importance
of the various pieces in addressing post-
implementation usability comprehensively.
REFERENCES
Babar, M. A.; Zhu, L.; Verner, J. & Huo, M. (2004).
Software Quality and Agile Methods. 28th Annual
International Computer Software and Applications
Conference. Hong Kong, IEEE Computer Society.
Bradford, G. (2003). 'What's Old is New Again: Training is
the Information Technology Constant', Special Interest
Group on University and College Computing Services,
Vol 14, pp.255 - 258
Calisir, F. & Calisir, F. (2003). 'The relation of interface
usability characteristics, perceived usefulness, and
perceived ease of use to end user satisfaction with
enterprise resource planning (ERP) systems', Elsevier,
Vol 20, pp.505-515
Carroll, J. & Carrithers, C. (1984). 'Training Wheels in a
User Interface', Human Aspects of Computing, Vol 27,
pp.800 - 806
Devaraj, S. & Babu, S. (2004). 'How To Measure The
Relationship Between Training And Job Performance',
Vol 47, pp.63 - 67
Dix, A.; Beale, R.; Finlay, J. & Abowd, G. (1993). Human-
Computer Interaction, Prentice Hall.
Fruhling, A.; Vreede, G.-J. d. & Chakrapani, A. (2005). A
repeatable Collaboration Process for Usability Testing.
Proceedings of the 38th Hawaii International
Conference on System Sciences. IEEE.
Halpin, L. (2003). Chaos and Complexity in the Post-
Implementation Phase of ISD. Physical & Quantitative
Science. Waterford, Waterford Institute of Technology
(WIT).
Hawking, P. & Stein, A. (2002). 'E-skills: The Next hurdle
for ERP implementations', 36th Annual Hawaii
International Conference on System Sciences
(HICSS'03), Vol 8, pp.235
Hix, D. & Hartson, H. R. (1993). Developing User
Interfaces: Ensuring Usability Through Product and
Process, Wiley Professional Computing.
Hunter, N. (2005). Hospital's PPARS warning to HSE.
http://www.irishhealth.com/?level=4&id=7916.
INBITE (2005). PPARS goes PEAR Shaped. INBITE
Conference Series.
Irish-Health (2007). PPARS system 'a nightmare'.
IrishHealth.com. Dublin.
ISO-9241 (1998). Ergonomics requirements for office
work with visual display terminals., Part 11 - Guidance
on Usability.
Jeffries, R.; Miller, J. R.; Wharton, C. & Uyeda, K. M.
(1991). 'User Interface Evaluation in the Real World:
A Comparison of Four Techniques', Association for
Computing Machinery, Vol 1, pp.119 - 124
John, B. E. (1996). 'Evaluating Usability Evaluation
Techniques', ACM Computing Surveys, Vol 28,
Kennedy, J. (2005). Plan or be damned, warns expert on
PPARS. Silicon Republic.com.
Krasner, H. (2004). Ensuring E-Business Success by
Learning from ERP Failures. IT Pro. IEEE.
Kueng, P. (2002). Impact of Workflow Systems on People,
Task and Structure: a post-implementation evaluation.
Fifth European Conference on the Evaluation of
Information Technology.
Lucas, H. C. (2005). Information Technology: Strategic
Decision Making for Managers, John Wiley & Sons.
Lyytinen, K. & Klein, H. (1985). The Critical theory of
jurgen habermas as a basis for a theory of information
systems. WG8.2 Conference: Research Methods in
Information Systems. Manchester UK, Elsevier
Science Publishing B.V.
Nicolaou, A. (2003). Quality of Post Implementation
Review for Enterprise Resource Planning.
Nicolaou, A. I. (2004). 'ERP Systems Implementation:
Drivers of Post-Implementation Success', DSS 2004
Conference Proceedings, Vol 1, pp.589-597
Nielsen, J. (1993). Usability Engineering, Florida, Morgan
Kaufmann.
Nielsen, J. & Coyle, K. P. (2001). Emerging Technology.
CIO Magazine. CIO Magazine.
PPARS (2005). Personnel, Payroll and Related Systems.
IN Health (Ed.) July.
Shang, S. & Seddon, P. (2000). A Comprehensive
Framework for Classifying the Benefits of ERP
Systems. Americas Conference on Information
Systems. Long Beach, California.
Shneiderman, B. (1998). Designing the User Interface:
Strategies for Effective Human-Computer Interaction,
Addison Wesley Longman, Inc.
Tallon, P.; Kraemer, K. L. & Gurbaxani, V. (2001).
'Executives' Perceptions of the Business Value of
Information Technology: A Process-Oriented
Approach', Journal of MIS, Vol 12,
Weston, F. C. (2001). 'ERP Implementation and Project
Management', Production and Inventory Management,
Vol 42, pp.75 - 80
Willcocks, L.; Feeny, D. & Islei, G. (1997). Managing IT
as a Strategic Resource, Shoppenhangers Road,
Maidenhead, Berkshire, SL6 2QL, England, McGraw-
Hill Book Company (Europe).
Wixon, D.; Holtzblatt, K. & Knox, S. (1990). Contextual
design: an emergent view of system design. Human
factors in computing systems: Empowering people.
Seattle, Washington, United States, ACM Press.
Woodings, T. L. & Everett, J. E. (1999). A Methodology
with Quality Tools to support Post Implementation
Reviews. 10th Australasian Conference on Information
Systems. New Zealand, ACIS.
Yu, C.-S. (2005). Causes influencing the effectiveness of
the post-implementation ERP system. Industrial
Management & Data Systems. Emerald Group
Publishing Limited.
ICEIS 2008 - International Conference on Enterprise Information Systems
392