EVALUATION CONCEPT FOR INTEGRATED KNOWLEDGE
AND CO-OPERATION PLATFORMS
Claudia Loroff
Fraunhofer Institute for Software and Systems Engineering (ISST), Berlin
Keywords: Evaluation concept, knowledge management s
ystems, CSCW systems, integrated knowledge and co-
operation platforms
Abstract: This article introduces a concept for evaluating integrated knowledge and co-operation platforms which was
derived from systematic examination of computer supported co-operative work (CSCW) and knowledge
management systems and from research of available evaluation approaches to CSCW and knowledge
management systems. It consists of various evaluation perspectives (individual, group, organisation,
environment and technical system), thereby introducing comprehensive objectives, specifying topics and
exemplary items for these perspectives. Considering experiences made with this concept, potential
implementation scenarios are introduced.
1 INTRODUCTION
The impact of so-called knowledge work on
enterprises and organisations is increasing. Merging
CSCW and knowledge management systems in an
integrated knowledge and co-operation platform can
technically support the process of knowledge work
in a comprehensive way (Fuchs-Kittowski and
Reuter 2002). The value of any technical support can
be determined by evaluation (Englberger 2001). As
the idea of an integrated knowledge and co-
operation platform is new, there is no evaluation
concept available at present. Former concepts
regarded either CSCW or knowledge management
systems.
This article introduces a concept for evaluation
o
f integrated knowledge and cooperation platforms
derived from systematic examination of computer
supported co-operative work (CSCW) and
knowledge management systems and from research
of available evaluation approaches to CSCW and
knowledge management systems. Based on this
approach different evaluation perspectives are
deduced which are described by comprehensive
surveys and specifying topics.
2 DEVELOPING THE CONCEPT
The first step in developing the concept was to
approach CSCW and knowledge management
systems systematically. As a result, the following
working definitions for CSCW and knowledge
management systems were specified:
CSCW systems consist of computer technologies
for supporting any kind of communication,
cooperation, and coordination in groups.
Knowledge management systems support
varying processes of generating, gathering,
securing, transmitting and utilising knowledge
for the benefit of an organisation.
Moreover, similarities, differences
, aims, and
supporting technologies of both system classes were
identified. In CSCW systems work on the object is
at the centre. Thereby, group work independent from
time and space shall be facilitated (see Teufel et al.
1995, Krcmar and Schwabe 1996b). In knowledge
management systems, knowledge of the object is in
the focus. Technical support is meant to make
knowledge in an enterprise easy to communicate,
recyclable and expandable.
Evaluations of knowledge management systems
m
ostly examine the availability of technical support
for certain knowledge management strategies, and
resp. how often these are applied by users. These are
predominantly easily quantifiable aspects that have
been determined in research (e.g. Zack 1999).
176
Loroff C. (2005).
EVALUATION CONCEPT FOR INTEGRATED KNOWLEDGE AND CO-OPERATION PLATFORMS.
In Proceedings of the Seventh International Conference on Enterprise Information Systems, pages 176-179
DOI: 10.5220/0002551601760179
Copyright
c
SciTePress
Further evaluations of CSCW and knowledge
management systems stress the comparison of
systems and their support functions (Stevens and
Scholtz 1999, Cobos et al. 2002, Mueller-Prothmann
and Siedentopf 2003). It is generally apparent that
there are only a few comprehensive concepts for
evaluating knowledge management systems (Maier
and Haedrich 2001, Quek and Shah 2004) and
CSCW systems (Ramage 1999). As a result of the
analysis of evaluation approaches for CSCW and
knowledge management systems deficiencies could
be found in the support of unpredictable, creative or
social processes and the impact an individual, a
group, the organisation or the environment/setting
has on a CSCW or knowledge management system.
In addition, there is still a lack of evaluation
concepts explicitly designed for integrated
knowledge and cooperation platforms. Based on the
analysis of CSCW and knowledge management
systems and the findings in scientific references, an
evaluation concept for integrated platforms was
deduced and will be introduced in the following part.
3 DESCRIPTION OF THE
EVALUATION CONCEPT
The evaluation concept introduced here regards five
perspectives:
The individual perspective, i.e. an individual
systems user (employee),
The group perspective including communities,
The organisational perspective, where the system
is (partly) applied,
The environmental perspective, i.e. the relevant
world beyond the organisation,
The system perspective, i.e. the integrated
knowledge and co-operation platform.
This subdivision is based on the following
consideration: individuals can form groups. They are
employees. Groups consist of employees of an
organisation. This organisation is embedded in an
environment. The technical system (platform) is
provided to some or all individuals/groups of the
organisation and thus penetrates the other
perspectives. For this reason, some items can for
example be attributed to the individual perspective
and/or, according to focus, to the system
perspective.
The evaluation concept provides every
perspective with comprehensive tasks, specifying
topics and exemplary items. Generating an
exhaustive list of items is not sensible as they must
always be depending on the evaluated system, the
evaluation context and the requirements of the
evaluating person.
The perspectives individual, group, organisation
and environment are similar in structure. The
comprehensive tasks refer to the respective
interdependency with the system. They are specified
for every perspective by identical topics. The topic
items differ from perspective to perspective. The
group perspective was additionally supplied with the
topic “founding a group” as groups are at the centre
of interests in an integrated platform. The system
perspective that all other perspectives refer to has a
different structure. Here are predominantly system-
describing components more important than
interaction with the system.
Table 1 shows the exemplary structure of the
evaluation concept with the perspective group. The
entire structure is shown in Table 2.
Table 1: Implementation of the Group Perspective
Task Topic Indications for generating
items
Improvement Have suggestions for
improving group work been
generated and introduced to
the system? What do they
refer to?
Which impact does a group
have on the system?
acceptance/
motivation for
application
To which extent is the system
utilised for group work if
there are occasions for it?
How is the system accepted
by groups?
facilitation of
group
formation
How far is self-directed and
task-related formation of
groups facilitated?
effect by
utilising the
system on the
work per se
Which are the precise
objectives of the groups when
applying such a system for
their collaboration, and how
are these regarded by the
system?
handling of
knowledge
and know-
ledge gain
Which system features were
used in the group for
generating new knowledge
and how does the system
motivate users to
preserve/secure it?
Which effects does the system have on the group?
support of
social
processes
How strong is the trust in
other group members?
EVALUATION CONCEPT FOR INTEGRATED KNOWLEDGE AND CO-OPERATION PLATFORMS
177
Table 2: Structure of the Evaluation Concept for integrated Knowledge and Co-Operation Platforms
Perspective Comprehensive Task Specifying Topics
Improvement Which impact has the individual/
group/organisation/environment on the system?
Acceptance and motivation for usage
Impact of usage on work per se
Handling of knowledge and knowledge gain
Support of social processes
Individual
Group
Organisation
Environment
Which impact has the system on the
individual/group/organisation/environment?
Support of group formation (only group)
Development phase Which is the system status?
Available functions
Independent selection of tools
Independent organisation of knowledge/contents
Can the system be adjusted to specific user needs,
habits and wishes?
Configurable message function
Codification
Socialisation
Personalisation
How does the system support different
knowledge management strategies?
Combination of approaches
Does the system support creative processes? Support of creativity by tools
Integration and organisation of knowledge Does the system support unpredictable processes?
Integration and organisation of or with persons
Available tools Which links of the system with tools available to
users are there?
Integration of available tools
Utility
Usability
Technical
System
What are the characteristics of the functions and
how are they dealt with?
Application of system by users
4 EXPERIENCES
During the project „Knowledge co-production in
knowledge-intense services“ (German abbr.: WiKo),
the Fraunhofer Institute for Software and Systems
Engineering Berlin in collaboration with the
Fraunhofer Institute for Applied Information
Technologies developed an integrated knowledge
and co-operation platform for supporting interaction-
oriented knowledge management. In the test phase
of the prototype valuable experiences with the
evaluation concept mere gathered. Two major
approaches of putting the concept into practice were
observed:
Top-down: Hypotheses are deduced from a
given task. In doing so, relevant perspectives,
questions and topics for these perspectives are taken
into consideration. On this basis, the hypotheses are
substantiated. Defining relevant perspectives is
important in this process. Reducing the complexity
of the hypotheses to a maximum of two perspectives
for consideration proved sensible. The system
perspective in combination with one of the other
perspectives was the common case. A considerate
selection of topics allowed a fast and easy
generation of corresponding items. In approaching
step by step, the hypothesis as well as the
corresponding questions can be looked at from
different angles. This allows for generating items
comprehensively and for emphasising certain
aspects in a considered, conscious and well-founded
way.
Bottom-up: In developing survey instruments,
the evaluating persons appeared to having had
considered certain items in advance that seemed to
be of higher interest or should generally be regarded
in an evaluation. Our evaluation system allows for
bringing such items into a context. They are
allocated to a perspective and a related topic. A
further step deals with analysis of the topics
relevance for the evaluation. If the topic is relevant,
further items can be generated. If the topic is
irrelevant, the items need not be regarded.
In analysing and interpreting data, perspectives
and topics help continuingly focussing on vital
aspects, affirm or disprove hypotheses.
Summarising, with the help of perspectives,
questions and topics the survey results can be
illustrated and demand for further evaluation or
further development of the system considered may
be shown. The contents of the system perspective
provide essential questions and topics for developing
integrated knowledge and co-operation platforms
and for iterative development processes. At the time
of introducing the prototype for practical work or
platforms in operation contents of all perspectives
must be regarded. Comparing platforms, they should
ICEIS 2005 - HUMAN-COMPUTER INTERACTION
178
be done on the basis of questions and topics of the
system perspective. If there are practical experiences
at hand, all other perspectives should also be
compared for drawing comprehensive conclusions.
No evaluation considers only the questions and
corresponding topics described above. Further
aspects influencing the results also need to be
regarded. Such influences may be survey
instruments, sample surveys, the setting for the
evaluation, available time, technical and financial
resources. Bringing these factors of influence into
consideration, the application of the evaluation
concept portrayed here leads to a systematic and
standardised approach to knowledge and co-
operation platforms.
5 DISCUSSION
It was proved that the concept supports the
following phases in evaluating integrated knowledge
and co-operation platforms: the generation of
questions, hypotheses and items, whereby a top-
down respectively a bottom-up approach could be
distinguished, furthermore data analysis, the
interpretation of data, the allocation of results into a
general concept, and the deduction of future
proceedings. Applying this concept allows the
comprehensive systematic comparison of different
platforms. Unfortunately, this could not be realised
so far. Furthermore, the concept entails different
opportunities for extension. It was already pointed
out that the system perspective influences all other
perspectives. The interaction of the system with all
other perspectives is continuously brought up. The
interaction amongst the other perspectives was
explicitly not regarded for not making the evaluation
concept too complex. An extension would be a
sensible addition to the concept. A further extension
could be the development of particular items for
single topics of the perspectives. On one hand, this
would result in higher standardisation of the
approach and better comparability of results. On the
other hand, it would lead to a loss of flexibility of
the concept. This openness allows for putting
different emphasis and for adding special cases
depending on the evaluation context. Putting the
concept into action motivates to view the evaluation
of the platform from different angles. This supports
the generation of new ideas and broadens
understanding.
An evaluation concept can only provide the
evaluating person with an appropriate setting if
he/she is able to properly classify his/her own
objectives and relevant objectives of other persons,
of groups, of the organisation or even of the
environment. Predictable and unpredictable frame
conditions have a great impact on the operation and
the results of any evaluation. The evaluation concept
must therefore be flexible. The deduced perspectives
individual, group, organisation, environment and
system can each be applied individually for
generating relevant data; but grasping the platform
to be evaluated in its entirety it is necessary and
advisable to include all perspectives in the
evaluation.
REFERENCES
Cobos, R.; Esquivel, J.; Alamán, X., 2002. IT Tools for
Knowledge Management: A Study of the Current
Situation. UPGRADE, Vol. 3, Nr. 1, P. 60-71.
Englberger, H., 2001. Evaluierung. In: Schwabe, G.;
Streitz, N.; Unland, R. (Hrsg.): CSCW-Kompendium.
Heidelberg: Springer Verlag.
Fuchs-Kittowski, F.; Reuter, P., 2002. E-Collaboration für
wissensintensive Dienstleistungen. Information
Management & Consulting, Bd. 17, Nr. 4, P. 64-71.
Krcmar, H.; Schwabe, G., 1996b. CSCW-Werkzeuge.
Wirtschaftsinformatik, Nr. 38, P. 205-225.
Maier, R.; Hädrich, T., 2001. Modell für Erfolgsmessung
von Wissensmanagementsystemen.
Wirtschaftsinformatik, Bd. 5, Nr. 43, P. 497-509.
Mueller-Prothmann, T.; Siedentopf, C., 2003. Designing
Online Knowledge Communities: Developing a
Usability Evaluation Criteria Catalogue. In:
http://www.knowledgeboard.com/download/2760/kms
s03_03.pdf. Last visit: 25.5.04.
Quek, A.; Shah, H., 2004. A Comparative Survey of
Activity-Based Methods for Information Systems
Development. In: Seruca, I.; Filipe, J.; Hammoudi, S.;
Cordeiro, J. (Eds.): 6th International Conference on
Enterprise Information Systems CiCEIS, Porto,
Portugal, 2004. Vol. 5, P. 221-229.
Ramage, M., 1999. The Learning Way. University of
Lancaster. In: http://systems.open.ac.uk/page.cfm?
pageid=MagnusRthesis. Last visit: 3.7.04
Stevens, M.; Scholtz, J., 1999. Modified Field Studies for
CSCW Systems. In: http://www.nist.gov/itl/iad/
IADpapers/modified_cscw.pdf. Last visit: 6.6.04.
Teufel, S.; Sauter, C.; Mühlherr, T.; Bauknecht, K., 1995.
Computerunterstützung für die Gruppenarbeit. Bonn:
Addison-Wesley Verlag.
Zack, M. H., 1999. Managing Codified Knowledge. Sloan
Management Review, Vol. 40, No. 4, P. 45-58.
EVALUATION CONCEPT FOR INTEGRATED KNOWLEDGE AND CO-OPERATION PLATFORMS
179