PAIRWISE COMPARISONS, INCOMPARABILITY AND PARTIAL
ORDERS
Ryszard Janicki
Department of Computing and Software, McMaster University, Hamilton, Ontario, Canada L8S 4K1
Keywords:
Decision theory, ranking, pairwise comparisons, weak and partial orders.
Abstract:
A new approach to Pairwise Comparisons based Ranking is presented. An abstract model based on the concept
of partial order instead of numerical scale is introduced and analysed. Many faults of traditional, numerical-
scale based models are discussed. The importance of the concept of equal importance or indifference is
discussed.
1 INTRODUCTION
The method of Pairwise Comparisons could be traced
to Marquis de Condorcet 1785 paper (see (Arrow,
1951)), was explicitly mentioned and analysed by
Fechner in 1860 (Fechner, 1860), made popular by
Thurstone in 1927 (Thurstone, 1927), and was trans-
formed into a kind of (semi) formal methodology by
Saaty in 1977 (called AHP, Analytic Hierarchy Pro-
cess, see (Dyer, 1990; French, 1986; Satty, 1977)).
In principle it is based on the observation that while
ranking (“weighting”) the importance of several ob-
jects is frequently problematic, it is much easier when
restricted to two objects. The problem is then reduced
to constructing a global ranking (“weighting”) from
the set of partially ordered pairs. The name “Pair-
wise” is slightly misleading, since for every triple
some consistency rules are required (Dyer, 1990;
Koczkodaj, 1993; Satty, 1986). For the orderings of a,
b, and c are made independently, it frequently occurs
that the consistency rules are not satisﬁed. To deal
with this problem various consistency concepts have
been introduced and analysed.
At present Pairwise Comparisons are practically
identiﬁed with the controversial Saaty’s AHP. On one
hand AHP has respected practical applications, on the
other hand it is still considered by many (see (Dyer,
1990)) as a ﬂawed procedure that produces arbitrary
Partially supported by NSERC of Canada grant
rankings.
The strange and contradictory examples of rank-
ings obtained by AHP cited in the literature (see
(Dyer, 1990)) follow from the following sources:
When two objects are being compared, interval
scales are mainly used; but later, when recipro-
cal matrices are constructed the values of interval
scales are treated as the values of ratio scales. The
concept of a scale was never formally deﬁned or
analysed.
The domain of real numbers,
R , has t
wo distinct
interpretations, and these two interpretations are
mixed up in the calculi provided by AHP. The ﬁrst
interpretation is standard, numbers describe quan-
tative values that can be added, multiplied, etc.
The second interpretation is that this is just a rep-
resentation of a total order, and for instance 1.0
is “better” than 3.51 and “much better” than 13.6,
but that’s it! In this interpretation one must be
very careful when any arithmetic operations are
performed.
Even though the input numbers are “rough” and
imprecise the results are treated as if they were
very precise; “incomparability” is practically dis-
allowed.
In the author’s opinion the problems mentioned
above stem mainly from the following sources:
The ﬁnal outcome is expected to be totally ordered
(i.e. for all a, b, either a < b or b > a),
297
Janicki R. (2007).
PAIRWISE COMPARISONS, INCOMPARABILITY AND PARTIAL ORDERS.
In Proceedings of the Ninth International Conference on Enterprise Information Systems - AIDSS, pages 297-302
DOI: 10.5220/0002363102970302
c
SciTePress
Numbers are used to calculate the ﬁnal outcome.
A non-numerical solution was proposed and dis-
cussed in (Janicki and Koczkodaj, 1996), but it as-
sumed that on the initial level of pairwise compar-
isons, the answer could be only a < b or a b”,
or “a > b”, i.e. slightly in favour and very strongly
in favour were indistinguishable. In this paper we
provide the means to make such a distinction without
using numbers.
The model presented below is an extension of the
one from (Janicki and Koczkodaj, 1996) with some
inﬂuence due the results of (Fishburn, 1985; Janicki
and Koutny, 1993).
2 TOTAL, WEAK AND PARTIAL
ORDERS
Let X be a ﬁnite set. A relation X ×X is a (sharp)
partial order if it is irreﬂexive and transitive, i.e. if
a b ¬(b a) and a b c a c, for all
a, b, c X. A pair (X, ) is called a partially ordered
set. We will often identify (X, ) with , when X is
known.
We write a
b if ¬(a b) ¬(b a), that is if
a and b are either distinct incompatible (w.r.t. ) or
identical elements of X.
We also write
a
b {x | x
a} = {x | x
b}.
The relation
is an equivalence relation (i.e. it is
reﬂexive, symmetric and transitive) and it is called the
equivalence with respect to , since if a
b, there
is nothing in that can distinquish between a and b
(see (Fishburn, 1985) for details). We always have
a
b a
b, and one can show that (Fishburn,
1985):
a
b {x | a x} = {x | b x}
{x | x a} = {x | x b}
A partial order is (Fishburn, 1985)
total or linear, if
is empty, i.e., for all a, b
X. a b b a,
weak or stratiﬁed, if a
b
c a
c, i.e.
if
is an equivalence relation,
interval, if for all a, b, c, d X, a c b d
a d b c,
semiorder, if it is interval and for all a, b, c, d X,
a b b c a d d c.
Evidently, every total order is weak, every weak order
is a semiorder, and every semiorder is interval. We
mention semiorders and interval orders to make the
t
t
t
t
?
?
?
a
b
c
d
<
1
total or
linear
t
t t
t
J
J
J
J^
J
J
J
J^
a
b
c
d
<
2
weak or
stratiﬁed
q
q
q
?
?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
{a}
{b, c}
{d}
2
total
t
t t
t
J
J
J
J^
?
a
b
c
d
<
3
neither total
nor weak
Figure 1: Various types of partial orders (represented as
Hase diagrams). The total order
2
represents the weak
order <
2
, The partial order <
3
is nether total nor weak.
classiﬁcation of our rankings (deﬁned in Section 4)
complete, as in this paper we will not use much of
their rich theory. An interested reader is referred to
(Fishburn, 1985; French, 1986).
Weak orders are often deﬁned in an alternative
way, namely (Fishburn, 1985),
a partial order (X, ) is a weak order iff there ex-
ists a total order (Y, ) and a mapping φ : X Y
such that x, y X. x y φ(x) φ(y).
This deﬁnition is illustrated in Figure 1, let
φ : {a, b, c, d} {{a}, {b, c}, {d}} and φ(a) = {a},
φ(b) = φ(c) = {b, c}, φ(d) = {d}. Note that for all
x, y {a, b, c, d} we have
x <
2
y φ(x)
2
φ(y).
Weak orders can easily be represented by step-
sequences. For instance the weak order <
2
from
Figure 1 is uniquely represented by the step-sequence
{a}{b, c}{d} (c.f. (Janicki and Koutny, 1993)).
Following (Fishburn, 1985), in this paper a b is
interpreted as a is less preferred than b”, and a
b
is interpreted as “a and b are indifferent”.
The preferable outcome of any ranking is a total
order. For any total order ,
is the empty rela-
tion and
is just the equality relation. A total order
has two natural models, both deeply embedded in the
human perception of reality, namely: time and num-
bers. Most people consider the causality relation and
its time related version “earlier than” as total orders,
even though their formal models are actually only in-
terval orders (Janicki and Koutny, 1993).
Unfortunately in many cases it is not reasonable to
insist that everything can or should be totally ordered.
We may not have sufﬁcient knowledge or such a per-
fect ranking may not even exist (Arrow, 1951). Quite
often insisting on a totally ordered ranking results in
an artiﬁcial and misleading “global index”.
ICEIS 2007 - International Conference on Enterprise Information Systems
298
Weak (stratiﬁed) orders are a very natural gener-
alization of total orders. They allow the modelling of
some regular indifference, their interpretation is very
simple and intuitive, and they are reluctantly accepted
by decision makers. Although not as much as one
might expect given the huge theory of such orders (see
(Fishburn, 1985; French, 1986)). A non-numerical
ranking technique proposed in (Janicki and Koczko-
daj, 1996) produces a ranking that is weakly ordered.
If is a weak order then a
b a
b, so
indifference means distinct incomparability or iden-
tity, and the relation can be interpreted as a se-
quence of equivalence classes of
. For the weak
order <
2
from Figure 1, the equivalence classes of
<
2
are {a}, {b, c}, and {d}, and <
2
can be inter-
preted as a sequence {a}{b, c}{d}.
There are, however, cases where insisting on weak
orders may not be reasonable. Physiophysical mea-
surements of perceptions of length, pitch, loudness,
and so forth, provides other examples of qualitative
comparisons that might be analysed from the perspec-
tive of semiorders and interval orders rather than the
more precise but less realistic weak and total orders.
The reader is referred to (Fishburn, 1985; Janicki and
Koutny, 1993) for more details. In this paper we will
only use total, weak, and general partial orders.
3 PARTIAL AND WEAK ORDER
APPROXIMATIONS
Let X be a set of objects to be ranked. The problem
is that X is believed to be partially or weakly ordered
but the data acquisition process is so inﬂuenced by
informational noise, imprecision, randomness, or ex-
pert ignorance that the collected data R is only some
relation on X. We may say that R gives a fuzzy pic-
ture, and to focus it, we must do some pruning and/or
extending. Without loss of generality we may as-
sume that R is irreﬂexive, i.e. (x,x) 6∈ R. Suppose
that R is not transitive. The “best” transitive approx-
imation of R is its transitive closure R
+
=
i=1
R
i
,
where R
i+1
= R
i
R (c.f. (Fishburn, 1985)). Evidently
R R
+
and R
+
is transitive. The relation R
+
may not
be irreﬂexive, but in such a case we can use the fol-
lowing classical result (which is due to E. Schr
¨
oder,
1895, see (Janicki and Koczkodaj, 1996)) .
Lemma 1 Let Q X ×X be a transitive relation. De-
ﬁne: x <
Q
y xQy ¬yQx. The relation <
Q
is a
partial order.
Following (Janicki and Koczkodaj, 1996) we will call
<
R
+
, a partial order approximation of (ranking rela-
tion) R. If R is a partial order then <
R
+
equals R. The
relation <
R
+
is usually not a weak order.
Let us assume that X is believed to be weakly or-
dered by a relation but the discriminatory power of
the data acquisition process, which seeks to uncover
this order, is limited. The acquired data establishes
only a partial order which is a partial picture of
the underlying order. We seek, however, an extension
process which is expected to correctly identify the or-
dered pairs that are not part of the data.
Note that weak order extensions reﬂect the fact
that if x
y than all reasonable methods for ex-
tending will have x equivalent to y in the extension
since there is nothing in the data that distinguishes
between them (for details see (Fishburn, 1985)),
which leads to the deﬁnition (Janicki and Koczkodaj,
1996) below (for both weak an total orders).
A weak (or total) order
w
X × X is a proper
weak (or total) order extension of if and only if :
(x y x
w
y) and (x
y x
w
y).
If X is ﬁnite then for every partial order its
proper weak extension always exists. If is weak,
than its only proper weak extension is
w
= . If if
not weak, there are usually more than one such exten-
sions. Various methods were proposed and discussed
in (Fishburn, 1985). For our purposes, the best seem
to be the method based on the concept of a global
score function, which is deﬁned as:
g
(x) = |{z | z x}| |{z | x z}|.
Given the global score function g
(x), we deﬁne
the relation
g
w
X ×X as
a
g
w
b g
(a) < g
(b).
Proposition 1 ((Fishburn, 1985)) The relation
g
w
is a proper weak extension of a partial order .
Some other variations of g
and their interpreta-
tions were analyzed in (Janicki and Koczkodaj, 1996).
From Proposition 2 it follows that every ﬁnite partial
order has a proper weak extension. The well known
procedure “topological sorting”, popular in schedul-
ing problems, guarantees that every partial order has
a total extension, but even ﬁnite partial orders may not
have proper total extensions. Note that the total order
t
is a proper total extension of if and only if the re-
lation
equals the identity, i.e a
b a = b.
For example no weak order has a proper total ex-
tension unless it is also already total. This indicates
that while expecting a ﬁnal ordering to be weak may
be reasonable, expecting a ﬁnal total ordering is of-
ten unreasonable. It may however happen, and often
does, that a proper weak extension is a total order,
which suggests that we should stop seeking a priori
total orderings since weak orders appear to be more
natural models of preferences than total orders.
PAIRWISE COMPARISONS, INCOMPARABILITY AND PARTIAL ORDERS
299
4 THE MODEL
Let X be a ﬁnite set of objects to be “ranked”, and let
, , , <, and be a family of disjoint relations
on X. The interpretation of these relations is the fol-
lowing (compare (Satty, 1986)), a b : a and b are
of equal importance, a b : slightly in favour of b,
a b : in favour of b, a < b: b is strongly better, a b
: b is extremely better. The lists , , <, may be
shorter or longer, but not empty and not much longer
(due to limitations of the human mind(Satty, 1986)).
We deﬁne the relations
b
,
b
,
b
<, and
b
as follows:
b
=
b
< = <
b
= <
b
= <
The relations
b
,
b
,
b
<, and
b
are interpreted as
combined preferences, i.e. a
b
b : at least slightly in
favour of b, a
b
b : at least in favour of b, a
b
<b: at
least strongly in favour of b, and a
b
b : at least b is
far superior than a.
A relational structure Rank = (X, , , , <, ) is
called a ranking if the following axioms (consistency
rules) are satisﬁed:
1.
b
,
b
,
b
<,
b
are partial orders,
2. =
b
, i.e.
b
b
1
= X ×X,
3. (a b b c) (a c a c c a),
4.1. (a bb c)(a bb c) (a ca c),
4.2. (a bb c)(a bb c) (a ca < c),
4.3. (a bb< c)(a< bb c) (a< ca c),
5. (a b b c) (a b b c) a c,
6.1. (a b b c) (a c a c),
6.2. (a bb c)(a bb c) (a ca < c),
6.3. (a < bb c)(a bb< c) (a< ca c),
7.1. (a b b c) (a b b c) a c,
7.2. (a b b c) (a b b c) a c,
7.3. (a < b b c) (a b b < c) a c,
7.4. (a b b c) (a b b c) a c.
The axioms 1 7.4 follow from the interpretation of
as equal importance and , , <, as increasing
preferences.
We will say that a ranking (X, , , , <, ) is to-
tally (weakly, semi-, intervally) ordered if the relation
b
is a total (weak, semi-, interval) order.
Due to the nature of stronger preferences it is
unreasonable to expect any speciﬁc ordering of
b
<, or
b
, however if such a speciﬁc (for example semiorder)
ordering occurs, it may give some important informa-
tion about the nature of hierarchy that is modelled by
a given ranking.
Let
b
w
be a proper weak extension of
b
and let
w
=
b
w
\
b
,
Rank
w
= (X, ,
w
, , <, ).
Proposition 2 The relational structure Rank
w
=
(X, ,
w
, , <, ) is a weakly ordered ranking.
We will call Rank
w
= (X, ,
w
, , <, ) a weak
order extension of Rank = (X, , , , <, ).
When transforming Rank into Rank
w
, we change only
the weakest preference, so if Rank = (X, , , <, )
then Rank
w
= (X, ,
w
, <, ), and if
Rank = (X, , <, ) then Rank
w
= (X, , <
w
, ).
We proceed similarly when the list of preferences is
longer.
As mentioned above (see (French, 1986; Satty,
1986)) deﬁning a proper ranking is problematic if
|X| > 2, but it usually can be done if |X| = 2. Note
that if |X| = 2 only one of the relations , , , <,
is not empty.
A relational structure (X, , , , <, ) is called
a pairwise comparisons pre-ranking, if the following
properties are satisﬁed:
1. the relations , , , <, are deﬁned by pair-
wise comparisons,
2. , , , <, are disjoint and their union equals
X ×X,
3. is interpreted as equal importance and a a for
each a X,
4. , , <, are interpreted as increasing prefer-
ences.
The pre-ranking is not usually a ranking. Our goal is
to ﬁnd such a, preferably weakly ordered, ranking that
is “the best” approximation of a given pre-ranking. In
the classical, “numerical” approach this is handled by
the concept of consistency (Koczkodaj, 1993; Satty,
1986). The case (X, , <) was solved in (Janicki and
Koczkodaj, 1996). The technique used in (Janicki
and Koczkodaj, 1996) does not require any assump-
tions about the pre-ranking relations and < (except
a b a < b a <
1
b, for all a, b). The technique
can be extended to the general case, however, at the
present stage, the algorithms are complex and lacking
the elegance of the simpler case of (X,, <). Instead,
we propose a more interactive approach.
First notice that it is very unlikely that a given pre-
ranking is “random”, i.e. it does not even resemble a
ranking. When the process of classiﬁcation of pairs
is well designed, the outcome is a pre-ranking that is
ICEIS 2007 - International Conference on Enterprise Information Systems
300
“almost” a ranking. Only some pairs violate the rank-
ing axioms, the majority of pairs satisfy the ranking
axioms. Checking those axioms is in principle a triad
analysis, and the major violators are usually easy to
detect.
After ﬁnding the pairs that violate more axioms
than the other pairs (violations of axioms propagate,
but “innocent” pairs violate much less frequently), we
can either:
Repeat the pairwise comparison process for those
pairs and change the judgment. Or
Introduce a “moderator” process which will
change the relationship between those pairs, to
satisfy the axioms.
In both cases, the changes are usually minor, like from
to , etc.
The resulting ranking Rank is usually not weak, in
most cases it is only semi- or intervally ordered, so, if
we expect the outcome to be weak or total ranking,
we need to compute Rank
w
.
Computing Rank
w
is an extension process which
is expected to identify correctly the ordered pairs that
are not part of the data. The order identiﬁcation power
of weak extension procedures is substantial and vastly
underestimated. If the ranked set of objects is, by its
nature, expected to be totally ordered, the weak ex-
tension can detect it, even if the pairwise compari-
son process is not very precise, and often results in
“indifference” (see example from Table 3 in the next
section). It is a serious error to attempt to ﬁnd a
total extension without going through a weak exten-
sion process (see comments at the end of the previous
chapter). In general, admitting incomparability on the
level of pairwise comparisons is better than insisting
on an order at any cost. The latter approach leads to
an arbitrary and often incorrect total ordering.
5 AN EXAMPLE
The following experiment has been conducted. A
blindfolded person compared the weights of the eight
different stones, named A, B, C, D, E, F, G, H. The
person put one stone in their left hand and another in
their right, and then decided which of the relations
, , , <, or held. The experiment was re-
peated for the same set of stones by various people;
and then again for different stones and different num-
ber of stones. The results were very similar, and Table
1 presents the results of one such an experiment.
The pre-ranking (X, , , , <, ), where X =
{A,B,C,D,E,F,G,H}, described by Table 1 is not a
ranking. However, a simple veriﬁcation of axioms
Table 1: The ﬁrst pre-ranking. It is not a ranking, the gray
cells indicate problematic relations.
A B C D E F G H
A <
B > <
C < <
D >
E <
F >
G > >
H > <
Table 2: The second (revised) pre-ranking which is a rank-
ing. The grey cells were revised.
A B C D E F G H
A <
B > <
C < <
D > >
E < <
F >
G > >
H > <
shows that the only violations are caused by relations
between C and H, and D and E (grey cells in Table
1). The person was asked to repeat the comparison
of pairs (C,H), and (D,E). This time he produced a
slightly different answers. The new pre-ranking pre-
sented by Table 2, is now a ranking. However, it is not
a weakly ordered ranking. The partial orders
b
,
b
,
b
<, and
b
are presented in Figure 2. The relations
b
,
b
,
b
< are semiorders while
b
is an interval order, i.e.
the ranking is semiordered. This was expected due to
the nature of this type of experiments (c.f. (Fishburn,
1985)).
The stones were weighed and their weights cre-
ated an increasing total order E, H,C, A, F, D, B, G.
Note that this is the same order as weak extensions
b
w
,
b
w
and
b
<
w
. The fact that
b
<
w
correctly describes
the real ordering is very interesting since it means that
the very rough ranking described in Table 3, where
only high preferences were recorded, all weak ones
were discarded and coded as “equal importance”, was
sufﬁcient to produce a correct total ordering. It also
shows the order identiﬁcation power of the weak or-
der extension procedure. Note also that if in the pre-
ranking from Table 1, which is not a ranking, we will
use only high preferences and replace weak by indif-
ference (i.e. we replace and with , and leave
only < and ), the new pre-ranking will be a ranking.
It will differ slightly from the one of Table 3 (E D in
PAIRWISE COMPARISONS, INCOMPARABILITY AND PARTIAL ORDERS
301
Table 3: The third pre-ranking, where only , < and
were recorded. It is a ranking. It can be obtained from
Table 2 by replacing and with .
A B C D E F G H
A <
B > <
C < <
D > >
E < <
F >
G > >
H > <
t t
t t
t t t t
?
A
A
A
A
AU
G
B
E H
C
A F D
b
t t t t
t t t t
? ? ? ?
=
=
B
G
F D
E H
C A
b
<
t
t
t t t
t t t
? ? ?
A
A
A
A
A
A
AU
A
A
A
A
A
A
AU
@
@
@
@
@
@
@R
=
E H
C
D B
G
F
A
b
t t
t t
t t
t t
? ?
? ?
? ?
A
A
A
A
A
A
AU
A
A
A
A
A
A
AU
B
F
C
E
G
D
A
H
b
s
s
s
s
s
s
s
s
?
?
?
?
?
?
?
G
B
D
F
A
C
H
E
b
w
=
b
w
=
b
<
w
s s s s
s
s
s
s
?
?
+
Q
Q
Qs
A
AU
Q
Q
Qs
A
AU
+
G
B
E
H
A F
D
C
b
w
Figure 2: Partial orders
b
,
b
,
b
<, and
b
deﬁned by the rank-
ing from Table 2, and their proper weak order extensions
b
w
,
b
w
,
b
<
w
, and
b
w
(created using global score function).
The orders
b
,
b
,
b
< are semiorders,
b
is an interval order,
b
w
,
b
w
,
b
<
w
are the same total. order, and
b
w
is a weak
order.
Table 1 and E < D in Table 3), but the order
b
< will be
the same in both cases! This suggests that if an origi-
nal pairwise comparison pre-ranking is not a ranking,
a “moderator” might replace lower preferences by in-
difference, and check if the outcome is a ranking. If
it is, a weak extension may be constructed and on its
bases the initial pre-ranking might be modiﬁed. Al-
ternately, a problematic decision could be re-judged.
Apparently, all of the popular techniques based on
pairwise comparison principles suffer from many se-
tive results. These problems follow mainly from their
treatment of imprecision, knowledge incompleteness
and indifference (incomparability) (Dyer, 1990; Jan-
icki and Koczkodaj, 1996).
The method proposed in this paper does not use
numbers at all, it is entirely based on the concept of
partial orders. It emphasizes and advocates using in-
comparability and weak orderings, as opposed to in-
sistence on the comparability of all objects and a ﬁnal
total ordering. The order identiﬁcation power of the
weak order extension procedure is discussed.
REFERENCES
Arrow K. J., 1951, Social Choice and Individual Values, J.
Wiley, New York.
Dyer J. S., 1990, Remarks on the Analytic Hierarchy Pro-
cess, Management Science, Vol. 36, No. 3, pp. 244-
258.
Fechner G. T., 1860, Elemente der Psychophysik, Breitkopf
und H
¨
artel, Leipzig.
Fishburn, P. C., 1985, Interval Orders and Interval Graphs,
J. Wiley, New York.
French S., 1986, Decision Theory, Ellis Horwood, New
York.
Janicki R., Koczkodaj W. W., 1996, Weak Order Approach
to Group Ranking, Computers Math. Applic., Vol. 32,
No. 2, pp. 51-59.
Janicki R., Koutny M., 1993, Structure of Concurrency,
Theor. Computer Science 112, pp. 5-52.
Koczkodaj W. W., 1993, A new deﬁnition of consistency of
pairwise comparisons, Mathematical and Computer
Modelling, Vol. 18, pp. 79-84.
Saaty T. L., 1977, A Scaling Methods for Priorities in Hier-
archical Structure, Journal of Mathematical Psychol-
ogy, 15, pp. 234-281.
Saaty T. L., 1986, Axiomatic Foundations of the Analytic
Hierarchy Process, Management Science, Vol. 32, No.
7, pp. 841-855.
Thurstone L. L., 1927, A Law of Comparative Judgments,
Psychological Reviews, 34, pp. 273-286.
ICEIS 2007 - International Conference on Enterprise Information Systems
302