Termination Criteria in Evolutionary Algorithms: A Survey

Seyyedeh Newsha Ghoreishi, Anders Clausen and Bo Noerregaard Joergensen

Center for Energy Informatics, The Maersk Mc-Kinney Moller Institute,

University of Southern Denmark, Odense M, Denmark

Keywords:

Evolutionary Computation, Evolutionary Algorithm, Termination Criterion, Stopping Criterion, Convergence,

Performance Indicator, Progress Indicator.

Abstract:

Over the last decades, evolutionary algorithms have been extensively used to solve multi-objective optimiza-

tion problems. However, the number of required function evaluations is not determined by nature of these

algorithms which is often seen as a drawback. Therefore, a robust and reliable termination criterion is needed

to stop the algorithm. There is a huge amount of knowledge encapsulated in the studies targeting termination

criteria in evolutionary algorithms, but an updated integrated overview of this knowledge is missing. For this

reason, we aim to conduct a systematic research through a comprehensive literature study. We extended the

basic categorization of termination criteria to a more advanced one that takes the most common used termi-

nation criteria into consideration based on their speciﬁcations and the way they have been evolved over time.

The survey is concluded by suggesting a road-map for future research directions.

1 INTRODUCTION

Evolutionary Algorithms (EAs) introduce a class of

probabilistic optimization algorithms inspired by the

principles of biological evolution which is known as

one of the main approaches in the computational in-

telligence ﬁeld (Konar, 2005). In general, the op-

timization process starts with a randomly selected

population and iterates over number of generations

by modifying the start population to obtain a near-

optimal or possibly optimal solution. The optimiza-

tion process runs until a given termination criterion

triggers. Inherently, EAs are not able to decide about

terminate of the optimization process and the ability

of automatic termination is not designed in the orig-

inal versions of EA. Therefore prespeciﬁed termina-

tion criteria should be considered either by the end-

user or the application programmer.

Since the termination criteria behave differently

for various EA, it is not possible to formulate a gen-

eral rule for optimal use of a termination criterion

(Jain et al., 2001). Therefore an automated termina-

tion is desired in real-world applications for practical

reasons. The ﬁrst reason is that the software end-users

are not often aware of the behavior of the EA. The

second reason is the necessity to ﬁnd a suitable value

for the parameters of the termination criteria to obtain

a proper ending of the EA. In most termination crite-

ria, suitable parameters can only be determined by a

trial-and-error method (Jain et al., 2001).

Recently, the rapid growth of many complex ap-

plications in science, engineering and economics

highlights the need of developing practical versions of

EA. The original versions of EA suffer from two main

weaknesses: unlearned termination and slow conver-

gence. Much of the current literature pays particu-

lar attention to overcome these two issues by iden-

tifying and evaluating a new termination criterion or

by modifying the standard versions of EA to acceler-

ate the convergence. Even though termination crite-

ria is widely used and investigated, but there are few

theoretical guidelines for determining a suitable and

proper point of time to terminate the search algorithm

(Bhandari et al., 2012).

Up to now, two prominent studies have paid par-

ticular attention to integrate the known approaches

used as termination criteria in EA (Jain et al., 2001)

and (Wagner et al., 2011). For the ﬁrst time in 2001, a

comprehensive study done by Jain to classify termina-

tion criteria into three main categories: direct, derived

and cluster-based termination criteria. Moreover, a

new cluster-based termination criteria proposed and

evaluated by comparing with other termination crite-

ria in terms of reliability and performance (Jain et al.,

2001). Finally, guidelines for the application of ter-

mination criteria provided (Jain et al., 2001). The

Ghoreishi S., Clausen A. and Joergensen B.

Termination Criteria in Evolutionary Algorithms: A Survey.

DOI: 10.5220/0006577903730384

In Proceedings of the 9th International Joint Conference on Computational Intelligence (IJCCI 2017), pages 373-384

ISBN: 978-989-758-274-5

proposed taxonomy of termination criteria is used to

establish formal guidelines and later has been consid-

ered as a basis for MATLAB toolbox implementation

to provide a framework for analysis and evaluation of

existing and new approaches.

The studies presented thus far provide evidence

that selecting an appropriate termination criterion is

a challenging task. Obviously, the new termination

criteria are not necessarily ﬁt to the basic three cat-

egories introduced by (Jain et al., 2001). Also, the

main contributions of the work done by Wagner have

no direct focus on providing an overview on the cur-

rent existing approaches (Wagner et al., 2011). In

this survey, we conducted a systematic research to

identify the most commonly used termination criteria

and grouped them into seven categories by extending

the basic categories proposed by (Jain et al., 2001).

Furthermore, we applied the proposed taxonomy of

termination criteria to discuss more sophisticated ap-

proaches which are proposed so far.

The survey is organized in the following way. In

next section, the study method used for writing the

state-of-the-art is explained. Later, termination cri-

teria are categorized based on different aspects and

discussed in details in section 3. For each category,

an overview on commonly used termination criteria

is presented. In section 4, pros and cons of using each

category of termination criteria are discussed in de-

tails. Section 5 concludes the key important notes in-

vestigated in the survey and propose a road-map for

future research directions.

2 THE STUDY METHOD

The study method we used to review termination

criteria is based on the systematic mapping studies

and literature reviews suggested by(Kitchenham and

Charters, 2007) and (Petersen et al., 2008). As it is

guided in (Petersen et al., 2008) we performed ﬁve

main steps in order to ﬁnd the most relevant works

targeting termination criteria in EA. These are: iden-

tifying the research questions, conduct search, screen-

ing of the papers, key-wording, data extraction and

mapping process. In the following, we discuss each

process step brieﬂy.

2.1 Research Questions

The main goals of doing research on termination cri-

teria are reﬂected by numbers of research questions.

In our case, we try to answer the main question: How

can we organize an structure based on the existing

contributions of termination criteria in EAs? More

research questions are raised to answer the main re-

search question in different aspects.

• What are the most frequently investigated termi-

nation criteria and what are their main character-

istics?

• How can we categorize them?

• What are the most frequently applied termination

criteria in evolutionary algorithms and how they

have been evolved over time?

2.2 Conduct Search

Two search strategies are suggested for performing a

systematic research, manual and automatic (Kitchen-

ham and Charters, 2007) and (Petersen et al., 2008).

We chose to only use manual search to ﬁnd the papers

in the most relevant journals and proceedings of the

related conferences in ﬁeld of Evolutionary Compu-

tations (ECs) and Evolutionary Algorithms (EAs).

2.3 Screening of Papers

All of the papers found in section 2.2 are analyzed ac-

cording to several selection criteria such as: index of

the conference or journal, number of citations, long

or short papers, technical or commercial papers and

whether they are part of a book or conference pro-

ceedings. In this case, we included the materials from

book chapters, the papers with high number of cita-

tions and the technical papers published by high index

conferences and journals in the ﬁeld of EC and EA.

Screening process continues by two iterations: study-

ing the abstracts and conclusion and a more detailed

review by reading the introduction and methodology

sections of the papers.

2.4 Keywording

In order to search for the right papers, we had to con-

sider most relevant keywords. Three categories of

keywords are used for this purpose. The ﬁrst cate-

gory is related to the keywords used to describe the

word termination such as termination criterion, stop-

ping criterion and convergence. The second category

refers to performance evaluation of an EA such as

performance indicator and progress indicator. The

third category refers to the problem domain in which

we are looking for termination criteria. They are

mainly evolutionary computation, computational in-

telligent, evolutionary algorithm, multi-objective op-

timization algorithms, multi-objective evolutionary

algorithms and genetic algorithms.

2.5 Data Extraction and Mapping

Process

Data extraction is the last process of the study method

to capture the relevant information from the selected

papers to address the main research questions from

section 2.1. In this step, the bibliography of the se-

lected paper, the contribution, the methodology and

application of the termination criteria are extracted.

Later, extracted data is used to maintain the catego-

rization of termination criteria.

3 TERMINATION CRITERIA IN

EVOLUTIONARY

ALGORITHMS

It is worthwhile to note that the concept of conver-

gence of an EA is different from termination criterion

even though they overlap each other. (Bhandari et al.,

2012) emphasized that the proof of convergence of an

algorithm to an optimal solution is very important as

it guarantees the utility of an algorithm to reach to

the optimal solution in inﬁnite iterations. Different

works have proved the convergence of genetic algo-

rithm to an optimal solution after running for inﬁnite

iterations (Rudolph, 1994), (Suzuki, 1995), (Bhandari

et al., 1996), and (Murthy et al., 1998).

More concisely, once the convergence of an algo-

rithm is assured, then the focus turns into the determi-

nation of stopping criteria of the algorithm. For that

reason, some of the termination criteria are deﬁned

using convergence phenomena. In this case, termina-

tion criterion is fulﬁlled if the algorithm is converged

to an optimal or near-optimal solution(s). The close-

ness of the found solution and the optimal solution

depends on the accuracy that the user desires. More

details are provided for each termination criterion fur-

ther in this section.

Thus far, one of the major challenges in the imple-

mentation of genetic algorithm is to deﬁne a proper

termination criterion or criteria to stop the algorithm

while no a-prior information regarding the objective

function is provided (Bhandari et al., 2012).

In the following of this section, a number of termi-

nation criteria are presented and classiﬁed into seven

main categories. For each criterion, description, main

properties and a termination condition is given. For

simplicity, notations and terminologies are introduced

stepwise where needed together with the respective

references.

3.1 Direct Termination Criteria

Direct termination criteria refer to the class of termi-

nation criteria which stop the algorithm if a prede-

ﬁned condition is satisﬁed without considering any

underlying data from the evolutionary search process

(Jain et al., 2001).

Maximal Time Budget. The maximal time budget

criterion is fulﬁlled if the given time budget is con-

sumed. The maximal time budget can be measured as

an absolute time or the CPU-time (Jain et al., 2001).

In this case, the algorithm runs for a predetermined

execution time and returns the ﬁnal solution.

Maximal Number of Generations. The maximal

number of generations/iterations criterion is fulﬁlled

if the algorithm has been running for a given max-

imum number of generations/iterations (Jain et al.,

2001) and (Bhandari et al., 2012).

Maximal Number of Objective Function Evalua-

tions. In (Jain et al., 2001), the maximal number of

objective function evaluations is deﬁned in the same

way similar to 3.1 where the algorithm stops after

reaching to a speciﬁc number of objective function

evaluations.

Hitting a Bound. The hitting a bound termination

criterion is fulﬁlled if the best value for the objective

function is obtained for a given bound for an objective

function (Jain et al., 2001). In this case, the found

solution is supposed to be close enough or equal to

the known global optimum (Hansen and Kern, 2004),

(Zhong et al., 2004), (Tsai et al., 2004) and (Ong

et al., 2006).

K-iterations. The K-iterations termination criterion

is fulﬁlled if there is no improvement in the best ﬁt-

ness values through a K number of consecutive iter-

ations. The user has to select a proper value for K

by assuming that it is impossible to obtain better re-

sult after K consecutive iterations (Leung and Wang,

2001) and (Bhandari et al., 2012).

3.2 Derived Termination Criteria

Contrary to direct termination criteria, derived termi-

nation criteria calculate auxiliary values using under-

lying data obtained from the current generation of the

evolutionary search process (Jain et al., 2001). They

have also been used as a measure of the state of con-

vergence (Jain et al., 2001).

Running Mean. The running mean termination cri-

terion is fulﬁlled if the difference between the best ob-

jective value of the current generation and the average

of the best objective values of the last t

last

generations

is equal to or less than a given threshold ε ≥ 0 (Jain

et al., 2001).

Standard Deviation. The standard deviation termi-

nation criterion is fulﬁlled if the standard deviation of

all objective values of the current generation is equal

to or less than a given threshold ε ≥ 0 (Jain et al.,

2001).

Best-Worst. The best-worst termination criterion is

fulﬁlled if the difference between the best and the

worst objective value of the current generation is

equal to or less than a given threshold ε ≥ 0 (Jain

et al., 2001).

Phi. The Phi termination criterion is fulﬁlled if the

quotient of the best objective value and the mean of all

objective values of the current generation is equal to

or greater than a given threshold 1 - ε with 1  ε ≥ 0

(Jain et al., 2001).

Kappa. The Kappa termination criterion is fulﬁlled

if the quotient of sum of all normalized distances be-

tween all individuals of the current population and

max

−µ

is equal to or less than a given thresh-

old ε ≥ 0 where µ is the number of individuals known

as population size (Jain et al., 2001).

POP-Var. The POP-Var termination criterion is ful-

ﬁlled if the variance of ﬁtness values of all the individ-

uals in the current population is equal to or less than

a given threshold 1  ε ≥ 0 (Bhandari et al., 2012).

ε-Variance. The ε-Variance termination criterion is

an extended version of a termination criterion similar

to the K-iterations and POP-Var termination criteria

(Bhandari et al., 2012). This criterion considers the

concept of elitism by preserving the ﬁttest individu-

als over the generations. The ε-Variance termination

criterion is fulﬁlled if the variance of the best solu-

tions over generations is equal to or less than a given

threshold ε with 1  ε ≥ 0 (Bhandari et al., 2012).

3.3 Cluster-based Termination Criteria

Cluster-based termination criteria use clustering tech-

niques to examine the distribution of individuals in

the search space at a given generation. Individuals

usually form clusters in the search space after a few

generations of stagnation (Jain et al., 2001). The

search process is terminated when the clusters show

the convergence of the evolutionary search. In this

case, the ﬁttest individuals are concentrated in few

small regions of the search space.

ClusTerm. The ClusTerm termination criterion is

fulﬁlled if the change of the average of the aggregate

size of elitist clusters averaged over the last t

last

gener-

ations is equal to or less than a given threshold ε with

1  ε ≥ 0 where t

last

is the maximal number of the

last generations to be considered (Jain et al., 2001).

3.4 Operator-based Termination

Criteria

In EA, ﬁne-tuning of parameters is normally done in

an empirical way, and usually has a signiﬁcant im-

pact on their performance (Zapotecas Mart

ınez et al.,

2011).

Over the past two decades, there has been an in-

creasing amount of literature on emphasizing the ef-

fects of genetic operators on the stopping time. One

of the ﬁrst systematic study was reported by Murthy

et al. in 1998. The authors introduced a new termi-

nation criterion known as ε-Optimal Stopping Time.

Based on this research, investigations are necessary

to judge theoretically the effect of selection, muta-

tion and crossover operators on the stopping time.

Even though the obtained stopping times are valid for

Elitism GA (EGA) with selection, crossover and mu-

tation operators (Murthy et al., 1998). In this work,

behavior of GA is studied with different values for the

probability of mutation by calculating the hamming

distance between the found solution and the optimal

solution. The ε-Optimal Stopping Time termination

criterion fulﬁlls when the distance between the found

solution and the optimal solution is equal to or less

than a given threshold ε. Nevertheless, the concept of

elitism helps the algorithm not to suffer from ﬁnding

the sub-optimal solutions like ε-Variance termination

criterion discussed earlier in section 3.2.

Later in 2000, Greenhalgh and Marshall discussed

about the convergence properties for genetic algo-

rithms by looking at the effect of mutation on con-

vergence (Greenhalgh and Marshall, 2000). They

showed that by running the genetic algorithm for a

sufﬁciently long time, convergence to a global opti-

mum with any speciﬁed level of conﬁdence, is guar-

anteed. Experimental results showed that the algo-

rithm is able to obtain an upper bound for the number

of iterations necessary to ensure convergence, which

improved the previous results.

More attention has focused on improving the evo-

lutionary operators which go further than modify-

ing their genetics to use the estimation of distribu-

tion algorithms in generating the offspring (Lee and

Yao, 2004), (Lozano et al., 2006) and (Hedar et al.,

2007). A recent study involved domain speciﬁc oper-

ators to enhance state-of-the-art MOEAs (Ghoreishi

et al., 2015). This work presents an evolutionary

algorithm CONTROLEUM-GA that applies domain

speciﬁc variables and operators to solve a real dy-

namic MOEA for a greenhouse climate control prob-

lem. By that, the domain speciﬁc operators only en-

code existing knowledge about the environment. In

CONTROLEUM-GA, domain speciﬁc operators are

combined with two other direct termination criteria

such Maximal Time budget and Maximal Number of

Generations to terminate the search process. Exper-

imental results shows improvements in convergence

time without compromising the quality of the ﬁnal so-

lutions compared to other state-of-art algorithms.

3.5 Performance Indicator Termination

Criteria

The criterion will be a local termination criterion if it

is calculated by using properties of the solutions be-

long to the current generation, which we discussed

earlier. Local criteria require the prior knowledge of

optimal solutions to some extent, which may not be

always available. On the other hand, global crite-

ria are those that compute the progress of evolution

through numbers of consecutive generations (Wagner

et al., 2011). Based on the deﬁnition, short-term char-

acteristics of the evolution can be drawn from local

termination criteria, whereas global termination cri-

teria show the long-term nature of the evolution. To

design a better termination criteria, it is preferable to

apply a global criteria that will not only terminate the

evolution at the appropriate time, but also guarantee

the quality of the solutions (Bhuvana and Aravindan,

2016b).

To measure the performance of the MOEAs, a

standard procedure introduced by Coello using per-

formance indicators (Coello et al., 2006). The

most commonly used performance indicators are Hy-

pervolume, Generational Distance, Inverted Genera-

tional Distance, Spacing, Additive-ε Indicator, Con-

tribution or Set Coverage Metric, R1-Indicator, R2-

Indicator, R3-Indicator and Maximum Pareto Front

Error (Zitzler et al., 2003) and (Coello et al., 2006).

Performance indicators use the concept of Pareto op-

timality to evaluate the performance of MOEA; the

same measures can be used to check the convergence

of the population. In this case, when the population

is converged, evolution can be terminated. In the fol-

lowing, two categories of performance indicator ter-

mination criteria are discussed.

3.5.1 Single Performance Indicator Termination

Criteria

In general, to stop the evolutionary search process,

it is needed to predeﬁne a desired value for a per-

formance indicator (Wagner et al., 2011). Termina-

tion criterion is fulﬁlled when the predeﬁned value is

triggered. In this section, the performance indicators

which have been used as termination criteria are dis-

cussed in a simpliﬁed way.

Hypervolume Metric. Hypervolume metric calcu-

lates the volume of the approximation set at genera-

tion t with respect to nadir point or the reference set

(Coello et al., 2006). Hypervolume metric is one of

the most widely used performance indicator. Hyper-

volume has been used as one of the metrics to de-

termine the stopping generation in (Trautmann et al.,

2009), (Guerrero et al., 2009) and (Guerrero et al.,

2010).

Additive Epsilon Indicator. Additive epsilon indi-

cator is known as one of quality indicators to mea-

sure the performance MOEAs. This indicator uses the

concept of domination to calculate the additive min-

imum distance in which the solution set covers ev-

ery solution towards the reference set (Zitzler et al.,

2003). (Trautmann et al., 2009), (Guerrero et al.,

2009) and (Guerrero et al., 2010) used additive ep-

silon indicator as termination criteria in their respec-

tive works.

Mutual Domination Rate Metric. Set coverage

metric or contribution percentage gives a percentage

with respect to the number of solutions from the ap-

proximation set at generation t which belong to the

reference set (Zitzler et al., 2000). Mutual domina-

tion rate metric is a customized version of set cov-

erage metric to show the number of non-dominated

solutions of generation t which can be dominated by

at least one solution belongs to non-dominated solu-

tions of generation t −1. Mutual domination rate met-

ric is used as a termination criteria in a work done by

(Guerrero et al., 2009)

3.5.2 Multiple Performance Indicator

Termination Criteria

Regarding to the characteristics of performance in-

dicators, the quality of approximation set cannot

be evaluated using only one performance indicator.

Therefore, it is recommended to apply multiple per-

formance indicators to measure the convergence of a

MOEA in terms of diversity and coverage (Hadka and

Reed, 2012) and (Ghoreishi et al., 2015).

Historically, the idea of using performance indica-

tors over the run of MOEA was ﬁrst presented in 2002

in a joint work by Deb and Jain (Deb and Jain, 2002).

They applied two performance indicators to compute

the convergence and the diversity of the approxima-

tion set at generation t.

Later, in another work performed by Trautmann

et al., an ofﬂine convergence analysis has been pre-

sented (Trautmann et al., 2008). Three performance

indicators generational distance, spread and hypervol-

ume are calculated in each generation. Then Kol-

mogorow Smirnov test is applied on the performance

indicators of the current and past ﬁve consecutive gen-

erations. The termination criterion is fulﬁlled if the

distribution of indicators value has no change, other-

wise the evolution is allowed to continue (Trautmann

et al., 2008).

3.6 Progress Indicator Termination

Criteria

As the previous sections describe, termination of

MOEA is often decided based on heuristic stopping

criteria, such as the maximum number of evalua-

tions or a desired value of a performance indicator.

Whereas the termination criteria are suitable for de-

ﬁned benchmark problems, where the optimal indi-

cator value is known, their applicability to the real-

world problems is still questionable (Wagner et al.,

2011). The challenge raises in cases where the eval-

uation budget or the desired indicator level is inap-

propriately speciﬁed, so the MOEA can either waste

computational resources or can be stopped although

the approximation still shows a signiﬁcant improve-

ment. For that reason, heuristic stopping criteria

for the online detection of the generation, where the

expected improvement in the approximation quality

does not justify the costs of additional evaluations,

provide an important contribution to the efﬁciency of

MOEA (Wagner et al., 2011).

Performance indicators described in section 3.6

are known as unary performance indicators computed

by comparing the approximation set at generation t

with the reference set. Unary performance indicators

except spacing, require the knowledge of known op-

timal solutions which is not always available. In or-

der to use performance indicators independent of the

reference set, a customized version of the indicator

is calculated considering two consecutive generations

t and t − 1. These indicators are known as binary

performance indicators or progress indicators which

measure the performance improvement of MOEA.

In recent years, research on sophisticated heuris-

tic Online Stopping Criteria (OSC) has became exten-

sively popular (Mart

ı et al., 2009), (Trautmann et al.,

2009), (Wagner and Trautmann, 2010), (Goel and

Stander, 2010) and (Wagner et al., 2009). OSC com-

pute the progression of single or multiple Progress In-

dicators (PI) during the run of the MOEA. At conver-

gence time, the expected improvement for considered

indicators seems to be lower than a speciﬁed threshold

and the MOEA is terminated in order to avoid need-

less computations. OSC are supposed to detect that

further improvements are unlikely, or are expected to

be too small even if no formal convergence is obtained

(Wagner et al., 2011).

In general, the procedure of OSC can be divided

into two main steps: 1. The progress improvement of

the MOEA is evaluated using progress indicators. 2.

Given a predeﬁned threshold and the value obtained

from PIs, a decision about termination of the MOEA

is made (Wagner et al., 2011). In the following, single

PI termination criteria and aggregated PI termination

criteria are presented in more details.

3.6.1 Single PI Termination Criteria

Single performance Indicator termination criteria re-

fer to a class of termination criteria in which one of

the binary performance indicators are used for calcu-

lation of termination time. In the following, single

progress indicator termination criteria are described

in details.

Hypervolume Metric. Customized version of hy-

pervolume metric as a binary performance indica-

tor is calculated by using non-dominated solution set

obtained at two consecutive generations t and t − 1

(Guerrero et al., 2009).

Additive Epsilon Metric. Similar to hypervolume

metric, Additive Epsilon Metric can be customized to

work upon solution sets of two consecutive genera-

tions, t and t −1. It has been used as termination cri-

terion in two works done by (Guerrero et al., 2009)

and (Trautmann et al., 2009).

Mutual Domination Rate Metric. This metric is

derived from set coverage metric which gives either 1

or 0 to check the number of non-dominated solutions

at generation t which dominate the non-dominated

solutions at generation t − 1 (Guerrero et al., 2009).

MDR equals to 1 indicates that the entire approxi-

mation set at generation t is better than its predeces-

sor. For MDR equals to 0, no substantial progress has

been achieved. MDR<0 indicates a deterioration of

the current population.

Dominance-based Quality of Pareto Front. Bui et

al. introduced the dominance-based quality of Pareto

front metric (DQP) for an approximation set at gener-

ation t (Bui et al., 2009). For each solution in the ap-

proximation set at generation t, the ratio of dominat-

ing individuals in the neighborhood of this solution is

calculated by performing Monte Carlo sampling sim-

ulation with 500 evaluations. DQP equals to 0 indi-

cates that no improved solutions can be found in the

neighborhood of the current solutions in the approxi-

mation set at generation t.

Consolidation Ratio. The Consolidation Ratio

(CR) is introduced by (Goel and Stander, 2010) as

a dominance-based convergence metric. To calculate

this metric, an external archive of all non-dominated

solutions found during the run of the MOEA is

needed. Given the archive, CR is deﬁned as the rela-

tive amount of the archive members in generation t

mem

which are still contained in the archive of the current

generation t.

3.6.2 Aggregated PI Termination Criteria

In single PI termination criteria, the value of the PI

calculated for generation t and used directly to de-

cide on termination time. However, a single PI evalu-

ation usually cannot provide enough information for a

robust conclusion. Because of the non-deterministic

nature of EAs, it can be advantageous to deﬁne an

evidence gathering process (EGP) to incorporate dif-

ferent values for PI (Wagner et al., 2011). EGP is

designed to store the calculations of more than one

PI for previous generations. Descriptive statistics

justiﬁes the need for aggregating different PIs ie.,

the standard deviation of PI values can evaluate the

variability within the PI values(Rudenko and Schoe-

nauer, 2004), (Wagner et al., 2009) and (Wagner and

Trautmann, 2010). In most known aggregated PI ap-

proaches, MOEA terminates when the current value

of the EGP triggers a threshold or a given probabil-

ity limit (Rudenko and Schoenauer, 2004), (Bui et al.,

2009) and (Goel and Stander, 2010).

In the following of this section, the most known

and common approaches for aggregated PI are pre-

sented in a chronological order. For simplicity, we

describe the approaches by presenting the structure of

EGP and the ﬁnal decision making for termination of

the MOEA without the mathematical representation

of each approach.

Running Metrics. In a joint work done by Deb and

Jain performance indicators are used to evaluate con-

vergence and diversity of the approximation set at

generation t known as convergence metric (CM) and

diversity metric (DVM). By this, all objective vectors

of the approximation set at generation t are projected

onto a dimensional hyperplane presented as discrete

grid cells (Deb and Jain, 2002). The convergence

metric (CM) calculates the average of the smallest

normalized euclidean distance from each individual

in the approximation set at generation t to a known

reference set. Whereas DVM tracks the number of at-

tained grid cells and computes the distribution by as-

signing different scores for predeﬁned neighborhood

patterns. In this approach, EGP is a visual investiga-

tion of the progression of CM and DVM done by user

which leads to the ﬁnal decision for termination of the

MOEA (Deb and Jain, 2002).

Stability Measure. According to the work done by

Rudenko and Schoenauer in 2004, the stagnation of

the maximum crowding distance (maxCD) within the

approximation set at generation t can be used as an in-

dicator to detect convergence of NSGA-II algorithm

(Rudenko and Schoenauer, 2004). After each gener-

ation, maxCD and standard deviation (STD) over ap-

proximation set is computed. In this approach, EGP

stores STD of the values of maxCD. The stability

measure termination criterion is fulﬁlled once SDT

falls below a user-deﬁned threshold ε.

MBGM Termination Criterion. MBGM Termina-

tion Criterion (according to the authors’ last names)

uses combination of the mutual domination rate

(MDR) and a simpliﬁed version of Kalman ﬁlter

(Mart

ı et al., 2007). For each generation, MDR in-

dicator is applied to the approximation set obtained

from two last consecutive generations t and t − 1.

EGP stores all MDR values. After that Kalman ﬁl-

ter is applied on EGP and the corresponding esti-

mated error is calculated. MBGM Termination Cri-

terion is fulﬁlled when the conﬁdence interval of

the a-posteriori estimation falls below the predeﬁned

threshold ε (Mart

ı et al., 2009).

Classic On-line Convergence Detection. In 2009,

Naujoks and Trautmann presented a new approach

called On-line Convergence Detection (OCD) to solve

a multi-objective optimization in an aerodynamic ap-

plication. This approach is known as classic OCD

because it is considered as original version of OCD

for further studies. In OCD approach, PI is incorpo-

rated three performance indicators hypervolume, R2-

Indicator and additive ε-indicator (Naujoks and Traut-

mann, 2009), (Wagner et al., 2009) and (Wagner and

Trautmann, 2010). At each generation, the approx-

imation set at generation t is considered as the ref-

erence set to update PI values stored in EGP. Later

for each PI, different variance tests, corresponding p-

values and standard errors are computed and stored in

EGP. Moreover, after standardizing the values stored

in EGP individually, a least-squares ﬁt of a linear

model with slope parameter β is performed. Finally,

the termination decision is made when the p-values of

two consecutive generations are below the conﬁdence

level α for one of the variance tests where α = 0.05.

ODC-Hypervolume. Later in 2010, Wagner and

Trautmann proposed a new version of classic ODC

for Multi-objective selection based on dominated hy-

pervolume to create PI. Since the hypervolume is a

unary indicator, only the absolute values for hyper-

volume have to be stored in EGP. Compared to ODC,

the complexity of OCD-Hypervolume is reduced by

concentrating on the variance test for one speciﬁc PI.

The termination decision is similar to the classic OCD

approach.

Least Squares Stopping Criterion. In 2010, Guer-

rero et al. presented a light version of classic ODC

which simpliﬁes PI computation, EGP, and termina-

tion criterion. Similar to classic OCD, generation t

as the reference set is considered to update PI val-

ues and EGP stores a regression analysis performed

on PI values. In contrast, in Least Squares Stopping

Criterion (LSSC), only one PI is considered while the

variance tests stored in EGP to make the termination

decision are omitted (Guerrero et al., 2010). In addi-

tion, in LSSC, PI values are not standardized in or-

der to be able to observe the expected improvement

by means of the slope β. Consequently, the analy-

ses performed in OCD and LSSC are different. The

termination decision is made when β falls below the

predeﬁned threshold ε. Interested audiences can read

more about the details of implementation of classic

ODC and LSSC in (Naujoks and Trautmann, 2009)

and (Guerrero et al., 2010), respectively.

Non-dominance-based Convergence Metric. An-

other work is done in 2010 by Goel and Standerto

present a non-dominance-based on-line termina-

tion criterion for MOEAs. This approach uses a

dominance-based PI based on an external archive of

non-dominated solutions which is updated in each

generation. In addition to that, utility is deﬁned as

a parameter to compute the difference in value of CR

between the approximation set at generations t and

the external archive of non-dominated solutions. In

this approach, EGP is designed as a moving aver-

age as utility U

to increase the robustness of the ap-

proach. The termination decision is made when the

utility falls below an adaptively computed threshold

called ε

adaptive

3.7 Termination Criteria in Hybrid

MOEA

Research on termination criteria in MOEAs has high-

lighted several approaches to identify the termina-

tion of evolution process. However previous stud-

ies have not dealt with hybrid MOEAs and it is not

certain whether the existing termination criteria are

suitable for hybrid MOEAs and whether they are ad-

equate enough to identify convergence in such al-

gorithms (Bhuvana and Aravindan, 2016b). Previ-

ous research has established that incorporating addi-

tional knowledge during search process can improve

the performance of evolutionary algorithms. The ad-

ditional knowledge is gained by applying a local re-

ﬁnement procedure in the evolution process (Bhuvana

and Aravindan, 2016b). The main advantage of hy-

brid MOEAs is to incorporate local reﬁnement proce-

dures to prevent premature convergence of the search

process (El-mihoub et al., 2006).

In a worked done by Bhuvana and Aravindan in

2016, a hybrid MOEA has been proposed in which the

termination happens respect to the maximum num-

ber of functional evaluations known as MAPLSAW

(Bhuvana and Aravindan, 2016a). The experimental

results show that ﬁxing an upper bound for number of

objective function evaluations or number of genera-

tions will not be an appropriate termination condition

in all cases (Bhuvana and Aravindan, 2016a). More-

over, another recent work is done by Bhuvana and Ar-

avindan in 2016, to develop a new termination crite-

ria for hybrid MOEAs. They proposed a termination

scheme including ﬁve termination criteria which two

of them are new termination criteria proposed for a

hybrid MOEA called MAPLS-AW (Bhuvana and Ar-

avindan, 2016b).

The termination scheme relies on three perfor-

mance indicators, hypervolume, mutual domination

metric and additive-ε indicator together with the con-

cept of maintaining the elites in the population over

generations. In the proposed schema, two new ter-

mination criteria derived based on the features of

MAPLS-AW to detect the convergence of the popula-

tion, named Preferential Local Search (PLS) and Elite

Average Fitness (EAF). In each generation, PLS and

EAF are computed and compared with the values ob-

tained in the previous generations. In the following

of this section, PLS and EAF are explained in more

details.

Preferential Local Search. In hybrid MOEA pre-

sented by In MAPLS-AW implementation presented

in Bhuvana and Aravindan, the local reﬁnement pro-

cedure relies on elite solutions identiﬁed at every gen-

eration (Bhuvana and Aravindan, 2016b). Over gen-

erations, PLS deepens on the elites iteratively until

such elite solutions become locally optimal. Locally

optimized elites are considered as the measure of con-

vergence to calculate the stopping criteria, SC

PLS

In order to calculate SC

PLS

, a local termination cri-

terion, L

PLS

is needed. L

PLS

is computed using the

current optimized elites and the total number of elites

at every generation and compared with its previous

generation to derive SC

PLS

. Termination decision is

made when TC

PLS

equal to 0 which means the value

of L

PLS

is not changed over last two generations, t and

t − 1. SC

PLS

equal to −1 shows that no improvement

have been observed from generation t to generation

t − 1. SC

PLS

is equal to 1 indicates that the popula-

tion is still evolving and the current generation has

progress toward optimal solutions (Bhuvana and Ar-

avindan, 2016b).

Elite Average Fitness. Similar to PLS, Elite Aver-

age Fitness stopping criterion SC

EAF

is proposed in

the same work by Bhuvana and Aravindan for the hy-

brid algorithm, MAPLS-AW (Bhuvana and Aravin-

dan, 2016b). EAF incorporates elite’s average ﬁtness

value at every generation using a local measure called

EAF

. Calculating I

EAF

is a challenging task in a

multi-objective problem when multiple objectives are

not combined into a single objective. For dealing with

this challenge, MAPLS-AW has proposed an adaptive

weight (AW) objectives by considering their positions

in the objective space. I

EAF

of two consecutive gener-

ations t and t − 1 are used to compute SC

EAF

. Termi-

nation decision is made if no improvement have been

observed by SC

EAF

4 DISCUSSION

Recalling the research questions in section 2.1, the

most frequent termination criteria and their main

characteristics explained in details in seven different

categories. We distinguished the categories by con-

sidering functionality and applicability of the termi-

nation criteria and the way they have been evolved

over time. Applying all aforementioned termination

criteria in different test cases is out of the scope of this

work. But nevertheless, we discuss on strengths and

weaknesses of termination criterion belong to each

category in the following section.

In overall, even though direct termination crite-

ria are easy to implement but determining the ﬁxed

value for termination condition is again a challenge.

Moreover, ﬁnding optimal values requires a good and

a-priori knowledge about the behavior of the GA for

the speciﬁc problem and the global optimal solution

which are not always available (Bhandari et al., 2012).

In 2001, Jain et al. evaluated direct and derived

termination criteria from two main aspects, reliabil-

ity and performance (Jain et al., 2001). They con-

cluded that all direct termination criteria except the

hitting a bound, are reliable criteria which guarantee

termination of EA within ﬁnite number of iterations

or ﬁnite time. One major drawback for the hitting a

bound termination criterion is that it does not termi-

nate the algorithm if it converges to a sub-optimal so-

lution because in this case the objective value of the

sub-optimal solution is always worst than the required

bound (Jain et al., 2001). In order to guarantee ter-

mination, it is suggested to apply the hitting a bound

condition in combination of any other direct termina-

tion criteria (Jain et al., 2001). Other ﬁnding of the

same work shows selecting suitable values for the pa-

rameters of the corresponding criteria has direct in-

ﬂuence on performance of direct termination criteria

which can be found by trial-and-error method while

setting a default value is not recommended (Jain et al.,

2001).

The reliability and performance of derived termi-

nation criteria has been analyzed and evaluated in

a research done by (Jain et al., 2001). Among all,

running means termination criterion is a reliable one

which assures the termination of the algorithm after

ﬁnite number of iterations. In contrast, four termina-

tion criteria presented in sections 3.2-3.2 can only ter-

minate the search process if the objective values of all

individuals at the current generation are sufﬁciently

similar. This means that termination is not guaranteed

and can be postponed if few outliers are existed in the

population. To avoid prevention of termination, it is

suggested to combine these criteria with other direct

termination criteria discussed earlier in section 3.1.

Bhandari et al. proposed ε-Variance termination

criterion which is similar to both K-Variance and

POP-Var termination criteria in some extents (Bhan-

dari et al., 2012). The main difference between the

ε-Variance and K-Variance termination criteria is that

the user does not need to predeﬁne the number of it-

erations in ε-variance termination criterion. By pre-

deﬁning K, user assumes that it is impossible to ob-

tain a better solution after K consecutive iterations.

But the main challenge for using K-Variance termi-

nation criterion is that, it is proven that given a ﬁnite

value for K, there is always a positive probability of

obtaining K consecutive equal sub-optimal solutions

(Bhandari et al., 2012).

On the other hand, in POP-Var termination crite-

rion, the variance is calculated using the individuals

in the current population while in ε-Variance termina-

tion criterion, the variance of the best solutions over

generations is used as termination condition. How-

ever, ε-Variance still has two main limitations. The

ﬁrst one is the necessity to have a prior knowledge

about the global optimal solution which makes it in-

applicable when the optimal solution is unknown or

not available. And the second is to choose the optimal

value for ε. A challenge to apply this criterion is that

no automatic way of choosing the value ε is suggested

(Bhandari et al., 2012). Experimental results show

that the value of ε varies in different problems with

the same size of search space to obtain global optimal

solution (Bhandari et al., 2012). Smaller number of ε

close to 0 leads to higher accuracy. This means that

the choice of ε is dependent on the level of accuracy

that the user desires.

In a similar way, in CLusTerm termination crite-

rion, selecting an appropriate value for ε is impor-

tant to obtain a proper termination. The parameter

ε controls the duration of stagnation until termina-

tion. Hence, smaller values of ε prolongs stagnation

phase without termination and vice versa. In prac-

tical use, it is recommended to employ ClusTerm or

running mean termination criterion presented in 3.2,

together with one of the four approaches provided in

sections 3.1 and 3.2-3.2, to prevent needless compu-

tations (Jain et al., 2001).

It is worthy to mention that operator-based ap-

proaches may not focus directly on proposing a new

termination criterion but they are designed mostly to

accelerate the convergence of the EA in an automatic

way to avoid unnecessary computations (Ghoreishi

et al., 2015). Operator-based approaches are varies

based on the implementation speciﬁcations. For in-

stance, as discussed earlier, CONTROLEUM-GA ap-

plies domain speciﬁc genetic operators to avoid gen-

erating new solutions that are not valid based on the

existing knowledge about the problem and environ-

mental measurements (Ghoreishi et al., 2015). The

strength point about domain speciﬁc approach is the

automatic way of accelerating the search process.

Other approaches such as ε-Optimal Stopping

Time termination criterion, has still two main draw-

backs. The ﬁrst one is a prior knowledge requires

about the optimal solution which is not always avail-

able for many real world problems. And the second

one is that it is proven that with existing operations

in GA, whatever value decided for the number of it-

erations, there is always a positive probability of not

obtaining an optimal solution if ε is not small enough

(Bhandari et al., 2012).

The main purpose of applying performance indi-

cators is to measure the convergence and diversity

of the solutions without compromising the quality.

Unary performance indicators (both single and multi-

ple termination criteria) except spacing are computed

by comparing the approximation set at generation t

towards the reference set which requires the knowl-

edge of known optimal solutions. But on the con-

trary, binary performance indicators are independent

of the reference set. Binary performance indicators

are known as progress indicators (PIs). They are cus-

tomized in a way that they consider two consecutive

generations t and t − 1 instead of using the approxi-

mation set and the reference set for the measurements.

Among all single PIs, DQP is the only PI that re-

quires many additional evaluations of the objective

functions for computing the ratio of dominating so-

lutions in the neighborhood of a speciﬁc solution.

Consequently, DQP is a very expensive, but power-

ful measure and it is only applicable if many addi-

tional evaluations of the objective functions can be

performed. In contrast, MDR is capable of measur-

ing the progress of the optimization with almost no

additional computational cost by reusing the compu-

tations done to apply Pareto-optimality. Therefore it

is suitable for solving large-scale or many-objective

problems with large population sizes. In addition, CR

is also not an efﬁcient metric for problems with many

objectives, due to the large archive size.

In single PI termination criteria, only the last value

of PI is used to decide on termination time which can-

not always provide enough information for a robust

conclusion due to the non-deterministic nature of EAs

(Wagner et al., 2011). For this reason, it can be ben-

eﬁcial to incorporate different values of PI for previ-

ous generations using an evidence gathering process

(EGP) in aggregated PI approaches.

One of the important issues in designing aggre-

gated PI approaches is concerning about using the sta-

tistical tests and conﬁdence intervals which make the

termination decision robust compared to monitoring

random variables overtime. In addition, parameter-

ization of aggregated PI approaches plays important

role in obtaining the desired results with respect to the

trade-off between runtime and approximation quality

(Wagner et al., 2011). One of the limitations with ag-

gregated PI approaches is that there is no clear guide-

line for parameterization and visualized analysis (Deb

and Jain, 2002) and (Bui et al., 2009). One of the

only guideline is proposed by Wagner and Trautmann

for parameterization of ODC approach based on sta-

tistical analysis of experimental results (Wagner and

Trautmann, 2010).

Discussion on termination criteria in hybrid ap-

proaches is a challenging task because most studies

in the ﬁeld of EA have focused on termination crite-

ria in single MOEA. Nevertheless, the experimental

results indicate that PLS and EAF together with three

performance indicators are promising approaches for

predicting the convergence of population by consider-

ing additional knowledge about the number of locally

optimized elites in the population and their average

ﬁtness (Bhuvana and Aravindan, 2016b).

5 CONCLUSION

The study was conducted in the form of a survey to

provide a concise categorization of prominent termi-

nation criteria in EA. This research will serve as a

base for future studies targeting termination criteria

in EA and makes several noteworthy contributions to

the current existing literature. This work extends our

knowledge about the existing termination criteria and

can be used to improve the development of such ap-

proaches in terms of convergence speed, computation

cost and assessment of the quality of approximation

set with or without known reference set. In addi-

tion to principal theoretical implication of this study,

strengths and weaknesses of the approaches are high-

lighted and discussed.

A key strength of the present study was to high-

light the role of combination of direct termination cri-

teria and threshold-based termination criteria in or-

der to guarantee the convergence of EA in a reliable

manner. Whilst the main focus of this study was not

the evaluation of aforementioned termination crite-

ria, rigorous and comprehensive set of experiments is

still needed. Therefore, it would be interesting to as-

sess the effects of selecting different termination crite-

ria and evaluating their impacts on different problem

domains with various number of objectives and also

their applicability to use in real-world applications.

It is worthy to note that, the termination criteria in

this survey are the most commonly used ones and the

reader can investigate about the details of implemen-

tation and mathematical formulations in the respec-

tive references.

REFERENCES

Bhandari, D., Murthy, C. A., and Pal, S. K. (1996). Genetic

algorithms with elitist model and its convergence. In-

ternational Journal of Pattern Recognition and Artiﬁ-

cial Intelligence, 10(6):731–747.

Bhandari, D., Murthy, C. A., and Pal, S. K. (2012). Variance

as a stopping criterion for genetic algorithms with eli-

tist model. Fundam. Inf., 120(02):145–164.

Bhuvana, J. and Aravindan, C. (2016a). Memetic algorithm

with preferential local search using adaptive weights

for multi-objective optimization problems. Soft Com-

put., 20(4):1365–1388.

Bhuvana, J. and Aravindan, C. (2016b). Stopping criteria

for mapls-aw, a hybrid multi-objective evolutionary

algorithm. Soft Comput., 20(6):2409–2432.

Bui, L. T., Wesolkowski, S., Bender, A., Abbass, H. A., and

Barlow, M. (2009). A dominance-based stability mea-

sure for multi-objective evolutionary algorithms. In

Proceedings of the Eleventh Conference on Congress

on Evolutionary Computation, CEC’09, pages 749–

756, Piscataway, NJ, USA. IEEE Press.

Coello, C. A. C., Lamont, G. B., and Veldhuizen, D. A. V.

(2006). Evolutionary Algorithms for Solving Multi-

Objective Problems (Genetic and Evolutionary Com-

putation). Springer-Verlag New York, Inc., Secaucus,

NJ, USA.

Deb, K. and Jain, S. (2002). Running performance metrics

for evolutionary multi-objective optimization. Tech-

nical report.

El-mihoub, T. A., Hopgood, A. A., Nolle, L., and Battersby,

A. (2006). Hybrid genetic algorithms: A review.

Ghoreishi, S. N., Sørensen, J. C., and Jørgensen, B. N.

(2015). Enhancing state-of-the-art multi-objective op-

timization algorithms by applying domain speciﬁc op-

erators. Proceeding of 2015 IEEE Symposium Series

on Computational Intelligence.

Goel, T. and Stander, N. (2010). A non-dominance-based

online stopping criterion for multi-objective evolu-

tionary algorithms. International Journal for Numer-

ical Methods in Engineering, 84(6):661–684.

Greenhalgh, D. and Marshall, S. (2000). Convergence

criteria for genetic algorithms. SIAM J. Comput.,

30(1):269–282.

Guerrero, J. L., Garcia, J., Marti, L., Molina, J. M., and

Berlanga, A. (2009). A stopping criterion based on

kalman estimation techniques with several progress

indicators. In Proceedings of the 11th Annual Con-

ference on Genetic and Evolutionary Computation,

GECCO ’09, pages 587–594, New York, NY, USA.

ACM.

Guerrero, J. L., Mart

ı, L., Berlanga, A., Garc

ıa, J., and

Molina, J. M. (2010). Introducing a robust and efﬁ-

cient stopping criterion for multi-objective evolution-

ary algorithms. IEEE Congress on Evolutionary Com-

putation.

Hadka, D. and Reed, P. (2012). Diagnostic assessment of

search controls and failure modes in many-objective

evolutionary optimization. Evol. Comput., 20(3):423–

452.

Hansen, N. and Kern, S. (2004). Evaluating the cma evolu-

tion strategy on multimodal test functions. Proceed-

ings of Eighth International Conference on Parallel

Problem Solving from Nature, PPSN VIII, 3242:282–

291.

Hedar, A.-R., Ong, B., and Fukushima, M. (2007). Genetic

algorithms with automatic accelerated termination.

Jain, B., Pohlheim, H., and Wegener, J. (2001). On termi-

nation criteria of evolutionary algorithms. pages 768–

768.

Kitchenham, B. and Charters, S. (2007). Guidelines for per-

forming systematic literature reviews in software en-

gineering.

Konar, A. (2005). Computational Intelligence: Principles,

Techniques and Applications. Springer-Verlag New

York, Inc., Secaucus, NJ, USA.

Lee, C.-Y. and Yao, X. (2004). Evolutionary programming

using mutations based on the levy probability distri-

bution. Trans. Evol. Comp, 8(1):1–13.

Leung, Y. and Wang, Y. (2001). An orthogonal genetic al-

gorithm with quantization for global numerical opti-

mization. Trans. Evol. Comp, 5(1):41–53.

Lozano, J. A., Larra

naga, P., Inza, I. n., and Bengoetxea, E.

(2006). Towards a New Evolutionary Computation:

Advances on Estimation of Distribution Algorithms

(Studies in Fuzziness and Soft Computing). Springer-

Verlag New York, Inc., Secaucus, NJ, USA.

Mart

ı, L., Garc

ıa, J., Berlanga, A., and Molina, J. M.

(2007). A cumulative evidential stopping criterion for

multiobjective optimization evolutionary algorithms.

In Proceedings of the 9th Annual Conference Com-

panion on Genetic and Evolutionary Computation,

GECCO ’07, pages 2835–2842, New York, NY, USA.

ACM.

Mart

ı, L., Garc

ıa, J., Berlanga, A., and Molina, J. M.

(2009). An approach to stopping criteria for multi-

objective optimization evolutionary algorithms: The

mgbm criterion. IEEE Congress on Evolutionary

Computation.

Murthy, C., Bhandari, D., and Pal, S. K. (1998). e-optimal

stopping time for genetic algorithms. Fundam. In-

form., 35(1-4):91–111.

Naujoks, B. and Trautmann, H. (2009). Online conver-

gence detection for multiobjective aerodynamic appli-

cations. In Proceedings of the Eleventh Conference

on Congress on Evolutionary Computation, CEC’09,

pages 332–339, Piscataway, NJ, USA. IEEE Press.

Ong, Y., Lim, M., Zhu, N., and Wong, K. (2006). Classi-

ﬁcation of adaptive memetic algorithms: A compara-

tive study. IEEE Transactions on Systems, Man, and

Cybernetics-Part B, 36(1):141–152.

Petersen, K., Feldt, R., Mujtaba, S., and Mattsson, M.

(2008). Systematic mapping studies in software en-

gineering. In Proceedings of the 12th International

Conference on Evaluation and Assessment in Software

Engineering, EASE’08, pages 68–77, Swindon, UK.

BCS Learning & Development Ltd.

Rudenko, O. and Schoenauer, M. (2004). A steady perfor-

mance stopping criterion for pareto-based evolution-

ary algorithms. The 6th International Multi-Objective

Programming and Goal Programming Conference.

Rudolph, G. (1994). Convergence analysis of canonical ge-

netic algorithms. Trans. Neur. Netw., 5(1):96–101.

Suzuki, J. (1995). A markov chain analysis on simple ge-

netic algorithms. IEEE Transactions on Systems, Man,

and Cybernetics, 25(4):655–659.

Trautmann, H., Ligges, U., Mehnen, J., and Preuss, M.

(2008). A convergence criterion for multiobjective

evolutionary algorithms based on systematic statis-

tical testing. In Proceedings of the 10th Interna-

tional Conference on Parallel Problem Solving from

Nature: PPSN X, pages 825–836, Berlin, Heidelberg.

Springer-Verlag.

Trautmann, H., Wagner, T., Naujoks, B., Preuss, M., and

Mehnen, J. (2009). Statistical methods for conver-

gence detection of multi-objective evolutionary algo-

rithms. Evol. Comput., 17(4):493–509.

Tsai, J.-T., Liu, T.-K., and Chou, J.-H. (2004). Hybrid

taguchi-genetic algorithm for global numerical opti-

mization. IEEE Transactions on Evolutionary Com-

putation, 8(4):365–377.

Wagner, T. and Trautmann, H. (2010). Online convergence

detection for evolutionary multi-objective algorithms

revisited. IEEE Congress on Evolutionary Computa-

tion.

Wagner, T., Trautmann, H., and Mart

ı, L. (2011). A

taxonomy of online stopping criteria for multi-

objective evolutionary algorithms. In Proceedings

of the 6th International Conference on Evolutionary

Multi-criterion Optimization, EMO’11, pages 16–30,

Berlin, Heidelberg. Springer-Verlag.

Wagner, T., Trautmann, H., and Naujoks, B. (2009).

Ocd: Online convergence detection for evolutionary

multi-objective algorithms based on statistical testing.

In Proceedings of the 5th International Conference

on Evolutionary Multi-Criterion Optimization, EMO

’09, pages 198–215, Berlin, Heidelberg. Springer-

Verlag.

Zapotecas Mart

ınez, S., Y

nez Oropeza, E. G., and

Coello Coello, C. A. (2011). Self-adaptation tech-

niques applied to multi-objective evolutionary algo-

rithms. In Proceedings of the 5th International

Conference on Learning and Intelligent Optimiza-

tion, LION’05, pages 567–581, Berlin, Heidelberg.

Springer-Verlag.

Zhong, W., Liu, J., Xue, M., and Jiao, L. (2004). A mul-

tiagent genetic algorithm for global numerical opti-

mization. IEEE Transactions on Systems, Man, and

Cybernetics-Part B, 34(2):1128–1141.

Zitzler, E., Deb, K., and Thiele, L. (2000). Comparison

of multiobjective evolutionary algorithms: Empirical

results. Evol. Comput., 8(2):173–195.

Zitzler, E., Thiele, L., Laumanns, M., Fonseca, C. M., and

da Fonseca, V. G. (2003). Performance assessment

of multiobjective optimizers: An analysis and review.

Trans. Evol. Comp, 7(2):117–132.