EVALUATION OF SEARCH ENGINE OPTIMIZATION

Experiments for a Business Site and a New Evaluation Measure

Julia Maria Schulz, Ralph Kölle, Christa Womser-Hacker and Thomas Mandl

Information Science, University of Hildesheim, Germany

Keywords: Search Engine Optimization, Internet Marketing.

Abstract: This paper reports experiments on search engine optimization (SEO) for a business site. Several search

terms have been optimized for three web search engines. From the business site, 300 pages have been

selected for optimization. In three phases several on- and off-page modifications have been carried out and

the results have been monitored. The results show that search engines do react to modifications and that the

target pages are ranked higher on average. The variance of the improvements is extremely large which

means that there is no guarantee that SEO activities are beneficial for one single page. We suggest a new

evaluation measure for the success of SEO which takes typical Web user behavior into account.

1 INTRODUCTION

Search Engine Optimization (SEO) is practical

information retrieval from the perspective of the

information provider. Web site authors and

providers are interested in attracting many visitors

mostly for economic reasons. They want that their

pages are ranked at top positions in search engine

results for relevant keywords. Relevance is defined

from the perspective of the provider in this case.

Usually, the user is considered as central for the

definition of relevance in information science.

SEO has become a prosperous business branch

for information professionals. Many companies have

been established and many guide books have been

published (e.g. Grappone and Couzin, 2006). There

is much anecdotic evidence on the effects of SEO on

single pages. However, surprisingly little reliable

empirical research is available for the topic. We

conducted an empirical study which showed the

effects of several SEO activities. The analysis

showed that new measures for analyzing the success

and value of SEO activities are necessary.

Search engines are ambivalent toward SEO. On

the one hand, they do not publish their algorithms

which would make SEO a more transparent task.

The search engines argue that such a step would lead

to much spam in the search results. On the other

hand, search engines companies give hints on

potential optimisation potential. In such suggestion

lists, the search engines clearly distinguish between

desired and undesired practices.

This makes SEO an information ethical issue.

Search engines draw the borderline between

tolerated and forbidden SEO activities. The

regulations are not publicly discussed but simply

proposed by the companies. The discovery of the

violation of these regulations may lead to

consequences. This can take the form of removal

from the index or lead to lower ranking positions.

Punishment in such a form is obviously problematic.

They are not transparent, information providers are

not properly notified and no appeal is possible.

Much rather, the violator needs to rely on the mercy

of the search engines.

Meanwhile, the internet is extremely important

for finding information and for being found from the

perspective of information providers. Search became

a basic service. There are even calls for a public

provision of search services (Maurer, 2007). From

the perspective of SEO, it would be preferable to

achieve a more democratic decision process for

regulations than to simply accept the regulation of

the search engine businesses.

2 EXPERIMENTAL SETUP

In a experiment, the effect of typical SEO services

were analyzed systematically. As a target site, a e-

commerce shop of a present vendor was selected

247

Maria Schulz J., Kölle R., Womser-Hacker C. and Mandl T. (2008).

EVALUATION OF SEARCH ENGINE OPTIMIZATION - Experiments for a Business Site and a New Evaluation Measure.

In Proceedings of the Tenth International Conference on Enterprise Information Systems - SAIC, pages 247-250

DOI: 10.5220/0001713602470250

 SciTePress

100

120

Google

(Pages

from

Germany)

Google

(Pages in

German)

Google

(World-

wide

search)

MSN

(Pages

from

Germany)

MSN

(Pages in

German)

MSN

(World-

wide

search)

Yahoo!

(Pages in

German)

Yahoo!

(World-

wide

search)

SearchEngines

normalized Rankingposition

Improvement

Body Meta data Body and Meta data Body-Tag and bold print all

Figure 1: Results of phase 1: average precision.

(

http://www.yamando.net). The presents are from

many different areas which allows the optimization

for many different keywords. Three search engines

with a high market share in the target market were

chosen for the optimization study: Google, MSN and

Yahoo. Phrases of two and three keywords were

identified as queries which should lead the user to

the pages of the company. The ranking positions

before and after the page modifications were

obtained by the SEO tool WebCeo

(www.webceo.com). The tool also determined the

competition for a search term. The competition is

later used for normalization It gives a measure for

the competition of providers for a specific term or a

phrase. The competition is measured as the number

of web pages including the keyword and can be seen

as an indicator of how many people want their

content to be found under this keyword.

No practices considered as spam were applied

during the study in order not to endanger the

business site with removal from the search engine

index.

The optimization experiments presented in this

paper were carried out in three phases. The first two

phases applied on page practices and the third phase

applied the central off page practice link

optimization (Fischer, 2006, Grappone and Couzin,

2006).

The first phase increased the keyword density to

a level of up to 3% and the second up to 5%. Higher

keyword densities are often considered as spam by

the search engines.

The phases included the following

modifications:

• Phase 1

o Increasing the keyword density (freq-

uency) of the terms or phrases to 2-3%

o For 50 pages in the page body

o For 50 pages in the Meta-Tags

(Description and Title)

o Using layout to emphasize the term

(bold print) for another 5 0 pages

o Using all above measures in one page

for another 5 0 pages

• Phase 2

o Increasing the keyword density (fre-

quency) of the terms or phrases to

some 5 %

• Phase 3

o Increasing the number of internal

incoming links to the page by adding

links between product pages

The results for the three phases are given in the

following section.

3 RESULTS

The first phase led to large improvements for some

pages. Figure 1 displays the normalized arithmetic

average of all practices for all search engines. A

considerable improvement in the ranking position of

the target pages can be seen. The improvement

reaches up to 100 positions on average when several

measures are taken simultaneously.

ICEIS 2008 - International Conference on Enterprise Information Systems

248

The variance between pages is quite

considerable as figure 2 shows.

-200

200

400

600

800

1000

1 3 5 7 9 11131517192123252729313335373941434547

Produkt Pages

Improvement in Rankingposition

Google (Pages in German) MSN (Pages in German) Yahoo (Pages in German)

Average Google Average MSN Average Yahoo

330

Figure 2: Results of phase 1: results for pages.

Phase two led to similar results. The large

variance in the improvements between different

pages can be observed again. Increasing the

keyword density beyond 4% is still beneficial at

least for Google.

Applying all measure in one page leads to a larger

improvement than the sum of individual practices.

The cumulative effect is shown in figure 3.

100

120

Google

(Pages

from

Germany)

Google

(Pages in

German)

Google

(World-

wide

search)

MSN

(Pages

from

Germany)

MSN

(Pages in

German)

MSN

(World-

wide

search)

Yahoo!

(Pages in

German)

Yahoo!

(World-

wide

search)

Search Engines

Normalized improvement

of the ranking position

Body Meta data Body and bold print all

Figure 3: Cumulative effects compared to all practices.

Phase 3 also led to considerable improvement. For

Google, there is an improvement in the average

ranking position. The number of incoming links

does not show a clear relation to the improvement

for Google. For seven incoming links there is a

considerable enhancement in the rankings. It seems

that search engines still use the internal link

information. Link based authority measures have

been discussed as a means to judge the quality of

Web pages and this information has been used in

rankings (Baeza-Yates, 2006, Mandl, 2006).

However, because search engine providers do not

publish their algorithms it is never clear to what

extent they apply which factor to the ranking.

100

150

200

250

300

012345678

Number of incoming Links

Improvement in Rankingposition

MSN Germany (Pages from

Germany)

MSN Germany (Pages in German)

MSN Germany (World-wide

search)

Linear (MSN Germany (Pages

from Germany))

Linear (MSN Germany (World-

wide search))

Linear (MSN Germany (Pages in

German))

Figure 4: Results for phase three for MSN and Yahoo.

The other search engines exhibit a linear trend for

the improvement. Figure 4 shows the improvements

with trend lines. The more links point to a page, the

better is the ranking. This is true at least for up to

seven in-links.

It can be seen that especially Google reacts

quickly and very sensitive to SEO activities. Links

pointing to a page are beneficial for the ranking of

pages even if they origin form the same site.

4 NEW EVALUATION MEASURE

The users of search engines do not view very many

results. A study of an Altavista Log found that

97.7% of all users only view the first page of results

in the search engine

This finding has been confirmed in user tests

(Cutrell & Guan, 2007). An improvement of 100

pages in the ranking is not of the same value for the

information provider if the final position is five or

300. Consequently, the target position needs to be

used as a factor in the evaluation of SEO. We

developed a new measure for evaluating the success

of SEO activities which takes this user behaviour

into account.

In the formula for the new success measure, p

and p

are the position of the target page in the

ranking before and after the application of SEO. The

sign function merely extracts the sign of the

difference. It assures that negative success values are

obtained when the target page drops in the ranking.

The success s is obtained b dampening the

EVALUATION OF SEARCH ENGINE OPTIMIZATION - Experiments for a Business Site and a New Evaluation

Measure

249

difference in the ranking with a logarithm and

dividing it by the resulting position.

),min(

)1(log

)(

ppsigns

+−

−=

The effect of this new measure is displayed in

figure 5 for several target positions. We used e as the

basis of the log. Ultimately, the user interest decay

function should be based on empirical evidence on

how many pages are typically viewed in the domain.

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Ranking Improvement (in Postions)

Success

Figure 5: Effect of new measure for six target positions.

5 CONCLUSIONS

SEO provides effective ways to improve the ranking

position of target pages in search engine result

pages. So called on-page improvements like

increasing the frequency of the keyword density

improve the ranking with a large variance between

pages. Increasing the keyword density even further

up to 5% still is beneficial. The modification of the

internal link structure can also greatly improve the

ranking for pages which receive in-links. Obviously,

the effect of activities needs to be monitored over a

longer period of time.

The business success obtained is obviously the

ultimate quality indicator for SEO activities. It can

be measured by the sale or the traffic on a web site.

However, it is difficult to measure and to separate

effects from other advertisement or business

decisions. Consequently, new means of evaluating

the success of SEO needs to be discussed. Such

measure like the one presented here might be

applied to estimate the value and costs of SEO

services in the future.

REFERENCES

Fischer, M., 2006. Website Boosting. Suchmaschinen-

Optimierung, Usability, Webseiten-Marketing. Heidel-

berg: Mitp.

Grappone, J.; Couzin, G. 2006. Search Engine

Optimization: An Hour a Day. Sybex

Maurer, H. 2007 Google - Freund oder Feind? In:

Informatik Spektrum, Vol. 30 (4) pp. 273-278

Mandl, T., 2006. Implementation and Evaluation of a

Quality Based Search Engine. In Proc 17th ACM

Conf. on Hypertext and Hypermedia (HT '06) Odense,

Denmark, Aug. 22nd-25th. ACM Press. pp. 73-84.

Schulz, J.M., 2007. Suchmaschinenoptimierung – Eine

empirische Studie zur Optimierung des Rankings am

Beispiel einer Erlebnisgeschenkefirma. Master Thesis,

International Information Management, University of

Hildesheim.

Baeza-Yates, R., Boldi, P., Castillo, C. 2006. Generalizing

PageRank: damping functions for link-based ranking

algorithms. In: Proceedings of the 29

Annual Intl.

ACM SIGIR Conference on Research and Develop-

ment in Information Retrieval (SIGIR). pp. 308-315.

Cutrell, E., Guan, Z. 2007. What are you looking for? an

eye-tracking study of information usage in web search.

In: Proceedings of the SIGCHI conference on Human

Factors in Computing Systems. pp. 407-416

ICEIS 2008 - International Conference on Enterprise Information Systems

250