Ontology Metrics as a Service (OMaaS)

Achim Reiz

, Henrik Dibowski

, Kurt Sandkuhl

and Birger Lantow

Chair of Business Information Systems, Rostock University, 18059 Rostock, Germany

Corporate Research, Robert Bosch GmbH, 71272 Renningen, Germany

Keywords: Ontology Metrics, Ontology Evaluation, OWL, Cloud, Bosch.

Abstract: The use of automatically calculated metrics for the evaluation of ontologies can provide impartial support for

knowledge engineers. However, even though the use of ontological representations is unabated – in opposite

expected to rise through the increasing use of AI technologies – most ontology evaluation tools today are no

longer available or outdated. At the same time, due to the growth of the computational cloud, service-driven

architectures are on the rise, and enterprises tend to prefer to consume services in a platform- or software as

a service model. In this paper, we argue that the change of the IT-landscape also requires a change in how we

offer and consume ontology metrics. This hypothesis is backed by an industrial use-case of Robert Bosch

GmbH and their application of ontologies, as well as their need and requirements for ontology evaluation. It

motivated the extension of the tool OntoMetrics with a REST-interface, offering a public endpoint for

ontology metrics on the Internet.

1 INTRODUCTION

Ontologies, widely regarded as „a formal

specification of a shared conceptualization” (Borst,

1997), are not new to the scientific community. The

importance of an ontological knowledge exchange

format for AI was outlined by Matthew Ginsberg as

soon as 1991 (Ginsberg, 1991). With the introduction

of the Semantic Web in 2001, where ontologies

intermediate a meaning between the representation

and data layer (Berners-Lee et al., 2001) and

recommendation of the W3C for RDF, RDF(S), and

OWL a little later, this technology got attention from

a broader set of audience and is now being used by

various research disciplines or industries like

medicine, biology, geography, astronomy, defense,

automotive or aerospace (Grau et al., 2008, p. 309).

The possible applications for computational

ontologies are diverse. Examples are, among others,

the knowledge management system of the food and

agriculture organization of the united nations, where

all reports of the fisheries and aquaculture department

are integrated into a network of ontologies for

https://orcid.org/0000-0003-1446-9670

https://orcid.org/0000-0002-9672-2387

https://orcid.org/0000-0002-7431-8412

https://orcid.org/0000-0003-0800-7939

enhanced information sharing, at the same time

enlarging the open linked data repository (Caracciolo

et al., 2012) or the storing of genomes, their

molecular functions, locations or biological processes

(Kelso et al., 2010). Ontologies are even used in the

security-sensitive military branch, allowing queries

like “which helicopters are used only for attacking”

(Mishra & Jain, 2019).

However, with the beginning of the introduction

of ontologies in various branches, a problem arose.

The classical engineering processes, like the

“Ontology Engineering Methodology” (Sure et al.,

2009) or the distributed NeOn Methodology (Suárez-

Figueroa, Gómez-Pérez, & Fernández-López, 2012),

heavily build on the involvement of knowledge

engineers for the engineering of the ontology itself.

But not only are ontology engineers a scarce resource

(Suárez-Figueroa, Gómez-Pérez, Motta, & Gangemi,

2012), often the development of large ontologies is

highly decentralized and requires a participative, self-

organizing structure of domain experts.

Methodologies like DILIGENT (Pinto et al., 2009)

propose a framework for this changing landscape by

putting more responsibilities on the domain experts

250

Reiz, A., Dibowski, H., Sandkuhl, K. and Lantow, B.

Ontology Metrics as a Service (OMaaS).

DOI: 10.5220/0010144002500257

In Proceedings of the 12th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2020) - Volume 2: KEOD, pages 250-257

ISBN: 978-989-758-474-9

and end-users, withdrawing the knowledge engineer

to a controlling and advisory role.

This puts the validation of the quality of an

ontology at an even more central position. Many

quality criteria like accuracy, adaptability, clarity,

completeness, computational efficiency, conciseness,

consistency, or organizational fitness (Vrandečić,

2009) are highly desirable. But the path to the perfect

ontology or even the measurement of such is difficult

to achieve. Even the quality ratings of experienced

knowledge engineers tend to be highly subjective

without a formal guideline (Tankeleviǧiene &

Damaševičius, 2009, p. 135). The application of

automatically calculated metrics within the

evaluation approach can guide the rating and,

therefore, the quality assessment of ontological

representations and offers an objective foundation for

the interpretation of fitness for a given task. Metrics

can support the evaluation of ontological

representations. The propagation of metrics was the

main driver for the development of the first GUI tool

“OntoMetrics” (Lantow, 2016). To widen the usage

of ontology metrics, we extended the OntoMetrics

web tool with a RESTful-API. We want to enable a

smooth integration of metrics for future applications

by allowing users to consume ontology metrics in a

convenient, service-driven model.

The rest of the paper is structured as follows:

Section 2 illustrates the landscape of cloud-driven

ontology services. In Section 3, we present the current

usage scenarios and challenges regarding the use of

ontologies and ontology evaluation of Robert Bosch

GmbH. Section 4 is concerned with the capabilities

and architecture of OntoMetrics, followed by the

conclusion and an outlook on future research with the

newly developed API.

2 RELATED WORK

Over the past years, various ontology evaluation tools

were developed, ranging from manual procedures to

semiautomated and fully automated approaches. This

related work section focusses on the latter – software

that allow the calculation of metrics without human

involvement.

The developed automatic evaluation tools focus

on different aspects like coupling, structure, coverage

or correctness and differ in their integration

approaches, ranging from web- or application-based

standalone tools (examples are OntoMetrics (Lantow,

2016) or OntoQA (Tartir et al., 2005)) to such

http://sele.inf.um.es/ontology-metrics/

integrated with larger development suites (examples

are Swoop (Kalyanpur et al., 2006) or Protégé

(Musen, 2015)). However, a lot of the software

developed is no longer available or heavily outdated.

To the best of our knowledge, there is currently just

one tool available with similar functionality,

developed by the Universidad de Murcia

. While their

OQuaRE based approach allows the direct calculation

of statistical relevant correlations and clusters within

the datapoints, it has less functionality regarding the

variety of different measurements and also does not

provide class-specific metrics (Franco et al., 2020).

Table 1 gives some examples without raising a claim

to completeness.

Table 1: Ontology evaluation tools.

Software Cate

Availabilit

Swoop

(Kalyanpur et al.,

2006

)

Web Ontology-

Editor

Last Update

from 2007

OntoQA

(

Tartir et al., 2005

)

Standalone

Evaluation Tool

Last update from

2010

OntoMetrics

(

Lantow, 2016

)

Web Evaluation

Tool

Available and

Usable

Protégé

(

Musen, 2015

)

Ontology

Edito

Available and

Usable

ODEval

(Corcho et al.,

2004)

Standalone

Evaluation Tool

Not available

OntoKBEval

(Qing Lu & Volker

Haarslev, 2006

)

Standalone

Evaluation Tool

Not available

OntoKeeper

(

Amith et al., 2019

)

Web-Tool

Not publicly

available

OQuaRE Metrics

Calculation

(Franco et al.,

2020)

Web-Tool &

Rest Service

Available and

Usable

This sets up the motivation for our research: We

strongly believe that ontologies will play an important

role in the future, providing a structured

representation of knowledge for the integration

within artificial intelligence. At the same time, most

ontology evaluation tools are no longer available or

outdated. Rostock University still offers a functioning

web-based evaluation tool, which led to the

collaboration with the Robert Bosch GmbH. Having

insights in the usage scenarios of ontologies and the

need for evaluating and measuring ontologies in such

a large corporation, we argue that as much as the

usage of ontologies has changed from expert

knowledge representation to a versatile tool in the

Ontology Metrics as a Service (OMaaS)

251

toolbox of AI, the evaluation tools have to change as

well.

Cloud services more and more replace or

complement former on-premise applications in

almost all areas of IT, including semantic

technologies, as they offer highly sophisticated

technologies and infrastructure as a service, thus

allowing the usage of these artifacts without the need

for investments in expertise and hardware. There are

already adaptions of ontological services for the

cloud: Flahive et al. proposed extracting and

replacing methodologies for tailoring sub-ontologies

out of large domain ontologies using cloud services

(Flahive et al., 2013). WebProtégé is a popular cloud-

based editor for collaborative ontology modeling

(Tudorache et al., 2013). Poveda-Villalón et al.

developed the OntOlogy Pitfall Scanner (OOPS) for

detecting anomalies within ontologies. The service is

available through a GUI and a RESTful interface

(Poveda-Villalón et al., 2014). VoCol is a git-based

version control system, including validation,

querying, analytics, visualization, and documentation

(Halilaj et al., 2016). The OQuaRE tool by (Franco et

al., 2020) provides a web- and REST-interface for 19

different metrics. However, currently, there is no

cloud-based ontology metrics tool with the same

measurement capabilities as OntoMetric available.

3 INDUSTRIAL USE-CASE OF

ROBERT BOSCH GmbH

This section motivates the need for applying metrics

as enhanced quality assurance mechanisms for

ontologies and knowledge graphs, from the viewpoint

of the industrial partner Robert Bosch GmbH. Bosch

partners with University Rostock and benefits from

using OntoMetrics in multiple ways and industrial use

cases, as described in the following.

Bosch is a large automotive and industrial

company that is heavily investing in becoming one of

the world-leading artificial intelligence (AI)

companies. The Bosch Center for Artificial

Intelligence (BCAI), which was founded in early

2017 and had continuously been growing since then,

employs over a hundred AI experts and is the

spearhead of the AI research and enablement ongoing

in Bosch. During the past years, the dominant AI

technologies that have been developed and deployed

are data-driven, subsymbolic approaches, in

particular machine learning (ML) and deep learning

(DL). This is changing, and symbolic approaches, in

particular ontologies and knowledge graphs, are

gaining strong momentum in many enterprises,

including Bosch. The demand for knowledge

engineering in industrial use cases is growing rapidly,

but yet, we are rather at the beginning of a potentially

long rise in knowledge graphs. Gartner identified

knowledge graphs to be amongst the most important

innovation triggers for artificial intelligence (Brant et

al., 2019), as well as amongst the most promising

emerging technologies of the year 2019 (Smith &

Burke, 2019), with an estimation of 5 to 10 years of

continuous growth.

After long years of propagating semantic

technologies, we finally see many lead architects and

decision-makers understand the power of semantic

technologies, which opens the doors for new

industrial use cases in various domains. Furthermore,

a huge potential of domain and application areas is

still unexplored, yet to be discovered. We believe that

there will be an even stronger demand and impact of

ontologies and knowledge graphs in the industry than

we have seen with ML and DL, as it does not depend

on the availability of large amounts of data for

training the algorithms, and as it can be applied for

representing knowledge of practically any domain.

But also Semantic AI or Explainable AI, currently

emerging as an interdisciplinary novel AI approach

from the combination of ML and knowledge

representation, promises a big potential for years to

come (Lecue, 2020).

The universality of ontologies and knowledge

graphs make them candidates for being applied in

potentially all business sectors in Bosch, ranging

from Mobility Solutions, over Industrial Technology,

Consumer Goods, Energy and Building Technology,

up to subsidiary companies like Healthcare Solutions,

and business units such as Bosch Connected Industry.

We see many different application areas for standard

semantic technologies and challenges that they can

help resolve, such as a semantic specification of our

data, systems, and factories; the interoperable

integration and interpretation of heterogeneous data;

the formalization and application of expert

knowledge, making products and services truly smart;

device interoperability in multi-vendor and cross-

domain settings; formal validation of systems and

products. There is already a series of success stories

from past and ongoing projects in Bosch, for

example, the application of knowledge graphs for

searching enterprise data lakes (Schmid et al., 2019),

the semantic search and reuse of autonomous driving

data (Henson et al., 2019), formal model checking

ontologies for the verification of autonomous driving

(Kaleeswaran et al., 2019), the semantic

interoperability and integration of manufacturing

KEOD 2020 - 12th International Conference on Knowledge Engineering and Ontology Development

252

(Mehdi et al., 2019) and IoT data (Svetashova et al.,

2019), semantic model extensibility (Svetashova,

2018), the computerized engineering of building

automation systems using knowledge graphs as

integrated semantic information models (Dibowski &

Massa Gray, 2020), and the application of semantic

technologies for improved complaint management in

commercial buildings (Massa Gray et al., 2020).

A challenge, however, is the skills and expertise

required for developing good ontologies and

knowledge graphs. There is a large number of

qualified engineers, software architects and

developers skilled in conventional database

techniques and programming languages available, but

ontology experts or developers with a background in

semantic technologies are low in number. The

continuously growing demand in enterprises cannot

be met by the job market, and the few ontology

experts in an enterprise cannot support all ongoing

modeling tasks. That is why the development of

models and ontologies often needs to be driven by

domain experts or developers, who lack that

expertise, with only little guidance from ontology

experts. Nonetheless, to make their work efficient and

successful, they need mature software tools that

support them in defining good ontology models on

their own. This is where ontology metrics play an

important role, as they can help to assess, ensure, and

improve the quality of ontologies. Ontology metrics

also help in determining the best ontologies from

multiple competing ones, which has become a

frequent task, as more and more ontologies have been

developed, published, and shared on the Internet.

We made a comprehensive study of available

ontology metric approaches and solutions, both in

scientific publications as well as in software tools.

Since the metrics need to be available within a short

time and without much human interaction (i.e., at no

or little cost), we were particularly searching for

metrics that can be computed automatically. That

strongly limited down the available solutions to a few

ontology tools that calculate and show some basic

ontology metrics such as counts of classes, instances,

different types of axioms, etc., e.g., Protégé and

TopBraid Composer. However, that rather is an

assessment in terms of quantity than quality.

Academic solutions, on the contrary, either propose

metrics that need time-intensive, costly assessment by

human experts, or that are not available (anymore) for

download or as online services. Fortunately, in

OntoMetrics, we found a sophisticated online service

that computes a comprehensive list of various

ontology metrics at the push of a button. OntoMetrics

excels all other solutions we could find in its

comprehensiveness of metrics it can calculate. It is

under active development, well documented,

platform- and tool-independent, and the makers have

been interested and supportive in enabling us to use

OntoMetrics within the corporation.

Over the past years, the IT landscape has

undergone a tremendous change, as IT infrastructure

and software architectures have moved from desktop

applications and in-house server farms to web-based

UIs, cloud-based infrastructure, and in-cloud data

lakes. That saves cost at the enterprise side and

improves flexibility, as it enables on-demand up- and

downscaling of cloud resources (storage capacity,

processing power). Semantic technologies fit very

well into that new landscape since IRIs,

dereferencability, and the OWL import mechanism

allow for distributing and storing linked information

in a decentralized way. The trend in the ontology

domain goes into the same direction, as several

ontology tools are now available as browser-based,

collaborative development environments, e.g.,

WebProtégé, TopBraid EDG, and VoCol.

The recent enhancement of OntoMetrics from a

web-based tool, where a user specifies or uploads the

ontology to be assessed in a web UI and afterward

sees the results, to a standalone REST service is a

perfect fit for the IT landscape of today and for Bosch.

It enables other applications and services to call and

connect with the offered REST APIs and enables

them to trigger the metrics calculation and consume

the results whenever needed. As a REST service

running inside the Bosch network, we have

accomplished an enterprise-wide availability of

OntoMetrics, accessible, and usable from within the

whole enterprise, without requiring local

installations. It can be consumed from various tools,

teams, and projects, independent of the operating

system, programming language, or hardware being

used.

4 THE OntoMetrics TOOLKIT

Lantow initially introduced the OntoMetrics platform

in 2016 (Lantow, 2016) as a Java EE based web

application. This section gives at first an overview of

the GUI-accessible metrics engine and later presents

the newly developed REST-interface.

Ontology Metrics as a Service (OMaaS)

253

Figure 1: Accessing the OntoMetrics API through the Firefox extension “RESTED”.

4.1 OntoMetrics Web Tool

The tool OntoMetrics is publicly available at the

server of Rostock University

. Using a simple user

interface, one can paste ontology content in a textbox,

a link to a web source, or upload an ontology file. The

content must conform to RDF, RDF(S) or OWL and

can apply a serialization via N3, turtle or RDF/XML.

Overall, the metrics engine allows the calculation of

81 distinctive measurements. These metrics are

mainly based on publications by (Gangemi et al.,

2005) and (Tartir et al., 2005). Seventy-two of the

measurements concern the ontology as a whole. They

are grouped into nine categories, shown in the table

below. Additionally, OntoMetrics calculates 9 class

metrics for every class in the ontology, rating e.g., the

connectivity or importance of a concept.

Table 2: Categories of the OntoMetrics evaluation.

Cate

Meanin

Base Metrics

Simple metrics measuring the

number of ontology elements

E.g. count of axioms, individuals, or

object

roperty links

Schema Metrics Analyses the design of the

ontology

E.g. attribute richness, inheritance

richness, or class-axioms ratio

Graph Metrics Analyses the taxonomy tree of the

ontology

E.g. depth, absolute root cardinality,

or average number of

aths

Knowledgebase

Metrics

Analysis of individuals and

ontology population

E.g. average population, class

ichness, or number of leaf classes

Class Metrics Evaluation of single classes

E.g. class readability, class inheritance

richness, or class instances

https://ontometrics.informatik.uni-rostock.de

A full description of every metric, including the

calculation formula, is available in the OntoMetrics

wiki

. Before the analysis, it is possible to limit the

evaluation to the categories that are needed.

Especially the calculation of the Class Metrics

requires a large amount of computational power.

Their exclusion can significantly reduce the required

calculation time.

The output prints all metrics with the

corresponding category and sub-category they belong

to. For further analysis in a data analysis software, it

is possible to download the metrics in an XML-file.

4.2 OntoMetrics Rest-API

Cloud software, especially in a software as a service

architecture, has the potential to lower the entrance

barrier for new technologies. Even though the

previous sections motivated the relevancy of

ontologies, their integration in various use-cases as

well as the need for their evaluation, simple metric

applications are scarce. The new OntoMetrics API

indents to fill that gap by providing an easy to use

programming interface for calculating ontology

metrics.

To evaluate an ontology from a web source, one

can use a

GET

request on the endpoint

http://opi.informatik.uni-

rostock.de/api

with the parameter ?url

pointing to the ontological resource. The full request

should look like the following example for a query to

the friend of a friend ontology:

http://opi.informatik.uni-

rostock.de/api?url=http://xmlns.com/

foaf/spec/20140114.rdf

https://ontometrics.informatik.uni-rostock.de/wiki/

KEOD 2020 - 12th International Conference on Knowledge Engineering and Ontology Development

254

For assessing a local ontology that is not available at

a web resource, it is possible to use a

POST request on

the same endpoint

http://opi.informatik.uni-

rostock.de/api

The ontology is then expected in the request body.

The response is delivered in an XML serialization

using the same top categories and terminology used

in the GUI-version and presented in Table 2. The

underlying computational engine is the same for the

API, as well as for the web-application. Like in the

GUI-version, the inclusion of class metrics

significantly increases the response time and is

therefore disabled by default. If the class-metrics are

required, a header key named

classmetrics with

the value

true enables the calculation.

If the target ontology is not consistent with an

RDF syntax or one of its extensions like OWL or

RDF(S), the service throws an HTTP 400 error and

returns an XML consisting of further information

regarding possible causes.

By default, all assessed ontologies are stored

internally at Rostock University for further research

purposes. This behavior can be disabled by adding the

parameter

save : false to the header. The

response header contains the parameter

saved :

true if the ontology is stored on the server or saved

: false

if otherwise.

Fig 1 displays an example query to the metrics

service utilizing the firefox extension “RESTED”,

with a declined database storage agreement and

activated class metrics.

5 CONCLUSION AND OUTLOOK

Even though the semiotic based ontologies are a long-

existing, mature technology compared to other AI

approaches, they have not become obsolete, but in the

opposite, experience much attention with an

increasing amount of use cases. However, as ontology

engineers are a scarce resource, these ontologies are

often developed not by knowledge engineering

experts, but developers within a given domain. This

boldens the need for automatic evaluation of the

ontological artifacts to ensure a high level of quality

and comparability between the different knowledge

representations.

The need for a software-based automatic

evaluation has been established for several years. And

https://gitlab.com/ and https://github.com/

over this time, various tools have been developed.

Nevertheless, most of these tools are not usable

anymore due to outdated data formats, deprecated

dependencies, or just unavailable online resources.

OntoMetrics is one of the few tools that are still

available for the upcoming rise in computational

ontologies. With the newly developed API, we hope

to lower the entrance barrier for the use of ontology

metrics. The cloud-based approach shall allow an

easy integration of the measurements in semantic

driven applications, thus on the one hand, further

spread the use of ontology metrics, on the other hand,

allow us to collect valuable data for further research.

The application scenarios of Robert Bosch GmbH

highlight the need for further development and more

research in the field of ontology evaluation. The

extension of the OntoMetrics tool with an API is the

first step to make ontology metrics more usable. The

next planned feature is the integration of historical,

evolutional data through the connection of a git-based

repository service like GitLab or Github

. It is argued

that through the analysis of the evolution of the

measured values, statements regarding the current

maturity of the research ontology can be derived.

Also, the comparing of various evolutionary

progressions could allow the inferring of

recommendations for the next modeling steps under

the consideration of their maturity.

These are just two exemplary use-cases for the

possible benefits of the analysis of historical ontology

data. We think that there is a tremendous research

potential in tapping these ontology repositories

regarding explorative quantitative analysis, finding

correlations between metrics, and deriving quality

statements and recommendations.

REFERENCES

Amith, M., Manion, F., Liang, C., Harris, M., Wang, D.,

He, Y., & Tao, C. (2019). Architecture and usability of

OntoKeeper, an ontology evaluation tool. BMC

Medical Informatics and Decision Making, 19(S4), 1–

18. https://doi.org/10.1186/s12911-019-0859-z

Berners-Lee, T., Hendler, J., & Lassila, O. (2001). The

Semantic Web. Scientific American, 284(5), 34–43.

Borst, W. N. (1997). Construction of engineering

ontologies for knowledge sharing and reuse [PhD-

Thesis, Enschede, NL]. WorldCat.

Brant, K., Hare, J., & Sicular, S. (2019, August 25). Hype

Cycle for Artificial Intelligence. Gartner Research (ID:

G00369840). Gartner Research.

Ontology Metrics as a Service (OMaaS)

255

Caracciolo, C., Heguiabehere, J., Gangemi, A.,

Baldassarre, C., Keizer, J., & Taconet, M. (2012).

Knowledge Management at FAO: A Case Study on

Network of Ontologies in Fisheries. In M. C. Suárez-

Figueroa, A. Gómez-Pérez, E. Motta, & A. Gangemi

(Eds.), Ontology Engineering in a Networked World

(Vol. 10, pp. 383–405). Springer. https://doi.org/10.

1007/978-3-642-24794-1_18

Corcho, Ó., Gómez-Pérez, A., González-Cabero, R., &

Suárez-Figueroa, M. C. (2004). Odeval: A Tool for

Evaluating RDF(S), DAML+OIL, and OWL Concept

Taxonomies. In M. Bramer & V. Devedzic (Eds.), IFIP

International Federation for Information Processing:

Vol. 154. Artificial Intelligence Applications and

Innovations: IFIP 18th World Computer Congress

TC12 First International Conference on Artificial

Intelligence Applications and Innovations (AIAI-2004)

22-27 August 2004 Toulouse, France (Vol. 154,

pp. 369–382). Springer. https://doi.org/10.1007/1-

4020-8151-0_32

Dibowski, H., & Massa Gray, F. (2020). Applying

Knowledge Graphs as Integrated Semantic Information

Model for the Computerized Engineering of Building

Automation Systems. In S. Kirrane & A. Ngonga

(Chairs), European Semantic Web Conference (ESWC),

Heraklion, Greece.

Flahive, A., Taniar, D., & Rahayu, W. (2013). Ontology as

a Service (OaaS): extracting and replacing sub-

ontologies on the cloud. Cluster Computing, 16(4),

947–960. https://doi.org/10.1007/s10586-012-0231-x

Franco, M., Vivo, J. M., Quesada-Martínez, M., Duque-

Ramos, A., & Fernández-Breis, J. T. (2020).

Evaluation of ontology structural metrics based on

public repository data. Briefings in Bioinformatics,

21(2), 473–485. https://doi.org/10.1093/bib/bbz009

Gangemi, A., Catena, C., Ciaramita, M., & Lehmann, J.

(2005). A theoretical framework for ontology

evaluation and validation. In P. Bouquet & G.

Tummarello (Eds.), Semantic Web Applications and

Perspectives. CEUR. http://ceur-ws.org/Vol-166/

Ginsberg, M. L. (1991, September). Knowledge

interchange format. The KIF of death. AI Magazine,

12(3), 57–63.

Grau, B. C., Horrocks, I., Motik, B., Parsia, B., Patel-

Schneider, P., & Sattler, U. (2008). OWL 2: The next

step for OWL. Journal of Web Semantics, 6(4), 309–

322. https://doi.org/10.1016/j.websem.2008.05.001

Halilaj, L., Petersen, N., Grangel-González, I., Lange, C.,

Auer, S., Coskun, G., & Lohmann, S. (2016). VoCol:

An Integrated Environment to Support Version-

Controlled Vocabulary Development. In E. Blomqvist,

P. Ciancarini, F. Poggi, & F. Vitali (Eds.), LNCS

sublibrary. SL 7, Artificial intelligence: Vol. 10024,

Proceedings of Knowledge engineering and knowledge

management: 20th International Conference, EKAW

(pp. 303–319). Springer. https://doi.org/10.1007/978-

3-319-49004-5_20

Henson, C [C.], Schmid, S [S.], Tran, T [T.], &

Karatzoglou, A. (2019). Using a knowledge graph of

scenes to enable search of autonomous driving data. In

M. C. Suárez-Figueroa, G. Cheng, A. L. Gentile, C.

Guéret, C. M. Keet, & A. Bernstein (Chairs),

Proceedings of the ISWC 2019 Satellite Tracks,

Auckland, New Zealand. http://ceur-ws.org/Vol-

2456/paper84.pdf

Kaleeswaran, A. P., Nordmann, A., & Mehdi, A. u. (2019).

Towards Integrating Ontologies into Verification for

Autonomous Driving. In M. C. Suárez-Figueroa, G.

Cheng, A. L. Gentile, C. Guéret, C. M. Keet, & A.

Bernstein (Chairs),

Proceedings of the ISWC 2019

Satellite Tracks, Auckland, New Zealand.

Kalyanpur, A., Parsia, B., Sirin, E., Grau, B. C., &

Hendler, J. (2006). Swoop: A Web Ontology Editing

Browser. Journal of Web Semantics, 4(2), 144–153.

https://doi.org/10.1016/j.websem.2005.10.001

Kelso, J., Hoehndorf, R., & Prüfer, K. (2010). Ontologies

in Biology. In R. Poli, M. Healy, & A. Kameas (Eds.),

Theory and Applications of Ontology: Computer

Applications (Vol. 25, pp. 347–371). Springer.

https://doi.org/10.1007/978-90-481-8847-5_15

Lantow, B. (2016). OntoMetrics: Application of on-line

ontology metric calculation. In B. Johansson & F.

Vencovský (Chairs), Joint Proceedings of the BIR -

Workshops and Doctoral Consortium, Prague, Czech

Republic. http://ceur-ws.org/Vol-1684/paper19.pdf

Lecue, F. (2020). On the role of knowledge graphs in

explainable AI. Semantic Web, 11(1), 41–51.

https://doi.org/10.3233/SW-190374

Massa Gray, F., Dibowski, H., Gall, J., & Braun, S. (2020).

Occupant Feedback and Context Awareness: On the

Application of Building Information Modeling and

Semantic Technologies for Improved Complaint

Management in Commercial Buildings. In 25th IEEE

International Conference on Emerging Technologies

and Factory Automation (ETFA), Vienna.

Mehdi, A. u., Kharlamov, E., Stepanova, D., Loesch, F., &

Grangel-González, I. (2019). Towards Semantic

Integration of Bosch Manufacturing Data. In M. C.

Suárez-Figueroa, G. Cheng, A. L. Gentile, C. Guéret,

C. M. Keet, & A. Bernstein (Chairs), Proceedings of the

ISWC 2019 Satellite Tracks, Auckland, New Zealand.

Mishra, S., & Jain, S. (2019). Towards a Semantic

Knowledge Treasure for Military Intelligence. In A.

Abraham, P. Dutta, J. K. Mandal, A. Bhattacharya, &

S. Dutta (Eds.), Advances in Intelligent Systems and

Computing: volume 755. Emerging technologies in

data mining and information security: Proceedings of

IEMIS 2018 (Vol. 755, pp. 835–845). Springer.

https://doi.org/10.1007/978-981-13-1951-8_74

Musen, M. A. (2015). The Protégé Project: A Look Back

and a Look Forward. AI Matters, 1(4), 4–12.

https://doi.org/10.1145/2757001.2757003

Pinto, H. S., Tempich, C., & Staab, S. (2009). Ontology

Engineering and Evolution in a Distributed World Using

DILIGENT. In S. Staab & R. Studer (Eds.), Handbook

on Ontologies (Vol. 51, pp. 153–176). Springer.

https://doi.org/10.1007/978-3-540-92673-3_7

Poveda-Villalón, M., Gómez-Pérez, A., & Suárez-

Figueroa, M. C. (2014). OOPS! (OntOlogy Pitfall

KEOD 2020 - 12th International Conference on Knowledge Engineering and Ontology Development

256

Scanner!). Semantic Web and Information Systems,

10(2), 7–34. https://doi.org/10.4018/ijswis.2014040102

Qing Lu, & Volker Haarslev (2006). OntoKBEval: A

Support Tool for DL-based Evaluation of OWL

Ontologies. In B. C. Grau, P. Hitzler, C. Shankey, & E.

Wallace (Chairs), OWL: Experiences and Directions,

Athens, Georgia (USA).

Schmid, S [Stefan], Henson, C [Cory], & Tran, T [Tuan].

(2019). Using Knowledge Graphs to Search an

Enterprise Data Lake. In P. Hitzler (Ed.), LNCS

sublibrary. SL 3, Information systems and applications,

incl. Internet/Web, and HCI: Vol. 11762. The semantic

web: ESWC 2019 Satellite Events: ESWC 2019 Satellite

Events, Portorož, Slovenia, June 2-6, 2019: revised

selected papers (Vol. 11762, pp. 262–266). Springer.

https://doi.org/10.1007/978-3-030-32327-1_46

Smith, D., & Burke, B. (2019, August 6). Hype Cycle for

Emerging Technologies. Gartner Research (ID:

G00370466).

Suárez-Figueroa, M. C., Gómez-Pérez, A., & Fernández-

López, M. (2012). The NeOn Methodology for

Ontology Engineering. In M. C. Suárez-Figueroa, A.

Gómez-Pérez, E. Motta, & A. Gangemi (Eds.),

Ontology Engineering in a Networked World (Vol. 5,

pp. 9–34). Springer. https://doi.org/10.1007/978-3-

642-24794-1_2

Suárez-Figueroa, M. C., Gómez-Pérez, A., Motta, E., &

Gangemi, A. (2012). Introduction: Ontology

Engineering in a Networked World. In M. C. Suárez-

Figueroa, A. Gómez-Pérez, E. Motta, & A. Gangemi

(Eds.), Ontology Engineering in a Networked World

(pp. 1–6). Springer. https://doi.org/10.1007/978-3-642-

24794-1_1

Sure, Y., Staab, S., & Studer, R. (2009). Ontology

Engineering Methodology. In S. Staab & R. Studer

(Eds.), Handbook on Ontologies (Vol. 13, pp. 135–

152). Springer. https://doi.org/10.1007/978-3-540-

92673-3_6

Svetashova, Y. (2018). Semantic Model Extensibility in

Interoperable IoT Data Marketplaces: Methods, Tools,

Automation Aspects. In S. Kirrane & L. Kagal (Chairs),

Proceedings of the Doctoral Consortium at ISWC 2018,

Monterey, USA.

Svetashova, Y., Schmid, S [Stefan], & Sure-Vetter York

(2019). Semantic Interoperability and Interrater

Agreement in Annotation of IoT Data. In M.

Lefrançois, A. Haller, & K. Janowicz (Chairs), co-

located with 18th International Semantic Web

Conference (ISWC 2019), Auckland, New Zealand.

Tankeleviǧiene, L., & Damaševičius, R. (2009).

Characteristics of domain ontologies for web based

learning and their application for quality evaluation [E-

mokymui(si) skirtos dalykinės srities ontologijos

kokybes charakteristikos ir ju taikymas ontologijos

kokybei vertinti]. Informatics in Education, 8(1), 131–

152. https://doi.org/10.15388/infedu.2009.09

Tartir, S., Arpinar, B., Moore, M., Sheth, A. P., & Aleman-

Meza, B. (2005). OntoQA: Metric-Based Ontology

Quality Analysis. In IEEE Workshop on Knowledge

Acquisition from Distributed, Autonomous,

Semantically Heterogeneous Data and Knowledge

Sources, Houston.

Tudorache, T., Nyulas, C., Noy, N. F., & Musen, M. A.

(2013). Webprotégé: A Collaborative Ontology Editor

and Knowledge Acquisition Tool for the Web.

Semantic Web, 4(1), 89–99.

https://doi.org/10.3233/SW-2012-0057

Vrandečić, D. (2009). Ontology Evaluation. In S. Staab &

R. Studer (Eds.), Handbook on Ontologies (Vol. 284,

pp. 293–313). Springer. https://doi.org/10.1007/978-3-

540-92673-3_13

Ontology Metrics as a Service (OMaaS)

257