Architecture for a Combined Mobile Robot and Human Operator

Transportation Solution for the Hierarchical Life Science Automation

S. Neubert

, T. Roddelkopf

, X. Gu

, B. Göde

, S. Junginger

, N. Stoll

and K. Thurow

Institute of Automation, University of Rostock, Friedrich-Barnewitz-Str. 8, 18119 Rostock, Germany

Center for Life Science Automation (celisca) Friedrich-Barnewitz-Str. 8, 18119 Rostock, Germany

Keywords: Life Science Automation, Laboratory Execution System, Mobile Robot Integration, Dynamical Scheduling,

Human Machine Interaction.

Abstract: Modern laboratories for life sciences often include several different integrated automation systems to

increase throughput and quality, to reduce efforts for human operators and to reduce the costs of processes.

Typically, the planning and monitoring of methods are prepared and executed directly on local computers of

the automation systems. Moreover, a manual replenishing of resources and a manual transfer of samples and

labware between interacting automation systems are required in order to ensure end-to-end operations in a

24/7 mode.

This work describes the architecture and the pilot solution of a hierarchical workflow management system

(HWMS), which integrates distributed automation systems by combined use of mobile robots and human

operators as transportation units. With a graphical process design tool a material flow-oriented diagram can

be created, which describes the correlations of distributed subsystems in a complex workflow. The HWMS

schedules the workflows and controls the execution autonomously dependent on the planned process

diagrams. Two front-end components located on the process control layer simplify the integration and

support the control of the required subsystems. With smart device communications human operators can be

integrated in the workflow for transportation and assistance tasks as a necessary alternative to the robots.

1 INTRODUCTION

The 24/7 operation and the integration of automated

workstations are basic requirements in life science

processes to increase the throughput and the

processing quality and to reduce the effort of

monotonous processing steps and the risk of

potentially hazardous setups for laboratory assistants

(Patel et al., 2014; Lam et al. 2012).

A further reason to invest into automation in life

science laboratories is cost reduction (Cork and

Sugawara, 2002). However, currently only single

devices, automated workstations (e.g. liquid

handlers including peripheral devices) and integrated

automation systems (workstations, which include

one or more local transportation robots) are

commonly found in life science automation, as for

example seen in (Andersen et al., 2012; Liu et al.,

2010; Sutherland et al., 2014). Especially larger

systems consisting of distributed automated devices,

workstations and integrated automation systems –

here called automation systems – often provide

unresolved challenges regarding their complete

automation. The obvious reason is the non-existent

sample transportation control between the

automation systems and a required higher level

control system (complex automation system) for the

synchronization of all sub processes.

For preparation matters such as manual supply of

resources or a manual transport, especially in

systems with frequently changing tasks, the

assistance of a human operator is still essential. A

continuous progress of several running life-science

processes requires an appropriate involvement of the

operators. The operators must always be informed

about all necessary manual steps required to keep

the process chain running.

This article describes a development to increase

the level of automation in life sciences by setting up

a hierarchical control solution combined with mobile

robots and mobile devices in the instrument layer. In

modern laboratory automation systems, these

components enable higher system integration and a

higher degree of effectiveness by reducing the

Neubert, S., Roddelkopf, T., Gu, X., Göde, B., Junginger, S., Stoll, N. and Thurow, K.

Architecture for a Combined Mobile Robot and Human Operator Transportation Solution for the Hierarchical Life Science Automation.

DOI: 10.5220/0006423600410049

In Proceedings of the 14th International Conference on Informatics in Control, Automation and Robotics (ICINCO 2017) - Volume 2, pages 41-49

ISBN: Not Available

waiting time between distributed sub processes. An

additional effect is the transition from automation as

part of a manual chain to manual steps as an integral

part of full automation in life science processes. This

transition is necessary due to the fact that non-

continuous interaction between human and machine

processes builds a bottleneck in the workflow,

especially when processes are executed parallel by

robots and involved humans. Thus, the

interconnection of robot- and human-controlled sub

processes needs to be managed for a continuous

effective workflow.

Personal digital equipment, such as smartphones

and tablet PCs, are often used in laboratories to

support the human operators. They allow to access,

e.g. laboratory information management systems

(LIMS) or electronic laboratory notebooks (ELN)

(Göde et al., 2007) or to use the wide range of

mobile device applications to calculate, to monitor

or to analyze data (de Souza, 2010). Mobile device

interfaces can also be found in special devices such

as the Optima XPN centrifuge of Beckman Coulter,

which can be monitored and controlled via an iOS

app (Muenz, 2012). Concerning more complex

applications smartphones and tablet PCs are a

comparatively new trend in life science laboratories.

2 ARCHITECTURE

The architecture of the developed systems follows

the typical automation structure of laboratory

automation as shown in fig. 1. The highest level in

this architecture is the hierarchical workflow

management system (HWMS). It is located in the

workflow control layer of the structured automation.

The name HWMS results from its position and

dependency to the complex hierarchical structured

sub-architecture below and it is classified as a

specialized manufacturing execution system (MES),

a so called laboratory execution system (LES). In

section III the HWMS is described in detail.

In the process control layer a general

differentiation is made between transportation and

automation systems' tasks, which is correspondingly

based on the two subsystems at the front end as

follows:

• The transportation and assistance control

system (TACS) permits the adaption and

distribution of transportation and assistance

orders from the HWMS and ensures their routing

to the transportation resources (pool of mobile

robots or alternative human operators).

Figure. 1: Architecture of the total system in the hierarchical structured laboratory automation.

ICINCO 2017 - 14th International Conference on Informatics in Control, Automation and Robotics

• The process control adapter system (PCAS)

consists of several instances, which interact

individually between the HWMS and the more or

less integrated automation systems, controlled by

process control systems (PCS) / scheduling

systems or vendor specific instrument control

systems (ICS).

Both systems adapt, execute and control the

required sub processes dependent on the orders of

the HWMS and return process-status information.

The systems are integrated by web-services-oriented

interfaces, which allow decoupled accesses from the

superordinated control systems. To relieve the

HWMS those systems have a sufficient application-

oriented intelligence to make own decisions for the

handling of sub processes

Dependent on the specificity of the order, the

TACS is able to dynamically schedule the

transportation and assistance orders for a planned

process between the available transport resources.

An internal list containing performance information

of all different transport resources allows an

automated selection of the best fitting mobile robot

or human operator. Each transportation resource

only receives one order at the same time, whilst the

orders can be more complex. For example, this can

be a labware transportation order of 20 different

objects (identified by barcode) or a list of assistance

orders to prepare and / or to monitor an automation

system (especially for human operators).

The group of transport resources is differentiated

into the following two options:

• The robot board center (RBC) executes the

transportation of labware via mobile robots (H20

Robot – Dr Robot Inc.). This unit includes the

navigation, the pick and place process, door

control procedures and the active collision

avoidance. The RBC requires the superordinate

robot remote center (RRC) in the process control

layer as central control instance for all mobile

robots. It has the responsibility for path planning,

central status information collection and decision

making regarding the most qualified robot,

which is selected by available power status and

the optimal current start position (Liu et al., 2012

& 2012 & 2013).

• The mobile human operator service (MHOS)

partly functions as a manual backup to the RBC.

However, by integration of human operators,

such as laboratory assistants, MHOS is

additionally able to provide orders for manual or

special tasks that cannot be performed by

automated systems. The MHOS is based on

mobile devices such as smartphones and tablet

PCs, which communicate directly with the

TACS.

The PCAS allows the connection to the involved

PCS to initiate runs of the automation systems for

complex laboratory tasks, including sample

preparation, dosage, analytical measurements, or

evaluation processes. General used PCS are complex

scheduling software systems such as SAMI® EX,

VWorks®, Microlab® VENUS or Clarity (Delaney

et al., 2013). Parallel to the automation systems with

PCS properties, single devices are required, which

can be integrated via the specified manufacturer

software (ICS) located in the instruments layer.

Therefore, the PCAS allows the direct access to the

instrument layer of the structured laboratory

automation and enables a higher integration

flexibility for the resource variety of the workflow

control layer.

A labware location guidance service (LLGS),

triggered by the PCAS, organizes the control of

special indicator systems to simplify human-

machine interactions. This refers for example to

complex labware storage hotels (400 positions) with

dynamic visual information that indicates the status

of every position slot to the human operator in

current and future processes of an automation line.

3 HIERARCHICAL WORKFLOW

MANAGEMENT SYSTEM

LES, like the HWMS, encapsulate complex

automation processes and ensure a stronger

decoupling of potentially superordinated control

systems (as e.g. the Business Process Management

System - BPMS) and the required automation

environment. Therefore, the HWMS includes the

complete adaption of the heterogeneous automation

systems' environments and simplifies the integration

of more real-time sensitive sub processes for the

end-to-end process automation (Neubert et al.,

2016).

The developed HWMS includes a relational

recursive database, which manages workflow

process definitions, workflow instances, locations,

labware, automation systems, mobile devices and

mobile robots. A web-portal allows the users and

system administrators to manage these environment-

tal parameters, which is most frequently required for

the definition of new labware and for printing the

appropriate barcodes. Based on these parameters an

Architecture for a Combined Mobile Robot and Human Operator Transportation Solution for the Hierarchical Life Science Automation

Figure 2: Material flow-oriented process model.

integrated graphical process design tool allows the

definition of an abstracted material flow diagram to

chain distributed heterogeneous PCS and ICS by

embedding all intersystem transport tasks in

laboratories (see fig. 2). Therefore, the user defines

the entry point for the labware into the workflow,

combines the required subsystems and determines

the already prepared methods of the PCS and ICS in

the process design tool. In practice these methods

are normally created for general procedures; thus it

can be used for different applications and

combinations. During the definition of the process

diagram the HWMS supports interactions with the

subsystems to receive planning-relevant information

(e.g. scheduling results of the PCS, labware start

positions of sub processes, status information) and to

implement them in the background into the technical

and the material-flow-oriented process model as e.g.

the number of the required labware path inputs and

outputs for a subsystems method execution

(see fig. 2). These labware paths allow to follow the

involved labware groups, which are defined by their

function in the process. In contrast to the labware

path input positions, which are defined in the start

conditions of a PCS-method, the output positions are

not always available during the planning phase. The

reasons for that are process-dependent logical

decisions in the PCS-method, which influence the

distribution of the labware on the final transfer

position (labware path output position). The HWMS

requests these output positions from the PCS during

the runtime, directly after finishing the subprocess,

to receive the actual positions of all paths.

The transportation tasks in the process model

will be integrated implicitly by the graphical linking

of the subsystems labware path inputs and outputs.

Process-model options allow e.g. the definition of

ICINCO 2017 - 14th International Conference on Informatics in Control, Automation and Robotics

Figure 3: Comunication protocol on the basis of xml.

the kind of transporter (mobile robot or human

operator). In the technical process model the

executability of the transporter decision will be

validated and corrected if necessary. Alternatively

this decision can also be passed to the TACS, if both

kinds of transporters can execute the specific task.

The end point determines the final position for the

labware after the competition of the workflow. The

complexity and the required usability of the process

diagram demand a specified notation as applied,

which can be very complex when using standards

from higher level control systems as e.g. BPMN 2.0

(Business Process Model and Notification) (OMG,

2011).

For the execution of the planned process

workflow, the material flow diagram has to be

translated in a processible format, separated in

logical sub processes (primary divided into transport

and automation system processes) and scheduled

together with all workflows to be performed. For the

latter a scheduler in the process controller is used,

which works with a genetic algorithm. It generates

an optimized execution order of the sub processes in

consideration of the limited numbers of resources

(e.g. applicable integrated automation systems or

transport units) and assigned priorities (Gu et al.,

2016).

Depending on the scheduling results the process

controller verifies the subsystems and triggers the

required subprocesses to execute arranged methods

or tasks (e.g. labware transport, pipetting process by

a liquid handler) on them, after starting the

execution by the user. The execution will be

assumed by the structured subsystems on the

subordinated process-control and instrument-control

layer.

Due to the fact that the conditions of the

environment can change, for example by newly

started workflows, which have to be implemented in

the total workflow, by changings in the number of

available transportation units (robots have to charge

or can be broken, human operators have a limited

working time or have to fulfil other tasks) or by

unforeseeable delays, which can occur especially

during the transportation processes, a dynamic

scheduling is required (Schäfer, 2004). It is realized

by a rescheduling of the total workflow during the

execution and is triggered by the detection of such

changings (delay measurements or notifications

about the available number of transportation units).

4 INTERACTION AND

COMMUNICATION

The HWMS is a web-based telematics platform,

which is primarily integrated by the laboratory's

computing environment. It provides the interfaces to

structured automation systems and considers the

usability of process definition and execution for a

wide range of life science laboratories. Thus, the

data transfer between the HWMS and the process

control layer's front-end systems is based on web-

services, which allow a simple and flexible

integration of all web-service providing subsystems.

The HWMS uses this to send process instructions

and to request the status of the front-end subsystems

to correlate the information and to initiate required

Architecture for a Combined Mobile Robot and Human Operator Transportation Solution for the Hierarchical Life Science Automation

Figure 4: UML Sequence diagram of the communication process between HWMS and the transportation control system.

processes in the subordinated architecture, following

the scheduling result.

In case of the PCAS, every PCS or ICS has a

preceded individual service instance, which uses

available service-oriented communication or

framework interfaces to access the automation

systems for workflow control. During the execution

of a sub process, the PCAS observes the running

process and offers status and error information on

request to the HWMS to inform the requesting user

and to resume the process chain.

Main function of the TACS is the distribution of

transportation orders for robots and human

operators. The TACS works centralized and

provides, in analogy to the PCAS, a web-service for

the communication to the HWMS. Depending on the

availability of the operators, the restrictions of the

user for the process and the reliability of the

different operators for the current order, the TACS

decides whether a robot or the human operator is

more qualified or can finish the order faster.

Furthermore, the TACS distributes assistance orders

such as sample preparation, restocking resources at a

workstation, prepare a laboratory automation system

for other processes or check their status after

execution faults. Currently these kinds of orders can

only be done by human operators.

The communication between the TACS and the

transportation resources (RRC or MHOS) is realized

by using a flexible XML-based message protocol

transmitted via TCP or UDP, dependent on the unit's

type. For the mobile robots, the RRC represents the

remote station to the TACS, which uses TCP-

communication and forwards the orders to the

requested robot (Liu et al., 2012). On the other hand,

the TACS integrates the separated MHOS units

based on several mobile devices via the UDP

protocol. The properties of UDP allow the TACS to

communicate with an almost unlimited number of

MHOSs without a continuous connection to the

single devices. It is less dependent on possible

connection interruptions, which may occur when

human operators leave the local wireless network.

Both kinds of transport resources work with a

highly structured XML-message protocol, which

contains all relevant information, such as the places

of source and destination and the list of labware

including the barcode. In fig. 3 a transportation order

is shown, which is based on a general transportation

rack with three positions in the ANSI/SBS footprint

ICINCO 2017 - 14th International Conference on Informatics in Control, Automation and Robotics

dimensions for microtiter plates. On these positions

combinations of microtiter plates and adapting racks

for different kinds of tubes, also in stack format, are

possible.

For every message dispatched by the TACS a

confirmation concerning the correctly received

command is expected. This implies completeness,

correct decompression and readability of the

message. After having finished the order, the

executive transport resources will send a confirming

reply back to the TACS. In fig. 4 the communication

cycle between HWMS, TACS and both kinds of

transportation resources are visualized in a sequence

diagram.

5 MOBILE HUMAN OPERATOR

SERVICE

As an alternative to the transportation by the mobile

robots, which is described more precisely in (Liu et

al., 2012 & 2012 & 2013), human operators are still

required for particular processing steps or for

limitations of robot pools and for special

transportation orders in life science processes. To

include a group of human operators into the

automated process, every single operator needs a

separate mobile interface to be flexible and to

receive orders from the TACS. Therefore, Android-

based mobile devices are used, which are available

in different formats. In general, mobile devices

combine a flexible I/O-interface with high

connectivity and other functionalities, such as

integrated cameras.

Via an Android-based laboratory application

(LApp), the human operators are logging-in

themselves to the TACS. Thus, they are registered as

being available to receive transportation and

assistance orders, dependent of the operator’s

individual role. All orders are listed and can be

selected by the operator to see more details and

further handling. A graphical overview of the

transportation order allows seeing the expected

positioning of the labware for the process on the

destination side. In case of preparing an automation

system, this has a determined arrangement of the

labware dependent on the current method. These

arrangements can consist of up to three stacks, each

with up to ten single microtiter plates or other

labware (see Fig. 5). The arrangement needs to be

maintained to avoid mistakes and to prevent the

workstation from re-sort processes, especially if the

space is limited.

Figure 5: Screenshot of the mobile device list and

graphical user interface with the required transport

arrangement of the labware.

To reduce the effort for the human operator, the

mobile device camera is used to identify the

labware's barcode and to allocate a labware to a

position in the arrangement, when the position is

allowed to be edited. This can appear when a

workstation is equipped with empty labware of the

same type. In this case, the MHOS initializes the

start position for the sample tracking across the

overall process.

Furthermore, the HWMS and even the integrated

stock management are available via the mobile

device's web browser. In this case, the human

operator can register, re-sort or find goods in the

stock management, which can be distributed all over

the institution. Label printers also enable the printing

of self-defined barcodes to identify new labware. All

these functionalities are also obtainable via

stationary computers.

6 CONCLUSION

This paper describes an architecture to combine

distributed conventional hybrid and heterogeneous

automation systems with mobile robots and human

operators (integrated by mobile devices) to solve the

demands of automated transportations between

Architecture for a Combined Mobile Robot and Human Operator Transportation Solution for the Hierarchical Life Science Automation

them. Thus, the automation level in life science

laboratories can be increased and the human

operators can also be integrated in required

assistance tasks in runtime.

The HWMS allows the handling of master data

and stock management, which is the basis for

planning life science processes over several

automation systems. By an integrated process design

tool the end user can plan workflows including

theses automation systems by a material-flow-

oriented process model. The resulting models can be

executed parallel by means of a dynamic scheduler

(based on genetic algorithm) to optimize the use of

the required resources (e.g. automation systems,

mobile robots, human operators) in the complex

workflow. Variable transfer positions between the

automation systems are also tolerated by the

HWMS. By using mobile robots, HWMS

accomplishes an important requirement of life

science automation as a complex process connector

between distributed automation systems in a

building.

The TACS and the PCAS build the front end of

the process-control layer and share the control over

the automation systems as well as the mobile robots

and mobile devices. Both systems distribute and

adapt the orders for the corresponding automation

subsystems, whereby the TACS is able to conduct

dynamic transportation-unit allocations for each

intersystem transport process. For the

communication between the TACS and the

subsystems, a uniform XML-message-protocol via

TCP and UDP is used. The PCAS consists of

individual services on the local computers of the

heterogeneous automation systems to implement the

whole performance range of the systems.

Especially the MHOS unit, available on smart

phone or tablet PC, integrates laboratory assistants

increasingly in the running workflow. The

requirements of the system, which cannot be

performed by mobile robots, will be directly

submitted via transportation or assistance orders to

human operators.

In summary, the presented workflow

management system located on the workflow control

layer is able to speed up the laboratory work in

general, to reduce the effort for human operators and

to combine human and machine in an automated or

semi-automated process albeit this puts the

observation function for the human operators more

in the center stage. Thus, the automated system

receives the control role concerning time

management, while the focus of the human operators

is on manual preparation steps and on observing the

process as a whole.

Although humans currently transport faster than

the robots, used in this solution, the process-

integrated robot operators have the advantage of

being able to react immediately to a command, to

manage dangerous transportation orders and to

operate 24/7. Thus, although the advantages of

human und robot operators differ, they are both

useful depending on the respective requirements and

a parallel availability of both operators cover

processes around-the-clock and high priority or

special transportation orders, too.

ACKNOWLEDGEMENTS

The authors thank the Ministry for Economic

Affairs, Construction and Tourism of Mecklenburg-

West Pomerania

(Germany, FKZ: V-630-S-105-

2010/352, V-630-F-105-2010/353

) and the Federal

Ministry of Education and Research (FKZ:

03Z1KN11, 03A1KI1) for the financial support of

this project. This work has been supported by the

European Union.

REFERENCES

Patel, S. N. Prajapati, K. R. Sen, D. J., 2014. Automation

by Laboratory Robotics in Pharmaceutical Research

Industry: A latest venture in innovative idea. In: World

Journal of Pharmacy and Pharmaceutical Science

(WJPPS), vol.3 (2), pp. 2098–2105.

Lam, C. W., Jacob, E., 2012. Implementing a laboratory

automation system: experience of a large clinical

laboratory. In: J Lab Autom, vol.17 (1), pp. 16–23.

Cork, D. G., Sugawara, T., 2002. Laboratory automation

in the chemical industry. New York: Marcel Dekker.

Andersen, D., Rasmussen, B., Linnet, K., 2012. Validation

of a fully automated robotic setup for preparation of

whole blood samples for LC-MS toxicology analysis.

In: J Anal Toxicol, vol. 36 (4), pp. 280–287.

Liu, L., Zhang, R., Pajak, L., 2010. Automation of Human

IgG, Glucose, Lactate, and Oxygen Assays on Biomek

NXP Span-8 Automation Workstation and

PARADIGM Detection Platform. In: Journal of the

Association for Laboratory Automation, vol. 15 (5),

pp. 414–418.

Sutherland, J. D., Tu, N. P., Nemcek, T. A., Searle, P. A.,

Hochlowski, J. E., Djuric, S. W., Pan, J. Y., 2014. An

Automated Synthesis-Purification-Sample-Manage

ment Platform for the Accelerated Generation of

Pharmaceutical Candidates. In: Journal of Laboratory

Automation, vol. 19 (2), pp. 176–182.

ICINCO 2017 - 14th International Conference on Informatics in Control, Automation and Robotics

Göde, B., Holzmüller-Laue, S., Rimane, K., Chow, M.-Y.,

Stoll, N., 2007. Laboratory Information Management

Systems – An Approach as an Integration Platform

within Flexible Laboratory Automation for

Application in Life Sciences. In: Proc. of the 3rd

Annual IEEE CASE, Scottsdale, AZ, USA, Sept 22-25,

pp. 841–845.

de Souza, N. (ed), 2010. The scientist and the smartphone.

In: NatMet, vol. 7 (2), p. 87.

Muenz, R., 2012. The right choice of centrifuges. In: Lab

Manager, p. 51.

Liu, H., Stoll, N., Junginger, S., Thurow, K., 2012. A

common wireless remote control system for mobile

robots in laboratory. In: Proc. of the IEEE I2MTC

Conference 2012, pp. 688-693.

Liu, H., Stoll, N., Junginger, S., Thurow, K., 2013. An

Application of Charging Management for Mobile

Robot Transportation in Laboratory Environments. In:

Proc. of the IEEE I2MTC Conference 2013, pp. 435-

439.

Liu, H., Stoll, N., Junginger, S., Thurow, K., 2012. A

Floyd-Dijkstra Hybrid Application for Mobile Robot

Path Planning in Life Science Automation. In: Proc. of

the 8

IEEE International CASE 2012, Seoul, Korea,

pp. 279-284.

Delaney, N. F., Echenique, J. I. R., Marx, C. J., 2013.

Clarity: an open-source manager for laboratory

automation. In: J Lab Autom, vol.18 (2), pp. 171–177.

Neubert, S., Göde, B., Gu, X., Stoll, N., Thurow, K., 2016.

Potential of Laboratory Execution Systems (LESs) to

Simplify the Application of Business Process

Management Systems (BPMSs) in Laboratory

Automation. In: JALA, Online First, December 2016.

OMG. Business Process Model and Notation (BPMN)

Version 2.0. 2011. http://www.omg.org/spec/BPMN/

2.0

Gu, X., Neubert, S., Stoll, N., Thurow, K., 2016 Intelligent

Scheduling Method for Life Science Automation

Systems. In: Proc. IEEE International Conference on

Multisensor Fusion and Integration 2016, pp. 156-

161.

Schäfer, R., 2004. Concepts for dynamic scheduling in the

laboratory. In: JALA, vol.9 (6), pp. 382–397.

Architecture for a Combined Mobile Robot and Human Operator Transportation Solution for the Hierarchical Life Science Automation