Development of a Test Environment for the Evaluation of

Human-Technology Interaction in Cockpits of Highly-Automated

Vehicles

Patrick Schn

oll

Institute for Media Technology, University of Technology Ilmenau, Germany

Keywords:

Automated Driving, Automation, Driver Take-over Task, Driving Simulator, Human-machine Interaction,

Human-technology Interaction, Interface Design, Take-over Request, Test Environment, Test Setup.

Abstract:

This paper presents a technologically independent framework to describe test environments suitable for the

examination of the driver take-over task present in highly-automated vehicles. As part of a structural analysis,

typical inﬂuencing factors and parameters deﬁning the driver take-over task are discussed. According to litera-

ture, existing studies examining the driver take-over task make use of various test environments. However, the

comparability of their results is not given without a detailed understanding of these. Hence, based on estab-

lished literature, a technologically independent framework has been developed which can be used to describe

the distinct test environments. It turned out, that the referenced models had to be partially restructured in order

to be suitable for the description of such test environments. The focus of the present paper lies on their techni-

cal implementation characterized by stimulus materials, which have been holistically examined for the driver

take-over task. Since stimulus materials provide the foundation of a speciﬁcation of test environments, this

work presents an initial step towards a test speciﬁcation aiming on making results obtained from examinations

of the driver take-over task comparable.

1 INTRODUCTION

The degree of automation in the ﬁeld of automo-

tive is steadily increasing. Highly-automated driving

is associated with a more comfortable and produc-

tive driving experience, allowing drivers and passen-

gers to perform a variety of non-driving related tasks

(NDRT).

From the perspective of the commercial vehicle

market, high automation degrees have the potential

to optimize the utilization by allowing professional

drivers to meet their rest times during transit which re-

sults in ﬂexibility regarding driving schedules. Hence

both, private end-users as well as commercial cus-

tomers can proﬁt from this trend.

However, as of now, there is no vehicle automa-

tion system in high volume production, which is ca-

pable of fulﬁlling all requirements of a L5 system per

(SAE International, 2018). Such systems shall have

an unlimited operational design domain, meaning that

they shall be able to reach a deﬁned target point from

a deﬁned starting point anywhere in the world while

incorporating redundancy as part of the automation

system. As long as systems capable of this are still

in development, the driving task as a whole is semi-

automated while the level of automation steadily in-

creases with advancing developments in the ﬁeld of

vehicle automation.

Therefore, in the near future, at least a small share

of the total distance traveled in a vehicle equipped

with an automation system will be driven manually.

Typically, this affects the passage of the ’ﬁrst-mile’

and the ’last-mile’, respectively the start of the drive

(e.g. from parking lot to highway) as well as the end

of the drive (e.g. from highway to parking lot).

Hence, in such cases, at minimum one transition

from the driver to the automation system (hand-over)

and vice-versa (take-over) is required. In order to ex-

ecute these hand-over and take-over tasks, a human-

technology interface, which must provide a safe tran-

sition into and out of the automation loop for the

driver, is required.

Depending on the share the driver is actively ma-

neuvering the vehicle compared to the automation

function having control, different requirements for in-

put and output devices arise. Due to the ongoing ad-

vancement of technology, there is a wide variety of

solutions available, from traditional cockpit designs

Schnöll, P.

Development of a Test Environment for the Evaluation of Human-Technology Interaction in Cockpits of Highly-Automated Vehicles.

DOI: 10.5220/0010144900640073

In Proceedings of the 4th International Conference on Computer-Human Interaction Research and Applications (CHIRA 2020), pages 64-73

ISBN: 978-989-758-480-0

with head-up displays up to advanced technologies

like, among others, e.g. wind-shield displays and aug-

mented reality devices.

In order to implement these technologies into au-

tomotive cockpits, they have to be assessed and tested

for their suitability. This process requires extensive

testing and evaluation, whereas at the same time, the

solution space for a cockpit design is increasing sig-

niﬁcantly.

2 RELATED WORK

2.1 Description of Driving Tasks

Hierarchy of the Driving Task. The hierarchy of

the driving task gives a subdivision into the execution

of primary, secondary and tertiary tasks based on their

contextual proximity to and relevance for the fulﬁll-

ment of the actual transport task (Bubb et al., 2015;

Winner et al., 2015).

The underlying human performance when execut-

ing such tasks can be described based on the skill-,

rule- and knowledge-based behavior (Rasmussen,

1983) whereas the driving task itself can be catego-

rized into a navigation-, guidance- and stabilization

task (Donges, 1982).

These models can be linked to each other content-

wise in order to describe the driver’s task execution

during driving (Winner et al., 2015).

Non-driving Related Tasks. The categorization of

the driver’s tasks refers to the content-related proxim-

ity of the respective tasks to the actual driving task,

whereas the primary tasks are closest to the actual

driving task in terms of content.

One of the main beneﬁts of highly-automated

driving is the potential of the driver focusing on non-

driving related (cf. tertiary) tasks while the vehicle is

in motion (Naujoks et al., 2017). Hence, automation

functions must support a fast and safe transition from

the execution of those non-driving related, tertiary

tasks to the execution of primary (Othersen, 2016)

and secondary tasks, as illustrated in Figure 1.

This paper focuses on the examination of the

driver take-over task representing the responsibility

shift between the automation system and the human

driver.

2.2 Driver Take-over Task

Deﬁnition. The driver shifting from an

’out-of-the-loop’-state, often being engaged in

DRIVING TASKS

Secondary

Task(s)

Primary

Task(s)

Tertiary

Task(s)

Driver Take-Over Task

Figure 1: Illustration of the driver take-over task with re-

spect to the hierarchy of the driving task following on from

(Bubb et al., 2015; Winner et al., 2015).

tertiary tasks, to having manual control over the ve-

hicle is commonly referred to as the driver take-over

task. It is triggered through a driver take-over request

(TOR) when initiated by the automation system.

The driver take-over task represents the responsi-

bility shift from the automation system to the driver

being handled through a human-technology interface.

Deﬁning Parameters. Based on (Eriksson and

Stanton, 2017), the driver take-over task can be de-

scribed by two distinct time intervals: the take-over

request lead time TOR

, giving the time interval re-

maining until the take-over has to be performed, as

well as the take-over reaction time TO

, the actual

time span of the take-over task itself. The examina-

tion of (Eriksson and Stanton, 2017) shows that TOR

and TO

positively correlate.

Inﬂuencing Factors. The completion of the driver

take-over task takes signiﬁcantly longer if the

driver has to return to the driving task from being

’out-of-the-loop’ (Merat et al., 2014). Hence, (Banks

and Stanton, 2016) suggest to keep the driver engaged

in secondary tasks, making the driver an ’active super-

visor’ of the automation system.

(Wu et al., 2020) indicate, that although the driv-

ing duration has a signiﬁcant effect on the drowsiness

level, it has no effect on the take-over performance.

However, they have shown that the take-over perfor-

mance is reduced with increasing drowsiness.

Another factor when dealing with automation sys-

tems is mode awareness, meaning that the driver must

be aware which portions of the system are presently

automated and which have to be manually controlled.

If this is not clear to the driver, this is called mode

confusion, which is subject to ongoing research re-

garding automotive interface design (Neuhuber et al.,

2020; Pretto et al., 2020).

Furthermore, the behavior of drivers is inﬂuenced

by their subjective interpretation of the functionali-

ties implemented in the automation function as well

as their characteristics, commonly referred to as the

driver’s mental model.

Development of a Test Environment for the Evaluation of Human-Technology Interaction in Cockpits of Highly-Automated Vehicles

(Neuhuber et al., 2020) indicate a correlation be-

tween age and mode confusion, mental models as well

as general difﬁculties with interaction with automa-

tion systems. Hence, they conclude that the age of the

driver has a signiﬁcant inﬂuence on the way driver as-

sistance systems are used and that state-of-the-art sys-

tems may not be easily used by all groups of drivers.

(Bundesanstalt f

ur Straßenwesen, 2012) lists per-

sonal characteristics, which have an inﬂuence on driv-

ing behavior with the focus on young drivers. How-

ever, the categorization can be transferred to all age

groups in order to give a qualitative, extensive im-

pression of personal factors which have to be con-

sidered when evaluating test environments for highly-

automated driving.

Table 1 lists personal factors which have an in-

ﬂuence on driving behavior based on (Bundesanstalt

ur Straßenwesen, 2012). Corresponding to (Bunde-

sanstalt f

ur Straßenwesen, 2012), the listed personal

factors are structured into stable and variable charac-

teristics. Stable characteristics contain, among oth-

ers, sociodemographic aspects, cognitive skills, expe-

rience as well as personal characteristics and are not

changed frequently.

Contrary to this, variable characteristics, such as

ﬁtness to drive, and the current emotional state are

subject to a frequent change, which may happen even

multiple times per day.

Take-over Times. (Eriksson et al., 2017) compared

driver hand-overs to and driver take-overs from the au-

tomation system in vehicles on public roads to driving

simulators. They found that the transitions between

the driver and the automation system happened gen-

erally faster when driving on public roads compared

to simulated environments.

Furthermore, (Eriksson and Stanton, 2017) have

compared various publications examining the take-

over task. Their results show, that the average time

needed for a successful take-over request differs in lit-

erature. Therefore, making a generic statement about

’typical’ take-over times is difﬁcult based on the avail-

able studies.

Comparability of Test Environments. Besides the

potential differences in research methods and designs

used in the studies, as referenced by (Eriksson and

Stanton, 2017), the respective test environment itself

as well as their technical implementation may have an

inﬂuence on the performance of the test person during

task execution.

In order to substantiate this hypothesis, the inﬂu-

ences on the driver take-over task examined in dis-

tinct test environments by different test persons and

Table 1: Personal factors inﬂuencing driver behavior (Bun-

desanstalt f

ur Straßenwesen, 2012).

Personal Factors

Stable Characteristics

Sociodemographic aspects

• Age

• Gender

Socioeconomic status

Mobility-related attributes

• Driving suitability

• Driving competence

Cognitive skills

• Perception of risk / danger

• Concentration and attention

• Assessment of own abilities

Schemes

Experience

• Mileage

• Level of practice

• Frequency of driving

Motives, attitudes and expectations

Personal characteristics

Diseases

Variable Characteristics

Fitness to drive

Information processing (e.g. attention control)

Emotions and sensitivities

Diseases

Subjective safety

Subjective task difﬁculty

their effects must be understood in detail to be able to

compare studies being performed in varying test en-

vironments and explain potential deviations.

This whitespot gives the motivation of a detailed

examination of the comparability of test environments

used for human-centered studies in the ﬁeld of auto-

motive cockpit design.

Hence, this paper aims on presenting an approach

to a framework providing a guideline to analytically

describe a test environment for highly-automated

driving along with typical personal inﬂuence factors

on the respective test persons.

CHIRA 2020 - 4th International Conference on Computer-Human Interaction Research and Applications

2.3 Test Environments for the Driver

Take-over Task

2.3.1 State-of-the-Art

Table 2 shows a categorization of test environments

being used to examine driver behavior and human

performance from a vehicle driving on public roads

1 to laboratory setups 2 - 8 . The test environ-

ments have been exemplary arranged based on avail-

able stimulus materials.

Immersion & Presence. Depending on the effort

put into building up a test environment, different lev-

els of presence perceived by the test person can be

achieved, which is especially important in laboratory

test environments such as 2 - 8 .

Immersion, as a prerequisite of presence (Schwind

et al., 2019; Witmer and Singer, 1998), can be reached

by the artiﬁcial environment matching the user’s ex-

pectations, being inﬂuenced by the user’s actions as

well as being consistent with regard to its conventions

(McMahan, 2003). Hence, stimulus material is an in-

tegral aspect for immersion.

Typically, test setups used in literature consist of

a narrowed down version of a vehicle cockpit, at least

incorporating gas- and brake pedals as well as a steer-

ing wheel. The implementation efforts being invested

in these depend on the respective research questions

being examined and vary widely (cf. Table 2).

Therefore, due to their conceptual differences,

results obtained from studies performed in varying

test environments can only be compared very selec-

tively and if the test environment, test design and test

method has been described in detail.

However, as soon as a study is performed in a lab-

oratory test environment, such as

2 - 8 , the inﬂu-

ence of the perceived presence of the test person on

the result of the examination has to be considered.

2.3.2 Analytical Description

In order to create a common baseline supporting the

comparison of results obtained with differentiating

test environments, they have to be described holisti-

cally on a generic level.

The established ’framework for analyzing the ef-

fects of tasks’ of (Hackman, 1969) has been chosen as

the foundation for this analysis because of its strong

relation to stimulus material.

Figure 2 shows an illustration of a generic test en-

vironment suitable for an examination of the driver

take-over task based on (Hackman, 1969) and is dis-

cussed in the following.

Table 2: Test environments being used to examine driver

behavior and human performance.

Test Environments

Exemplary arranged in descending

order based on available stimulus materials

Description

Exemplary

Implementation

Vehicle on public roads

/ test track

(Banks et al.,

2018; Eriksson

et al., 2017)

Dynamic driving

simulator with screen

projection (VR CAVE)

(Sch

omig et al.,

2015; Jamson

et al., 2013)

Dynamic driving

simulator with VR

headset

(Hartﬁel et al.,

2019)

Static vehicle with

screen simulation

(Petermeijer

et al., 2017;

Larsson et al.,

2015)

Static simulation with

screen projection (VR

CAVE)

ohn et al.,

2019)

Static simulation with

multiple displays

(Li et al., 2019)

Static simulation with

VR headset

(Walch et al.,

2017)

Static screen

simulation with

minimal input device

set

(Capallera et al.,

2019)

The objective task input is characterized by stim-

ulus materials, instructions about operations as well

as instructions about goals. It is subjectively rede-

ﬁned by the test person resulting in the redeﬁned task

input, whereas this redeﬁnition is inﬂuenced by per-

sonal factors (cf. Table 1).

Following (Hackman, 1969), based on the rede-

ﬁned task input, the test person formulates hypothe-

ses on how to approach the given task best in order to

achieve maximum success. The hypothetical actions

required are executed in the process step and result

in a trial outcome. The process outcome links give a

causal link between performed actions and outcomes.

The evaluation of the performed process can ei-

ther happen on system or personal side. If the pre-

ceding process execution did not lead to a satisfying

outcome, new hypotheses are formed and another it-

eration is carried out in order to receive a new trial

outcome which can be evaluated.

Development of a Test Environment for the Evaluation of Human-Technology Interaction in Cockpits of Highly-Automated Vehicles

Stimulus

Materials

Instructions

About

Operations

Instructions

About

Goals

Objective

Task Input

Redeﬁned

Stimulus

Materials

Redeﬁned

Instructions

About

Operations

Redeﬁned

Instructions

About

Goals

Redeﬁned

Task Input

Hypotheses

Process

Trial

Outcome

Final

Outcome

Process

Outcome

Links

System

Evaluation

Personal

Evaluation

Test

Person

Personal Factors

1 Understanding of Task

2 Acceptance of Task

3 Idiosyncratic Needs, Values, etc.

4 Previous Experience with Similar Tasks

5 Ability

6 Performance Motivation

7 Level of Arousal

∆

Error(s)

Figure 2: Illustration of a generic test environment suitable for an examination of the driver take-over task based on (Hackman,

1969).

If the evaluation of the trial outcome turns out pos-

itive, the result of the last performed process gives the

ﬁnal outcome.

The objective task input as well as the ﬁnal out-

come are factors extrinsically deﬁned or evaluated by

third parties such as the research team.

The potential difference between the objective

task input and the ﬁnal outcome can be assessed and

connected to errors of the test person or systematic

errors of the technological implementation of the test

environment itself and represents the individual per-

formance of each test person.

3 RESEARCH QUESTION,

DESIGN AND METHOD

As discussed in Section 2.2, (Eriksson and Stanton,

2017) reference various studies examining the driver

take-over task. However, comparing the results of

these studies is difﬁcult due to the varying test en-

vironments, -methods and -designs being used.

The ﬁrst step towards a qualitative comparability

of these studies is the understanding of the distinct

characteristics of the used test environments. Hence,

a generic, common basis for a speciﬁcation of these is

required.

3.1 Research Question

An essential characteristic of the common basis for

such test environments is the technical implementa-

tion required to execute a driver take-over task. This

leads to the following research question:

Which stimulus materials are required for the ex-

ecution of the driver take-over task?

3.2 Research Design

The research question is answered by deriving the

technical implementation for a test environment suit-

able for the examination of the driver take-over task

from its structured analytical description.

3.3 Research Method

Established models have been applied to analyze typ-

ical test environments for highly-automated driving

used for the examination of the driver take-over task.

Based on this, the intention was to be able to holis-

tically describe the driver take-over task considering

the respective test environment the task is performed

in.

CHIRA 2020 - 4th International Conference on Computer-Human Interaction Research and Applications

4 RESEARCH RESULTS

As part of the analysis, the hierarchy of the driving

task (Winner et al., 2015; Bubb et al., 2015) as well

as the framework of (Hackman, 1969) (cf. Figure 2)

have been merged into a common context, shown in

Figure 3.

Focus on

Primary /

Secondary

Task(s)

Manual

Driving

Mode

Focus on

Tertiary

Task(s)

Automated

Driving

Mode

Stimulus Materials

Instructions

About

Goals

Instructions

About

Operations

TECHNICAL IMPLEMENTATION

PROCESS

DRIVER TAKE-OVER TASK

Final

Outcome

TEST PERSON

TEST ENVIRONMENT

TEST METHOD / -DESIGN

Figure 3: Illustration of a test environment suitable for the

examination of the driver take-over task following on from

(Hackman, 1969; Winner et al., 2015; Bubb et al., 2015).

As shown in Figure 3, the driver take-over task is

performed by a test person inside a test environment

as part of the process step (cf. Figure 2). It is charac-

terized by a transition of responsibilities when shift-

ing from the automated to the manual driving mode.

The test person’s focus typically lies on tertiary tasks

during the automated driving mode whereas a fast

transition to primary and secondary tasks has to be

conducted during the driver take-over task, as already

indicated in Figure 1.

Contrary to (Hackman, 1969), who aggregates

stimulus materials, instructions about operations and

instructions about goals as the ’objective task input’

(cf. Figure 2), in the context of a test environment

suitable for the examination of the driver take-over

task, these can be separated into two distinctive com-

ponents.

As illustrated in Figure 3, a test environment can

be described through its technical implementation as

well as the underlying process to be executed. The

process is speciﬁed by the stimulus materials, the in-

structions about operations and the instructions about

goals which are communicated to the test person.

The instructions about operations and the instructions

about goals depend on the test method and -design

and thus are closely related to the research question

in scope of the respective examination.

As shown in Figure 3, the technical implementa-

tion must provide stimulus materials allowing an in-

teraction of the test person during the process execu-

tion (cf. Figure 2).

Moreover, the stimulus materials serve as an input

for the evaluation of the quality of the trial outcome

in order for the test person to accept a trial outcome

as the ﬁnal outcome.

Stimulus Materials. The task qua task approach

outlined by (Hackman, 1969; McGrath and Altman,

1966) provides a suitable analytical foundation for the

speciﬁcation of a technical system implementation of

a test environment due to its close relationship to ac-

tual physical signals (Hackman, 1969).

Hence, it has been chosen to describe the stimulus

materials being part of the technical implementation

required to perform the driver take-over task in a test

environment.

The stimulus materials listed in Table 3 are struc-

tured into categories referring to the ego vehicle,

the static environment, the dynamic environment, the

drivable space, the trafﬁc as well as the weather at the

time the driver take-over request is initiated.

As illustrated in Figure 3, a subset of stimulus ma-

terials, typically related to the ego vehicle, are subject

to an active interaction by the test person as part of the

process execution (cf. Figure 2-3) during the driver

take-over task.

Table 4 gives an overview about these and lists the

respective processes on a high level.

5 CRITICAL ANALYSIS

This work pursues the idea of creating a common ba-

sis for the speciﬁcation of test environments for auto-

motive cockpit designs in highly-automated vehicles.

The results aim to be applicable to a wide vari-

ety of driving tasks and test environments. The driver

take-over task has been exemplary selected as a repre-

sentative task being present in highly-automated driv-

ing due to its safety relevance and complexity. In or-

der to get conﬁdence in the generic suitability of this

approach, additional work tasks must be described ac-

cording to the introduced framework.

Development of a Test Environment for the Evaluation of Human-Technology Interaction in Cockpits of Highly-Automated Vehicles

Table 3: Stimulus materials for a technical implementation of a test environment suitable for the examination of the driver

take-over task.

Ego vehicle — Vehicle the test person is located in

Intended destination Waypoint(s)

For each:

• Latitude / longitude, alternatively:

• Relative position to the next waypoint

Planned path

Vehicle dynamics Velocity (x, y, z) • Velocity in [x- / y- / z-] directions

Acceleration (x, y, z) • Acceleration in [x- / y- / z-] directions

Vehicle speed • Vehicle speed signal

Heading • [Heading / steering / steering wheel] angle

Dynamic behavior • Articulation point(s)

• Physically possible max. velocity and accelerations

• Slip

Location Position • Latitude / longitude, alternatively:

• Relative position with regard to the road

Geometry Type • [Car / van / pickup truck / truck / bus / tractor / other]

Size • Length / width / height

State Status • System ’health’, automation system state, active gear, lights, etc.

Errors • System faults

Static environment

Static obstacles Position • Latitude / longitude, alternatively:

• Relative position with regard to the road

Type • [Road boundary / construction site]

• [Road user in standstill / lost cargo / other]

Size • Length / width / height

Passability • Passability signal

Dynamic environment

Other road users

• Vehicle dynamics

• Geometry

• Position

• See ’Ego vehicle’ section

Non-road users

Drivable space

Drivable space Road material • [Asphalt / gravel / sand / dirt / ice / snow], opt. surface condition

Road information • Curvature / road boundary position / no. of lanes / lane markings

Trafﬁc

Trafﬁc Trafﬁc rules • Local trafﬁc rules, e.g. [left-hand / right-hand] trafﬁc

Trafﬁc characteristics • e.g. Trafﬁc [ﬂow / density / jam], [maximum / mean] speed

Weather

Weather Precipitation • [Rain / ice pellets / snow]

Light • [Dawn / day / twilight / night / direct sunlight / clouds / fog]

Meteorological data • Temperature / humidity / etc.

Visual range Visual range • Resulting from e.g. light, precipitation, meteorological data

CHIRA 2020 - 4th International Conference on Computer-Human Interaction Research and Applications

Table 4: Stimulus materials being actively inﬂuenced by the

test person as part of the process execution (cf. Figure 2-3)

during the driver take-over task.

Process (cf. Figure 2)

Stimulus

Material

Control the

ego velocity

Control the

gas pedal

• Acceleration

Control the

brake pedal

• (Negative)

Acceleration

Control the

ego heading

Control the

steering

wheel angle

• Heading

Control the

signaling

equipment

Control the

indicator(s)

• Lights

End the

automation

Conﬁrm the

take-over

• Automation

system state

Furthermore, the amount of potential use-cases,

scenarios, environmental conditions and personal fac-

tors inﬂuencing the driver take-over task lead to a

large number of different variations. Hence, the pre-

sented tables, ﬁgures, lists and examples do not claim

to be exhaustive.

However, the presented approach can support the

comparability of studies by providing an initial but

important step towards a documentation guideline for

studies investigating human performance in driver

take-over situations.

The presented approach was developed through

analytical research. To prove its suitability, it should

be applied to further test environments and studies ex-

amining the driver take-over task.

The content of this paper covers the description

of the technical implementation of a test environ-

ment. The instructions about operations and the in-

structions about goals, which depend on the respec-

tive test method and -design, closely related to the

research question in scope, have to be assessed sep-

arately.

6 CONCLUSIONS

6.1 Summary

In order to be able to compare studies examining

the driver take-over task using differentiating test se-

tups, a common descriptive basis is required to under-

stand their coinciding results and potential deviations.

Therefore, this paper presents a technologically inde-

pendent description of test environments in the ﬁeld

of highly-automated driving based on established re-

search models and literature. It can be used likewise

for the analysis of existing test environments as well

as for the deﬁnition of requirements for new test en-

vironments to be build up.

The analysis has shown that a test environment

is inﬂuenced by two major components: the techni-

cal implementation as well as the test method and

-design (cf. Figure 3). The test environment contains

aspects being subjectively inﬂuenced by the test per-

son, such as the execution process of the task itself

(cf. Figure 2). The technical implementation consists

of stimulus materials which represent interaction with

the process.

The test environment can be used to examine the

research question in scope along with the test method

and -design. These are derived based on the research

question and contain instructions about operations

as well as instructions about goals (cf. Figure 2-3,

(Hackman, 1969)).

The presented analysis focused on the identiﬁca-

tion of necessary requirements for the technical im-

plementation of a test environment suitable for exam-

ining the driver take-over task. The speciﬁcation for

stimulus materials for such test environments is given

in Table 3, while those being actively inﬂuenced by

the test person during the driver take-over task are

highlighted in Table 4 respectively.

As Table 3 shows, various stimulus materials must

be made available to the test person due to their in-

ﬂuence on the situation interpretation and hypothesis

selection of the driver when performing the take-over

task (cf. Figure 2-3).

6.2 Discussion

Although the structural analysis presented in this pa-

per builds on (Hackman, 1969), it turned out to be

more suitable to describe test environments by sep-

arating stimulus materials, instructions about oper-

ations and instructions about goals into a compo-

nent with technical relation (technical implementa-

tion) as well as another component with relation to

the test method and -design used for the examination.

Hence, the objective task input, as it was introduced

by (Hackman, 1969), can be subdivided further for

this application.

However, the redeﬁned task input, as it is de-

scribed by (Hackman, 1969), is part of the test envi-

ronment since it is inﬂuenced by the test person’s per-

sonal factors which corresponds to (Hackman, 1969)

and gives an input to the process (cf. Figure 2).

The presented approach is meant to create aware-

ness for the multitude of factors and stimulus materi-

Development of a Test Environment for the Evaluation of Human-Technology Interaction in Cockpits of Highly-Automated Vehicles

als, which have to be considered when conducting an

examination related to the driver take-over task. This

is especially relevant for studies being conducted in

laboratory test environments (cf. Table 2). However,

the discussed aspects similarly apply to other driving

tasks.

To achieve comparability of the available stud-

ies examining the driver take-over task, researchers

should describe the stimulus materials provided to the

test person in the respective studies in detail in order

to support a comprehensive interpretation of their re-

sults. Therefore, especially when publishing results

obtained from using laboratory test environments, it is

recommended not only to describe the technical setup

itself, respectively how certain stimuli have been pro-

vided, but additionally provide details on why it has

been decided to tailor certain stimuli compared to

driving in a vehicle on public roads, if applicable.

Hence, this sets new demands on the documenta-

tion of studies conducted in this ﬁeld.

6.3 Outlook

In a subsequent analysis, instructions about opera-

tions and instructions about goals provided to the test

person as part of the test method and -design have to

be examined in order to holistically understand poten-

tial inﬂuencing factors on the driver take-over task.

The differences between the objective task input

and the ﬁnal outcome, as introduced in Figure 2 as er-

rors, can be categorized based on human error models

available in literature, which supports the understand-

ing of human interaction with automated driving sys-

tems. The ability to understand the origin of human

errors during the driver take-over task is essential dur-

ing the design phase of human-machine interfaces for

the driver take-over task.

Furthermore, the proposed models should be im-

plemented in existing studies in order to prove their

suitability. Following on from this, a generic guide-

line can be developed, which supports the planning,

execution and evaluation phase of studies.

To continue working on this aspect, a test speciﬁ-

cation containing a common set of criteria, descriptive

categories and boundary conditions can be developed.

This can then be published as an appendix to the re-

spective studies in order to facilitate their comparabil-

ity leading to more resilient research results.

REFERENCES

Banks, V. A., Eriksson, A., O’Donoghue, J., and Stanton,

N. A. (2018). Is partially automated driving a bad

idea? Observations from an on-road study. Applied

Ergonomics, 68:138–145.

Banks, V. A. and Stanton, N. A. (2016). Keep the driver in

control: Automating automobiles of the future. Ap-

plied Ergonomics, 53(June):389–395.

Bubb, H., Bengler, K., Gr

unen, R. E., and Vollrath, M.

(2015). Automobilergonomie. Springer Fachmedien

Wiesbaden, Wiesbaden.

Bundesanstalt f

ur Straßenwesen (2012). Einﬂussfaktoren

auf das Fahrverhalten und das Unfallrisiko junger

Fahrerinnen und Fahrer. Berichte der Bundesanstalt

ur Straßenwesen, Heft M 229(Mensch und Sicher-

heit).

Capallera, M., de Salis, E., Meteier, Q., Angelini, L., Car-

rino, S., Khaled, O. A., and Mugellini, E. (2019). Sec-

ondary task and situation awareness, a mobile applica-

tion for conditionally automated vehicles. In Proceed-

ings of the 11th International Conference on Auto-

motive User Interfaces and Interactive Vehicular Ap-

plications: Adjunct Proceedings, pages 86–92, New

York, NY, USA. ACM.

Donges, E. (1982). Aspekte der Aktiven Sicherheit bei

der F

uhrung von Personenkraftwagen. Automobil-

Industrie, 27:183–190.

Eriksson, A., Banks, V. A., and Stanton, N. A. (2017). Tran-

sition to manual: Comparing simulator with on-road

control transitions. Accident Analysis & Prevention,

102:227–234.

Eriksson, A. and Stanton, N. A. (2017). Takeover Time in

Highly Automated Vehicles: Noncritical Transitions

to and From Manual Control. Human Factors: The

Journal of the Human Factors and Ergonomics Soci-

ety, 59(4):689–705.

Hackman, J. R. (1969). Toward understanding the role

of tasks in behavioral research. Acta Psychologica,

31:97–128.

Hartﬁel, B., Kroys, A., Kruithof, N., and Stark, R. (2019).

Driving Simulator with VR Glasses for Evaluation of

New Interior Concepts. ATZ worldwide, 121(11):16–

23.

Jamson, A. H., Merat, N., Carsten, O. M., and Lai, F. C.

(2013). Behavioural changes in drivers experienc-

ing highly-automated vehicle control in varying trafﬁc

conditions. Transportation Research Part C: Emerg-

ing Technologies, 30(May):116–125.

ohn, T., Gottlieb, M., Schermann, M., and Krcmar, H.

(2019). Improving take-over quality in automated

driving by interrupting non-driving tasks. In Proceed-

ings of the 24th International Conference on Intelli-

gent User Interfaces, volume Part F1476, pages 510–

517, New York, NY, USA. ACM.

Larsson, P., Johansson, E., S

oderman, M., and Thomp-

son, D. (2015). Interaction Design for Communi-

cating System State and Capabilities during Auto-

mated Highway Driving. Procedia Manufacturing,

3(Ahfe):2784–2791.

Li, S., Blythe, P., Guo, W., Namdeo, A., Edwards, S., Good-

man, P., and Hill, G. (2019). Evaluation of the ef-

fects of age-friendly human-machine interfaces on the

CHIRA 2020 - 4th International Conference on Computer-Human Interaction Research and Applications

driver’s takeover performance in highly automated ve-

hicles. Transportation Research Part F: Trafﬁc Psy-

chology and Behaviour, 67:78–100.

McGrath, J. E. and Altman, I. (1966). Small Group Re-

search: A Synthesis and Critique of the Field. Holt,

Rinehart and Winston.

McMahan, A. (2003). Immersion, Engagement, and Pres-

ence: A Method for Analyzing 3-D Video Games. In

The Video Game Theory Reader, chapter 3, pages 67–

86.

Merat, N., Jamson, A. H., Lai, F. C., Daly, M., and Carsten,

O. M. (2014). Transition to manual: Driver behaviour

when resuming control from a highly automated vehi-

cle. Transportation Research Part F: Trafﬁc Psychol-

ogy and Behaviour, 27(PB):274–282.

Naujoks, F., Wiedemann, K., and Sch

omig, N. (2017). The

Importance of Interruption Management for Useful-

ness and Acceptance of Automated Driving. In Pro-

ceedings of the 9th International Conference on Au-

tomotive User Interfaces and Interactive Vehicular

Applications, pages 254–263, New York, NY, USA.

ACM.

Neuhuber, N., Lechner, G., Kalayci, T. E., Stocker, A., and

Kubicek, B. (2020). Age-related differences in the in-

teraction with advanced driver assistance systems - a

ﬁeld study. In Kr

omker, H., editor, HCI in Mobility,

Transport, and Automotive Systems. Automated Driv-

ing and In-Vehicle Experience Design, pages 363–

378, Cham. Springer International Publishing.

Othersen, I. (2016). Vom Fahrer zum Denker und

Teilzeitlenker. Springer Fachmedien Wiesbaden,

Wiesbaden.

Petermeijer, S., Bazilinskyy, P., Bengler, K., and de Winter,

J. (2017). Take-over again: Investigating multimodal

and directional TORs to get the driver back into the

loop. Applied Ergonomics, 62(July 2017):204–215.

Pretto, P., M

ortl, P., and Neuhuber, N. (2020). Fluid in-

terface concept for automated driving. In Kr

omker,

H., editor, HCI in Mobility, Transport, and Automo-

tive Systems. Automated Driving and In-Vehicle Expe-

rience Design, pages 114–130, Cham. Springer Inter-

national Publishing.

Rasmussen, J. (1983). Skills, Rules, and Knowledge; Sig-

nals, Signs, and Symbols, and Other Distinctions in

Human Performance Models. IEEE Transactions on

Systems, Man, and Cybernetics, 13(3):257–266.

SAE International (2018). Ground Vehicle Standard (SAE

J3016:JUN2018): Surface Vehicle Recommended

Practice J3016: (R) Taxonomy and Deﬁnitions for

Terms Related to Driving Automation Systems for

On-Road Motor Vehicles. SAE International.

Sch

omig, N., Hargutt, V., Neukum, A., Petermann-Stock,

I., and Othersen, I. (2015). The Interaction Between

Highly Automated Driving and the Development of

Drowsiness. Procedia Manufacturing, 3(Ahfe):6652–

6659.

Schwind, V., Knierim, P., Haas, N., and Henze, N. (2019).

Using Presence Questionnaires in Virtual Reality. In

Proceedings of the 2019 CHI Conference on Human

Factors in Computing Systems - CHI ’19, pages 1–12,

New York, New York, USA. ACM Press.

Walch, M., Frommel, J., Rogers, K., Sch

ussel, F., Hock,

P., Dobbelstein, D., and Weber, M. (2017). Evalu-

ating VR Driving Simulation from a Player Experi-

ence Perspective. In Proceedings of the 2017 CHI

Conference Extended Abstracts on Human Factors in

Computing Systems - CHI EA ’17, volume Part F1276,

pages 2982–2989, New York, New York, USA. ACM

Press.

Winner, H., Hakuli, S., Lotz, F., and Singer, C., editors

(2015). Handbuch Fahrerassistenzsysteme. Springer

Fachmedien Wiesbaden, Wiesbaden.

Witmer, B. G. and Singer, M. J. (1998). Measuring Pres-

ence in Virtual Environments: A Presence Question-

naire. Presence: Teleoperators and Virtual Environ-

ments, 7(3):225–240.

Wu, Y., Kihara, K., Takeda, Y., Sato, T., Akamatsu, M.,

and Kitazaki, S. (2020). The relationship between

drowsiness level and takeover performance in auto-

mated driving. In Kr

omker, H., editor, HCI in Mobil-

ity, Transport, and Automotive Systems. Driving Be-

havior, Urban and Smart Mobility, pages 125–142,

Cham. Springer International Publishing.

Development of a Test Environment for the Evaluation of Human-Technology Interaction in Cockpits of Highly-Automated Vehicles