Integration of Voice Assistant and SmartDeviceLink to Control Vehicle

Ambient Environment

Ayush Shah

and Anastacia Gusikhin

Ford Research and Advanced Engineering, Dearborn, MI, U.S.A.

University of Michigan, Ann Arbor, MI, U.S.A.

Keywords:

Personal Voice Assistant, Connected Vehicles, Ambient Intelligence, Amazon Alexa, Voice Skills,

SmartDeviceLink.

Abstract:

Over the past few years, the popularity of personal voice assistants has grown, particularly for use in the

vehicle environment where voice is a preferred mode of interface to minimize driving distractions. Amazon

Alexa, one of the most popular voice assistants, has been integrated in many vehicle brands. While existing

Alexa car applications provide vehicle occupants with access to a multitude of voice skills in infotainment and

smart home control, these applications lack the capability to manage the vehicle’s ambient environment. This

paper discusses an efﬁcient and effective integration of Amazon Alexa with vehicle climate control, potentially

augmented with brought-in devices, using SmartDeviceLink API. The paper overviews the architecture, Alexa

skill development, and examples of dialogue. We also present the results of a customer evaluation of the

presented system and directions for future research and development toward Ambient Intelligence.

1 INTRODUCTION

In recent years, Personal Voice Assistants (PVA)

and Ambient Intelligence (AmI) are technologies that

have received substantial attention in academia and

industry (Gams et al., 2019). A personal voice assis-

tant is a digital assistant that uses voice interactions to

aid users through dialogue, invoke infotainment audio

applications or remotely control Internet of Things

devices, such as thermostat, light, TV, etc. Voice As-

sistants help users perform a task with minimal ef-

fort and knowledge of the system. A study conducted

(Ammari et al., 2019) concludes that more that 50 per-

cent of use of voice assistants is for search, music and

the control Internet of Things devices. Well-known

examples of PVA technology are Amazon Alexa, Mi-

crosoft Cortana, Apple’s Siri, and Yandex’s Alice.

Ambient Intelligence (AmI) refers to digital en-

vironments that provide ﬂexibility, adaptation, antic-

ipation and a sensible interface to support people’s

needs (Augusto and McCullagh, 2007). PVA and

AmI technologies are complimentary, heavily rely on

AI and Machine Learning, and support the trend to-

ward human-centered ubiquitous computing. Smart

Homes technology is one of the prominent examples

of PVA and AmI applications. One can use natural

language to interact with voice assistants to perform

tasks like changing the color of lights, changing ther-

mostat settings, playing music, or doing a web search.

The automotive industry has been pursuing in-

telligent driver assistant technology, including voice

applications, for over two decades (Gusikhin et al.,

2008). The early implementations of vehicle intel-

ligence were focused on the driver only and were

constrained by in-vehicle hardware limitations. The

progress in connected car technology has enabled ve-

hicle controls to leverage cloud-based architecture

(Gusikhin et al., 2011) (Siegel et al., 2018) and seam-

less integration of aftermarket sensors and devices

into vehicle ambient environment control. It has al-

lowed to separate fast-paced information and con-

sumer electronics technology applications from the

vehicle development cycle. Connected car technol-

ogy allows the introduction of new capabilities at any

point of a vehicle’s life cycle.

Connectivity allows automakers to efﬁciently

leverage existing popular PVAs, such as Apple’s Siri

as a part of CarPlay, or Google Voice as a part of An-

droid Auto. In recent years, Amazon Alexa has be-

come more and more popular for application to vehi-

cles. It has been widely used as a voice control system

for smart home devices since 2014. There are many

devices that have been developed that are compati-

ble with Amazon Alexa, and there are several ways in

522

Shah, A. and Gusikhin, A.

Integration of Voice Assistant and SmartDeviceLink to Control Vehicle Ambient Environment.

DOI: 10.5220/0009465305220527

In Proceedings of the 6th International Conference on Vehicle Technology and Intelligent Transport Systems (VEHITS 2020), pages 522-527

ISBN: 978-989-758-419-0

Figure 1: System Architecture.

which Amazon Alexa can be used with vehicles, in-

cluding integration with a number of aftermarket de-

vices. Amazon also released an SDK that allows for

tighter integration of Alexa with the vehicle infotain-

ment unit. For the vehicles with an embedded mo-

dem, Alexa allows for integration of vehicle remote

control functionality such as start/stop or lock/unlock.

There is, however, one area of PVA vehicle applica-

tion that has been lacking: the control of vehicle am-

bient environment, particularly climate control.

We believe that vehicle environment is the next

frontier for large scale application of PVA and AmI

technologies. AmI and PVA are clearly promising

for providing superior user experience and personal-

ization. In addition, due to the trend to shift from

personal vehicle ownership to shared mobility, par-

ticularly using autonomous vehicles, AmI and PVA’s

ability to anticipate each new rider’s preferences and

modify vehicle ambiance using natural dialogue in-

teraction will eliminate the daunting task for setting

adjustment for each new ride. One of many beneﬁts

of voice control within shared autonomous vehicles is

that during health concerns like the COVID-19 pan-

demic, for example, the rider does not need to touch

dials or the vehicle’s screen in order to change the

settings. The goal of the presented research is to in-

vestigate the capability, potential merits and user ac-

ceptance of an in-vehicle AmI with PVA. This paper

describes the experimental system that integrates the

Amazon Alexa voice assistant device with vehicle cli-

mate control and other brought-in devices, such as a

scent dispenser or ionizer. The management of ve-

hicle climate control is implemented using SmartDe-

viceLink APIs. The paper presents some results of the

user acceptance survey from representatives of differ-

ent functional areas of the company. We also discuss

the areas of future research and development.

2 VEHICLE SMART

ENVIRONMENT PROTOTYPE

To study user experience with in-vehicle AmI, we de-

veloped a prototype that includes embedded vehicle

climate control, a brought-in scent dispenser by In-

halio (INHALI

O, 2020) and Alexa Voice Services.

The connectivity is provided by a WiFi LTE modem.

The overall architecture of the system is presented

in ﬁgure 1. The individual components of the sys-

tem are linked through Smart Environment Gateway.

Smart Environment Gateway (SEG) is implemented

on the Raspberry Pi. SEG can support multiple con-

nection protocols - in our case WiFi and BLE that

allows SEG to link heterogeneous IoT devices with

different connectivity standards.

We use Amazon Echo Dot as a voice-input de-

vice for Alexa Voice Services. We use Alexa Cus-

tom Skills to implement dialogue to manage vehi-

cle ambience. The output of the skill is connected

to the Amazon Web Services (AWS) Lambda func-

Integration of Voice Assistant and SmartDeviceLink to Control Vehicle Ambient Environment

523

(a) Cartridge.

(b) 3D Printed Container.

Figure 2: Scent Dispenser.

tion. The AWS Lambda function communicates

through MQTT protocol that is connected to the AWS

MQTT broker. When AWS Lambda function pub-

lishes MQTT command, the SEG is triggered, and it

maps the MQTT message to the speciﬁc commands to

control a vehicle embedded subsystem, such as heat-

ing, ventilation, and air conditioning (HVAC) unit,

and control other external IoT devices, such as Inhalio

scent dispenser.

SEG is integrated with vehicle controls using

SmartDeviceLink API. SmartDeviceLink is a plat-

form that enables integration to access and trigger ve-

hicle sensors and actuators. Such a platform can help

develop vehicle features and applications without go-

ing through a vehicle development cycle. SmartDe-

viceLink can also enable a mobile application to con-

trol vehicle modules as cited in the paper (Yeung

et al., 2017). Recently, SDL consortium released Java

SE SmartDevicelink SDK for embedded platforms

that we used in our prototype (SmartDeviceLink Java

Suite, 2020). The example of the Java code to imple-

ment switching AC on is presented below:

public void enableAC(boolean value) {

ClimateControlData cd =

new ClimateControlData();

cd.setACEnabled(value);

cd.setInteriorDataType

(ModuleType.CLIMATE);

ModuleData mdata = new ModuleData();

mdata.setModuleType(ModuleType.CLIMATE);

mdata.setControlData(cd);

SetInteriorVehicleData sd =

new SetInteriorVehicleData();

sd.setModuleData(mdata);

try {

proxy.sendRPCRequest(sd);

} catch (SdlException e) {

e.printStackTrace();

}

SEG is also connected to Inhalio scent dispenser over

WiFi. The scents are incorporated in the cartridges

installed within the device. The device allows to hold

up to four cartridges at the same time (ﬁgure 2). In

our experiment, we used Cool Mint, Pure Odyssey,

Vanilla Dream, and Black Woods scents. The device

is controlled using WiFi commands. Following is the

java code that was used to trigger the Scent dispenser.

int timeOut=60;

Client client = ClientBuilder.newClient();

WebTarget resource = client.target(

"http://192.168.1.1:5551/diffuse?name=

CoolMint&intensity="+String.valueOf(timeOut));

Builder request = resource.request();

request.accept(MediaType.APPLICATION_JSON);

Response response = request.get();

if (response.getStatusInfo().getFamily()

== Family.SUCCESSFUL) {

increaseFanSpeed(timeOut);

}

SEG is connected to both the Inhalo and the vehicle,

which work hand in hand. The advantage of such an

implementation is that it would be possible to con-

nect, read sensor data and trigger actuators of various

devices simultaneously. For example, we can increase

the HVAC fan blower speed when the scent dispenser

is triggered.

3 VEHICLE AMBIENCE VOICE

SKILLS

In this section we will discuss the skill interaction de-

sign model. The code below shows the sample imple-

mentation of the Alexa Skills interaction model. We

used Custom Skills with an invocation word called

“car”. We created Slots for various nouns and verbs

which can be used in sentences. Each slot had all

the different alternative words for that word or phrase.

For example, AC can also be called heater, HVAC, Air

Conditioner, Fan, cooling system and so on. These

slots were then used in the intents. As shown in

the code, custom Slot “ACsyn” is used in the intent

“HVACTurnOnAC.” This helped us create a rich and

robust set of utterances due to all of the available com-

binations. The user can say a phrase using natural

language, (a sentence, which contains at least a noun

and a verb) and the program will understand numer-

ous different variations of the same phrase due to the

different sentence structures provided in the list of ut-

terances and the slots for each noun, verb, and even

preposition depending on the sentence’s structure. We

VEHITS 2020 - 6th International Conference on Vehicle Technology and Intelligent Transport Systems

524

Figure 3: Sequence Diagram.

also added a help intent which works if the skill is in-

voked but an appropriate intent is not triggered. It

would explain to the user how they can phrase the ut-

terances to actuate the model or get sensor data.

{"interactionModel": {

"languageModel": {

"invocationName": "my car",

"intents": [

{"name": "HVACTurnOnAC",

"slots": [

{"name": "Ac",

"type": "ACsyn"}],

"samples": [

"power on the {Ac}",

"switch on {Ac}",

"activate the {Ac}",

...]},

{"name": "SmellOnIntent",

"slots": [],

"samples": [

"make it smell good ",

"I want a good smell",

"I want it to smell good",

...]}],

...

"types": [

{"name": "ACsyn",

"values": [

{"name": {

"value": "AC",

"synonyms": [

"a. c.",

"fan speed",

"fan",

"air ",

"air conditioning"]}

}]}]}}}

The responses and follow-up questions to the user are

randomly selected, which allows for the exchange be-

tween the user and the device to be much like a con-

versation as opposed to a mechanical response. For

instance, if a user asks to increase the fan speed, the

skill can ask a question, which may be phrased dif-

ferently each time. Some sample questions include

“Would you like to turn on AC?”, “Shall I switch on

the AC?”, ”How about turning on the AC?”. In addi-

tion, responses are also randomly chosen from the list,

which include “sure!”, “ok”, “done”,” got it”, “there

we go”, etc. The design of the model was focused to

provide a natural language interaction. Whenever a

user interacts with the skill, it generates the request

to change ambient environment and responds with a

conﬁrmation or a follow-up question. For example, a

user says, “tell my car to activate air conditioning”.

In this case, SEG would turn on the air conditioning

and then Alexa would ask the user “what temperature

would you like HVAC to set to?” In this implemen-

tation, we would ask or respond to a user command

randomly. Whenever a follow-up question is asked

to a user, a session is created to keep track of all the

inputs the user is providing corresponding to the ques-

tion. Based on these user answers, SEG would actuate

the corresponding modules. As seen in the sequence

diagram (ﬁgure 3), initially the AWS Lambda Func-

tion would establish a connection with AWS MQTT

Broker. Then SEG would establish connection and

subscribe to all the topics at startup. Once a user in-

teracts with Alexa, for example by saying “Alexa tell

my car to turn on AC,” the Lambda function would

be called and the corresponding intent would be trig-

gered. In this case, it is the “HVACTurnOnAC” intent.

The Lambda function would then publish a message

on topic “/ford/ac” with data ”ON.” SEG would re-

Integration of Voice Assistant and SmartDeviceLink to Control Vehicle Ambient Environment

525

Figure 4: Proposed System Architecture.

ceive a message to turn on the Air Conditioner and it

would perform respective operations. A response or

follow-up question is asked, which is triggered from

the Lambda function.

4 CUSTOMER EVALUATION AND

FUTURE WORK

The system prototype has been presented at the In-

ternal Technology show as a part of autonomous ve-

hicle user experience. The evaluators were from dif-

ferent organizations of the company. All evaluators

provided feedback on the experience with the technol-

ogy. Everybody responded to the question: “Do you

agree this concept will improve the customer experi-

ence for private customers?” The feedback was over-

whelmingly positive. Figure (ﬁgure 5) shows a quad-

rant graph that represents the average score among

each group and to what extent the group participants

agreed with each other. Quadrant I: low score, high

agreement; II: low score, high spread in answers, III:

high score, high spread in answers; IV: high score,

high agreement. Quadrant IV is the most important.

The size of the bubble is determined by the size of the

group.

It is interesting that the representatives from non-

engineering functions, such as marketing, ﬁnance,

global programs, and general management expressed

more excitement than the engineering community,

such as PD, R&A and AV.

Based on the experience with the initial prototype

and feedback from the customer evaluation, we iden-

tiﬁed the several directions that need to be addressed

in the future.

One of the future enhancements of the dialogue

system is to eliminate the need for the invocation

word ”my car”. In the current implementation, we

used custom skills that provide a simple and ﬂexi-

ble way to relatively quickly try different scenarios,

but they require a key word to invoke the skill. Al-

ternately, we could use Smart Home Skills and treat

Smart Environment Gateway, discussed earlier in the

paper, as an IoT device with various features like

HVAC, media, radio, Ambient Light, etc. Each indi-

vidual SEG would have to be registered with the OEM

Cloud. A high-level architectural diagram is shown

in ﬁgure 4) where all the commands from Amazon

Lambda function would go to the OEM cloud. OEM

would then trigger the corresponding vehicle SEG us-

ing a unique key/identiﬁer.

Figure 5: Evaluation.

VEHITS 2020 - 6th International Conference on Vehicle Technology and Intelligent Transport Systems

526

To provide a more personal experience, the customers

can connect their Voice Assistant account to the SEG.

Then the user can play their favorite music, open

their calendar, turn on/off Smart Home devices and

leverage other user-speciﬁc services associated with

a given account. In the case of ride-sharing applica-

tions, the user proﬁle can be dynamically associated

with a given vehicle. When a new passenger is picked

up, a session on Alexa is created with the given user’s

proﬁle. The session is cleared when the ride ends.

Another promising area of future enhancement is

provisioning the integration with wearables (Gusikhin

et al., 2016). Typically, wearables provide at least one

type of biometric sensor, such as a heart rate moni-

tor, skin temperature sensor, or blood oxygen sensor.

Using this biometric data, the system can parameter-

ize the intensity of the action. For example, if a user

requests “Increase temperature,” the current version

of the system would always increase the temperature

by 5 degrees. If skin temperature is available, the in-

crease of the cabin temperature can vary based on the

sensor data.

The critical aspect of intelligence is the ability to

learn user preferences to anticipate and automatically

adjust ambience based on the context. While our cur-

rent prototype is focused on dialogue-based interac-

tion, the next step in the system development would

be to implement machine learning capability.

5 CONCLUSIONS

The paper discusses the opportunities and beneﬁts

of Ambient Intelligence within vehicle environment.

The advancement of intra- and inter- vehicle connec-

tivity technology, proliferation of connected devices

and sensors, and the emergence of Personal Voice As-

sistants enable the efﬁcient implementation of vehi-

cle AmI even as an aftermarket feature. To illustrate

this point, the paper presents a prototype that inte-

grates Amazon Alexa personal voice assistant with

vehicle in-cabin ambient control. The ambient control

is exempliﬁed by vehicle climate control and brought-

in scent dispenser by Inhalio. The proposed system

provides the capability for easy interaction to adjust

in-cabin ambiance using commonly used natural lan-

guage expressions and commands.

The prototype of the system has been evaluated

at the Internal Technology show. The evaluation has

been done by representatives from different groups

representing different company functions, including

engineering and non-engineering. The evaluation

shows that this functionality has been positively per-

ceived. The results of the study showed that non-

engineering functions had more positive responses

than engineering.

The paper also discusses potential future direction

of the development to further enhance and simplify

the interactions. Another area of enhancement in-

cludes the integration with machine learning to facil-

itate ambient intelligence to anticipate the user needs

based on the prior experience and environmental con-

ditions.

REFERENCES

Ammari, T., Kaye, J., Tsai, J. Y., and Bentley, F. (2019).

Music, Search, and IoT: How people (really) use voice

assistants. ACM Transactions on Computer-Human

Interaction.

Augusto, J. C. and McCullagh, P. J. (2007). Ambient intel-

ligence: Concepts and applications. Comput. Sci. Inf.

Syst., 4:1–27.

Gams, M., Gu, I. Y.-H., H

arm

a, A., Mu

noz, A., and Tam,

V. (2019). Artiﬁcial intelligence and ambient intel-

ligence. Journal of Ambient Intelligence and Smart

Environments, 11:71–86.

Gusikhin, O., Filev, D., and Rychtyckyj, N. (2008). Intel-

ligent vehicle systems: applications and new trends.

In Informatics in Control Automation and Robotics,

pages 3–14. Springer.

Gusikhin, O., Klampﬂ, E., Filev, D., and Chen, Y. (2011).

Emotive driver advisor system (EDAS). In Lecture

Notes in Electrical Engineering.

Gusikhin, O., Makke, O., Yeung, J., and MacNeille, P.

(2016). SmartDeviceLink application to intelligent

climate control. In ICINCO 2016 - Proceedings of

the 13th International Conference on Informatics in

Control, Automation and Robotics.

INHALI

O (2020). The world’s 1st connected platform for

scent. https://inhalio.com/, accessed 2020-01-10.

Siegel, J. E., Erb, D. C., and Sarma, S. E. (2018). A survey

of the connected vehicle landscape—architectures,

enabling technologies, applications, and development

areas. IEEE Transactions on Intelligent Transporta-

tion Systems, 19(8):2391–2406.

SmartDeviceLink Java Suite (2020). Smartdevicelink

libraries for android, java se, and java ee.

www.github.com/smartdevicelink/sdl java suite,

accessed 2020-01-10.

Yeung, J., Makke, O., MacNeille, P., and Gusikhin, O.

(2017). SmartDeviceLink as an Open Innovation Plat-

form for Connected Car Features and Mobility Appli-

cations. SAE International Journal of Passenger Cars

- Electronic and Electrical Systems.

Integration of Voice Assistant and SmartDeviceLink to Control Vehicle Ambient Environment

527