Real-Time Sonification of Motor Coordination to Support Motor

Skill Learning in Sports

Toshitaka Kimura

, Takemi Mochida

, Tetsuya Ijiri

and Makio Kashino

1,2

Human Information Science Laboratory, NTT Communication Science Laboratories, Kanagawa, Japan

CREST, Japan Science and Technology Agency, Saitama, Japan

1 OBJECTIVES

It is essential that we recognize our own posture and

body movement if we are to acquire and improve

goal-directed actions such as sports. However, it is

not easy to adequately realize one’s own state where

multiple body segments are temporally coordinated.

Although visual feedback (e.g., snapshots and video

streaming) has been the most widely used way of

displaying motor information (reviewed by Sigrist et

al. 2013), they are expected to be limited in terms of

showing the temporal structure of segmental activity

because of the insufficient temporal resolution of

human visual perception compared with that of

auditory perception. In fact people often utilize

auditory feedback to realize temporal features of

motor behavior in daily life such as speech

articulation (Sasisekaran 2012). The sonification of

motor information is designed to extend the human

auditory system so that it can utilize motor control in

limbs in addition to speech. It is expected that the

artificially provided auditory feedback integrates

somatosensory information in action and such

integration enhance motor learning.

The results of previous studies have already been

applied to sports training, but most of them have

sonified the behavioral outcomes (e.g., ski

displacement, Kirby 2009) or individual segmental

activity (e.g., the timing of wrist and ankle

movement in karate, Yamamoto et al. 2004). Instead

our idea is to sonify the temporal coordination of

multiple body segments, that is, how the body is

controlled. In this study we propose the sonification

of segmental coordination in sports action using

muscle activity and acceleration signals in multiple

target segments.

2 METHODS

To realize auditory feedback motor coordination in

sports action we used surface electromyography

(EMG) and acceleration signals, which we recorded

at sampling rates of 2000 Hz and 150 Hz,

respectively, by using a wireless sensor (Trigno

Wireless EMG, DELSYS, USA) attached to the

surface of the target muscles. The recorded EMG

and acceleration signals were delivered to a PC at 10

kHz and were processed for conversion into sound

using the custom-made software developed in

Processing 2.

3 RESULTS

For most sports it is critical to coordinate the activity

of multiple muscles sequentially. For instance, in

baseball pitching, a pitcher needs to operate his/her

muscles in sequence from the leg to the arm, which

is well known as a segmental kinetic chain (Feltner

1989). Thus we attempted to sonify the precision of

such sequential muscle activity patterns, based on

multiple EMG signals from the lower and upper

limb muscles.

Figure 1: Sonification of the “precision” of pitching

coordination.

Figure 1 demonstrates an example of the

sonification of the “precision” of pitching

coordination. Good pitching almost always includes

clear leg and arm muscle activity (left panel in Fig.

Kimura T., Mochida T., Ijiri T. and Kashino M..

Real-Time Soniﬁcation of Motor Coordination to Support Motor Skill Learning in Sports.

 2014 SCITEPRESS (Science and Technology Publications, Lda.)

1A), while bad pitching frequently does not (left

panel in Fig. 1B) indicating the poor precision of

muscle coordination. We aimed to convert this

precision of coordination into sound. For good

pitching, the sound signal consisted of a chirp

waveform. Its fundamental frequency was initially

low. It gradually increased after the onset of EMG

activity was detected in the thigh muscle, and then

gradually decreased after the onset of EMG activity

in the wrist muscle (right panel in Fig. 1A). In

contrast, for bad pitching where the thigh EMG

activity was insufficient, the first increase in the

chirp waveform was not present (right panel in Fig.

1B). The pitcher could hear a wind-like sound with

good pitching, but not with bad pitching.

Figure 2: Sonification of excessive muscle activation

during preparation for putting in golf.

A sports player often could not perform well

because he/she was trying too hard. This

corresponds to unintentional excessive activation of

the muscles caused by mental pressure, and it is very

common and serious in sports. Thus it would be

useful to make an athlete aware of this potential for

excessive muscle activation.

Figure 2 shows an example of the sonification of

excessive muscle activation in the initial stages

(address and take back) of putting in golf, based on a

wrist extensor EMG and acceleration signals. The

sound signal consisted of a periodic waveform

whose amplitude and fundamental frequency were

varied in proportion to the short-term power of the

EMG activity only when the acceleration was below

a certain threshold, while these values were kept at

almost zero when the acceleration was above the

threshold. Therefore, the golfer could hear sounds

during the preparation (address) period when the

muscle was activated more than necessary while the

wrist moved slightly (Fig. 2B). Conversely, he/she

could not hear these sounds when the muscle was

relaxed (Fig. 2A). No sound was produced when the

wrist was moving (taking back).

4 DISCUSSION

Although this study showed two examples of the

sonification of the temporal coordination of multiple

segmental activities in sports action, the combination

of segmental information and auditory ways of

presenting them should vary according to the

objective of the players. For example, the

sonification of the precision of segmental activities

(Fig. 1) may be more useful to a novice or an

intermediate player for improving his/her basic

temporal pattern of a target form, while the

sonification of excessive muscle activation (Fig. 2)

is beneficial as regards recognizing one’s own

mental state during a given action.

We should consider other elements of sonification.

For instance, it is very important to detect the key

features of motor coordination in a target action. It is

also critical to assess the effectiveness of auditory

feedback. Further empirical studies are needed to

help a player learn a desired sports-related motor

skill effectively.

REFERENCES

Feltner, ME., 1989. Three-dimensional interactions in a

two-segment kinetic chain. II. Application to the

throwing arm in baseball pitching. International

Journal of Sport Biomechanics 5: 420–450.

Kirby, R., 2009. Development of a real-time performance

measurement and feedback system for alpine skiers.

Sports Technology. 2: 43-52.

Sasisekaran, J., 2012. Effects of delayed auditory feedback

on speech kinematics in fluent speakers. Perceptual

and Motor Skills. 115: 845–864.

Sigrist, R., Rauter, G., Riener, R., Wolf, P., 2013.

Augmented visual, auditory, haptic, and multimodal

feedback in motor learning: a review. Psychonomic

Bulletin & Review. 20: 21-53.

Yamamoto, G., Shiraki, K., Takahata, M., Sakane, Y.,

Takebayashi, Y., 2004. Multimodal knowledge for

designing new sound environments. In The

International Conference on Human Computer

Interaction with Mobile Devices and Services. Citeseer.