Visual Navigation for Rendezvous and Docking using PMD Camera

Ksenia Klionovska and Heike Benninghoff

German Aerospace Center (DLR), Muenchener Str. 20, 82234 Wessling, Germany

1 RESEARCH PROBLEM

There is one common problem of the satellite which

is quite widely discussed: the lifespan. Depending

on the mission the lifespan of large and expensive

geostationary satellites can last about 10-15 years,

whereas the satellites in Low Earth Orbit have

significant shorter life cycles. Therefore an idea to

extend the operational lifetime or improve the

performance of a satellite on the orbit instead of

replacing it by a new one is concluded in On Orbit

Servicing (OOS) projects (Ellery et al., 2008; Stoll

et al., 2009).

The typical scenario of OOS is Rendezvous and

Docking (RvD) of the approaching chaser to the

target and taking over attitude control. There have

been several studies focused on providing OOS

(Figure 1): DEOS (DEutsche Orbital Servicing

mission) for Low Earth Orbit and Smart-OLEV

(Orbital Life Extension Vehicle) for geostationary

orbit. In DEOS like scenarios, a service satellite

approaches a tumbling client in Low Earth Orbit and

uses a robotic manipulator in order to capture the

uncooperative target satellite (Wolf et al., 2012;

Boge et al., 2013). In the OLEV scenario the

servicer satellite will dock with a specific docking

tool to the apogee thruster of client satellite in the

geostationary orbit (Kaiser et al., 2008).

Figure 1: DEOS Servicer capturing a tumbling client

satellite in Low-Earth Orbit (left) and OLEV approaching

a geostationary client satellite (right).

The rendezvous phase starts at the distance

where the navigation process switches from absolute

navigation to the relative navigation. For this

purpose, a variety of 3D-vision rendezvous and

docking approaches based on laser scanners, stereo

vision, structured light and LIDAR (LIght Detection

And Ranging) has been developed. In 2000, the new

product Photonic Mixer Device (PMD) camera

appeared at the market as a new sort of the 3D Time-

of-Flight (ToF) sensor which, moreover, has never

been used in space environment (Schilling and

Regoli, 2011). The main feature distinguishing PMD

cameras from the above described systems is the

following. A PMD camera emits modulated light to

derive by phase shift a distance measurement for

every pixel. The PMD technologies are rather new

and have some drawbacks. A general problem of the

PMD sensors is a limited distance range due to the

measurement principle. Also an output data provided

by PMD sensor is affected by many factors and quite

noisy with different side effects. These side effects

cannot be neglected, as they influence the system

performance. Therefore, further research in this

direction with a prerequisite for the future to use

PMD camera in space environment is of great

interest to date.

2 OUTLINE OF OBJECTIVES

The main focus of this work is to investigate the use

of a PMD camera for motion detection and pose

estimation with regard to space applications. During

the research phase the following points are

considered:

 Initial pose estimation with a subsequent real-

time object tracking.

 Extension of the measurement range of the PMD

sensor. In a work of Tzschichholz (Tzschichholz,

2014), the measurement range was extended with

an appropriate algorithm up to 75 m.

 Data preprocessing. Camera calibration.

 Performance measurements with a RvD

simulation facility like EPOS 2.0 (European

Proximity Operations Simulator)

This paper mainly focuses on the first objective,

pose estimation with a PMD sensor.

Klionovska, K. and Benninghoff, H.

Visual Navigation for Rendezvous and Docking using PMD Camera.

In Doctoral Consortium (DCGISTAM 2016), pages 3-7

3 STATE OF THE ART

As it was already mentioned, the use of a PMD

camera for space applications for visual navigation

is of interest up to date. Therefore, there is no

variety of prototypes of ToF cameras. However,

some have been applied and tested under quite

realistic space environment. Tzschichholz

(Tzschichholz, 2011) and Schilling (Schilling and

Regoli, 2011) used PMDTec 19 k for pose

estimation and motion prediction of the spacecraft.

Later, Tzschichholz (Tzschichholz, 2014) applied a

newer model of a ToF camera, the PMDTec

CameCube 3.0, used in conjunction with a CCD

camera for rendezvous and docking.

3.1 Available ToF Cameras

The previous prototypes of the PMD cameras have a

low resolution of the sensor chip and camera

measurements are subjected to external influences.

Especially the pixels of a sensor are oversaturated

under strong illumination of background light.

Table 1: Available ToF cameras and their relevant

properties.

Camera

Resolution

[pixel]

Max. range

[m]

Field of view

Argos 3D,

Bluetechnix

Group

GmbH

160 x 120

352 x 288

up to 10

Indoor,

up to 3

Outdoor

30°,60°,90°

110°

iZ™ OIVS-

1000,

United

Kingdom

1280×1024 0.5-10 15°, 30°,60°

Fotonic,

Sweden

160 x 120 0.15 - 10

70º x 53º

45º x 34º

Ifm,

O3D303,

Germany

176 X 132

64x50 64x16

up to 30

Indoor

60°x45°

40° x 30°

Since the science in different fields moves

forward and new technologies are developed, it was

rationally to start exploring new hardware devices

with more ameliorated characteristics. Products with

their characteristics, which could be presently

purchased from companies, are depicted in Table 1.

For future tests and experiments, we intend to use

the Argos 3D sensor with a resolution of 352 x 288

pixels and with a field of view of 30°.

3.2 PMD Principal

PMD camera is a ranging imaging system, based on

the time-of-flight principle (Boge et al., 2013). The

camera measures the distance to the object for every

pixel of its sensor chip. Figure 2 depicts the distance

measurements, where color indicates distance in

meters.

Figure 2: 3D image obtained with PMD camera showing a

satellite mockup in EPOS laboratory.

The target area is illuminated by incoherent

modulated light (LED or laser diodes), where the

wavelength of the light source has to be

synchronized with the spectral sensitivity of the

detector. ToF cameras usually employ NIR light in

range of 780nm to 850nm (Fuchs, 2012). In order to

reduce the impact of the background illumination, it

would be also practically to use a narrow band-pass

filter. The light is reflected by the scene, collected

by the lens system of the camera and projected onto

the sensor matrix. The distance from the target is

calculated by correlation of the received modulation

light with the modulation signal of the illumination

unit, where the phase shift corresponds to the

distance. Noise from the background light can be

suppressed by the Suppression of Background Light

(SBI) circuitry but SBI cannot be controlled by

software. In the Figure 3, the functionality of the

PMD principal is depicted.

Figure 3: Measurement principal of the PMD camera.

4 METHODOLOGY

Before calculating the pose estimate, the problem of

DCGISTAM 2016 - Doctoral Consortium on Geographical Information Systems Theory, Applications and Management

actually finding target must be solved. The

initialization need not be accurate. The coarse

capture of the model is sufficient for the further

tracking process. In work of Tzschichholz

(Tzschichholz, 2014),

the author uses the amplitude

channel of the PMD camera to locate the target

object in image-space and then expands a point

cloud in 3D space from a bootstrap point. The

resulting point cloud provides a centroid, which is

used to determine the translational components of

the relative pose. A sequential probing of all the

model surfaces to the point cloud provides the planar

orientation of the matching model surface, and

finally, the determination of the principal

components of the measured point cloud provides

the rotation about the normal vector, what completes

the initial relative pose estimate. The algorithm

suggested by Tzschichholz has one drawback,

namely it does not reliably determine all six degrees

of freedom of a symmetric object without additional

hints to the initializer. The hints are some pieces of

information which allow the initializer to narrow

down the set of possible poses until only one pose

remains.

The key idea of the matching method is quite

simple: features found in the image are matched

against the set of features in the model. The

algorithm proposed in this paper is addressed to the

free-form 3D objects and discussed in section 5.

4.1 Related Work

To this date, many techniques have been developed

for the purpose of object recognition and pose

estimation using 3D data with the focus to find a

robust and efficient manner to identify objects in a

scene.

This can be done in various ways, depending

on the format of the scene: camera image, range

image, or a combination of both.

One standard

approach for object pose estimation is the

Iterative

Closest Point (ICP) method. The ICP algorithm

needs a good initial estimate and is thus not suitable

for initial object detection, but often used as a

subsequent pose refinement step after the object is

detected (Haarbach, 2015).

The techniques for detecting free-form models

could be divided in two groups: global methods and

local methods (Drost et al., 2010).

4.1.1 Global Methods

The idea of global methods is to not divide a model

into parts or any geometrical objects. Instead, the

complete model is used. To initialize objects the so-

called Generalized Hough Transform is suggested in

(Rabbani and van den Heuvel, 2005) but it is limited

to primitive objects as the recognition of 6 degrees

of freedom (DoF) is computationally too expensive.

Another automatic routine based on random

sample consensus (RANSAC) to find basic shapes

with point cloud input was designed in (Schnabel et

al., 2007). The input data is a set of observed values,

sampled and deconstructed in a model which can be

fitted to the observation. The inliers are generated

from random subset of the original data and tested

against entire set of data in order to determine

correlation between inliers and data set. The

algorithm is proved to be simple and easy to

implement for the primitive shapes but for complete

objects it would not be the best solution.

4.1.2 Local Methods

A second class of methods, local methods, uses so-

called point descriptors. The scheme of this type of

algorithms identifies point to point correspondence

between model and the scene. Point correspondence

is built by comparing descriptors of the scene and of

the model. Therefore, the descriptors must

accurately and robustly distinguish between cluster

and noise.

Mian (Mian et al., 2006) built a three-

dimensional tensor from multiple unordered range

images. The tensors are stored in a hash table and

used like a lookup table during the matching phase.

In (Johnson, A. and Hebert, M., 1999) Johnson and

Hebert introduce a recognition algorithm based on

surfaces’ correspondence by matching points using

the spin image representation. Spin images from

points on one surface (model) are constructed and

stored, the same procedure is done with the spin

image of scene surface. Best point correspondence is

then established and grouped for the further surface

matches.

5 CONTRIBUTION

Object recognition, pose estimation and navigation

for the RvD demand factors such as accuracy,

robustness and also computing speed. The approach

outlined below addresses the problem of estimation

of the 6 DoF object pose (3D position and 3D

orientation) using PMD camera measurements in

cluttered point clouds. Prior knowledge of the

geometry of the object (e.g. a given CAD model or

any other 3D model) is needed to be known.

The method developed in the thesis is the

Visual Navigation for Rendezvous and Docking using PMD Camera

following: A global model description is created

using an oriented point feature. It is matched with

the measurement data by using a fast voting scheme,

similar to the Generalized Hough Transform (Drost

et al., 2010). An oriented point is a three-

dimensional vertex with a directional vector (normal

vector). Consequently, point pair feature describes

the relative position and orientation of two oriented

points. Global model description consists of all

model point pair features (single four-dimensional

vector 



). Vector 



is determined as follow:

m





‖

,∠





,∠n

,d,∠n



(1)

where d is difference between two points,

‖



‖

represents Euclidean distance, the second and third

components are angles between the vector d and the

point normal vector 



and 



, respectively.

Equal point pair features grouped together in

order to reduce dimension of the hash tables for

voting process. An object, in general, is represented

as a set of feature vectors spaced in a hush table for

the further simple retrieval. Random points should

be sampled from the PMD data and each pair votes

for a particular pose. The pose which collects the

most votes belongs to the required pose. In order to

increase the accuracy of the estimated pose, we

suggest to apply at the previous step the hierarchical

clustering algorithm, for example, Agglomerative

Clustering, and to group together similar poses.

Figure 4: Visualization of the steps in proposed algorithm.

The similarity of the poses is determined by

comparison with a predefined threshold for rotation

and translation components. The method consists of

the following steps (cf. Figure 4):

1. Let 



∈ denote points of the scene and 



∈

 points of the model. The reference point 



paired with all other points of the scene S and

their feature 







,



 is calculated using the

same principal as for vector 



2. The global model description consisted of all

model point pair features 







,



 is

calculated and generated in an offline phase.

Feature 



is matched to the global model

description, which output is a set of point pairs

on the model M that have similar distances and

orientations.

3. For every point pair of the model matched to the

point pair of the scene, the local coordinate α is

calculated. The rigid motion from the model

space into the scene space can thus be described

by a point on the model and a rotation angle α.

4. After α is calculated, the vote is casted in a so-

called accumulator space, which is similar to a

hash table.

5.1 Expected Outcome of Algorithm

As the algorithm for pose initialization and

estimation of the 6 DoFs is still in investigation

phase, it is expected that the novel algorithm

achieves significant improvement compared to the

method of Tzschichholz (Tzschichholz, 2014) in

terms of efficiency, stability and accuracy. Once a

pose estimate is available, the next step is planned to

develop an approach for tracking the target for static

and dynamic cases.

6 EXPECTED OUTCOME

In this chapter some description of the additional

research objectives is given.

6.1 Data Preprocessing

Before the delivered data from PMD sensor can be

used for navigation tasks, the camera should be

calibrated. For this, the following effects and issues

must be taken in account when calibrating a PMD

camera (Tzschichholz et al., 2011):

 fixed pattern noise,

 amplitude dependent distance offset,

 Wiggling effect,

 integration time dependent distance error,

 motion artifacts.

6.2 Extension of Measurement Range

of PMD Sensor

Usually, in PMD cameras the modulation frequency

can be tuned accordingly with regard to the needs of

the user or the application. Lower modulation

frequencies allow larger measurement ranges. There

are also methods which can extend the measurement

range of the PMD camera, for example, by using

two or more modulation frequencies (Tzschichholz,

2014). Therefore, an appropriate approach in

DCGISTAM 2016 - Doctoral Consortium on Geographical Information Systems Theory, Applications and Management

accordance with a new PMD camera’s

characteristics is a part of the study.

6.3 Performance Measurements with

EPOS Facility

In order to ensure safe and reliable rendezvous and

docking on the orbit by OOS, the processes must be

analyzed and simulated under utmost realistic

conditions with respect to space environment. For

this purpose, simulations and tests of the PMD

camera and the navigation algorithms will be

performed using the European Proximity operations

simulator (EPOS 2.0), a new simulation facility

located at DLR, Oberpfaffenhofen, for this purpose.

It is a hardware-in-the-loop simulator which

comprises two industrial robots for physical real-

time simulations of rendezvous and docking

maneuvers (Boge et al., 2013). For such hardware-

in-the loop RvD simulation, a client satellite mockup

is mounted on one robot of the EPOS facility and the

PMD camera is mounted on the second robot.

The

PMD camera measures the relative position and

attitude of the client satellite and the onboard

attitude and orbit controller calculates on this basis

the necessary thrusters or reaction wheel commands.

7 STAGE OF THE RESEARCH

At the moment study and investigation of the

intended algorithm for pose initialization and

estimation described in section 5 are conducted.

Intermediate steps are implemented and tested using

a Matlab toolbox. In parallel, the process of

collecting data from the new purchased sensor for

subsequent conduction of experiments is ongoing as

well as decimation procedure for the raw data from

the PMD camera.

REFERENCES

Boge, T., Benninghoff, H., Tzschichholz, T., 2013.Visual

Navigation for On-Orbit Servicing Missions. 5th

International Conference on Spacecraft Formation

Flying Missions and Technologies.

Choi, C.,Taguchi, Y., Tuzel, O., Liu, M-L., Ramalingam,

S., 2012. IEEE International Conference on Robotics

and Automation (ICRA.).

Drost, B., Ulrich, M., Navab, N., Ilic, S., 2010. Model

Globally, Match Locally: Efficient and Robust 3D

Object Recognition. IEEE Conference Computer

Vision and Pattern Recognition (CVPR.)

Ellery, A., Kreisel, J., Sommer, B., 2008. The case for

robotic on-orbit servicing of spacecraft: spacecraft

reliability is a myth. Acta Astronautica 63(5-6).

Fuchs, S., 2012. Calibration and Multipath Mitigation for

Increased Accuracy of Time-of-Flight Camera

Measurements in Robotic Applications. Dissertation,

Technical University of Berlin.

Haarbach, A.,2015. 3D Object Reconstruction using Point

Pair Features. Bachelor’s thesis, Technical University

of Munich.

Johnson, A., Hebert, M., 1999. Using Spin Images for

Efficient Object Recognition in Cluttered 3D Scene.

IEEE Transactions on pattern analysis and machine

intelligence, vol. 21 , no. 5.

Kaiser C., Sjöberg, F., Delcura, J. M., Eilertsen, B., 2008,

SMART-OLEV – An orbital life extension vehicle for

servicing commercial spacecrafts in GEO, Acta

Astronautica 63.

Lonsberry, A., 2012. Fast recognition and pose estimation

for the purpose of bin-picking robotics. Master thesis,

Case Western Reserve University, USA.

Mian, A,S., Bennamoun, M., Owens, R., 2006. Three-

Dimensional Model-Based Object Recognition and

Segmentation in Cluttered Scenes. IEEE Transactions

on pattern analysis and machine intelligence, vol. 28,

no. 10.

Rabbani, T.,van den Heuvel, F., 2005. Efficient Hough

Transform for Automatic Detection of Cylinders in

point Clouds. ISPRS WG III/3, III/4, V/3 Workshop

"Laser scanning 2005", Enschede, the Netherlands.

Schilling, K., Regoli, L., 2011. The PMD 3D-camera for

Rendezvous and Docking with Passive Objects. In 8

International ESA Conference on Guidence,

Navigation& Control Systems.

Schnabel, R., Wahl, R., Klein R., 2007. Efficient

RANSAC for Point-Cloud Shape Detection. The

Eurographics Association and Blackwell Publishing.

Stoll, E., Letschnik, J., Walter, U., Artigas, J., Kremer, P.,

Preusche C., Hirzinger, G., 2009. On-orbit servicing.

IEEE Robotics & Automation Magazine 16(4).

Tzschichholz, T., 2014. Relative pose estimation of known

ridged objects using a novel approach to high-level

PMD-/CCD- sensor data fusion with regard to

applications in space. Dissertation, University of

Würzburg, Germany.

Tzschichholz, T., Ma, L., Schilling, K., 2011. Model-

based spacecraft pose estimation and motion

prediction using a photonic mixer device camera. Acta

Astronautica 68, (2011).

Wolf, T., Reintsema, D., Sommer, B., Rank, P., Sommer,

J., 2012. Mission DEOS – Proofing the Capabilities of

German's Space Robotic Technologies. International

Symposium on Artificial Intelligence, Robotics and

Automation in Space–i-SAIRAS.

Visual Navigation for Rendezvous and Docking using PMD Camera