Large Language Model-Informed Geometric Trajectory Embedding for Driving Scenario Retrieval

Tin Sohn, Maximilian Dillitzer, Maximilian Dillitzer, Tim Brühl, Robin Schwager, Tim Dieter Eberhardt, Michael Auerbach, Eric Sax

2025

Abstract

This paper introduces a Large Language Model-informed geometric embedding for retrieving behavioural driving scenarios from unlabelled trajectory data, aimed at improving the search of real driving data for scenario-based testing. A Variational Recurrent Autoencoder with a Hausdorff Distance-based loss generates trajectory embeddings that capture detailed spatial patterns and interactions, offering enhanced interpretability over traditional mean squared error-based models. The embeddings are further organised through unsupervised clustering using HDBSCAN, grouping scenarios by similarities at the scene, infrastructure, behaviour, and interaction levels. Using GPT-4o for describing scenarios, clusters, and inter-cluster relationships, the approach enables targeted scenario retrieval via a Graph Retrieval-Augmented Generation pipeline, enabling a natural language search of unlabelled trajectories. Evaluation demonstrates a retrieval precision of 80.2% for behavioural queries involving infrastructure, multi-agent interactions, and diverse traffic conditions.

Paper Citation

in Harvard Style

Sohn T., Dillitzer M., Brühl T., Schwager R., Eberhardt T., Auerbach M. and Sax E. (2025). Large Language Model-Informed Geometric Trajectory Embedding for Driving Scenario Retrieval. In Proceedings of the 11th International Conference on Vehicle Technology and Intelligent Transport Systems - Volume 1: VEHITS; ISBN 978-989-758-745-0, SciTePress, pages 66-75. DOI: 10.5220/0013276500003941

in Bibtex Style

@conference{vehits25,
author={Tin Sohn and Maximilian Dillitzer and Tim Brühl and Robin Schwager and Tim Eberhardt and Michael Auerbach and Eric Sax},
title={Large Language Model-Informed Geometric Trajectory Embedding for Driving Scenario Retrieval},
booktitle={Proceedings of the 11th International Conference on Vehicle Technology and Intelligent Transport Systems - Volume 1: VEHITS},
year={2025},
pages={66-75},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013276500003941},
isbn={978-989-758-745-0},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 11th International Conference on Vehicle Technology and Intelligent Transport Systems - Volume 1: VEHITS
TI - Large Language Model-Informed Geometric Trajectory Embedding for Driving Scenario Retrieval
SN - 978-989-758-745-0
AU - Sohn T.
AU - Dillitzer M.
AU - Brühl T.
AU - Schwager R.
AU - Eberhardt T.
AU - Auerbach M.
AU - Sax E.
PY - 2025
SP - 66
EP - 75
DO - 10.5220/0013276500003941
PB - SciTePress