Visual-Interactive Similarity Search for Complex Objects by Example of Soccer Player Analysis

Jürgen Bernard, Christian Ritter, David Sessler, Matthias Zeppelzauer, Jörn Kohlhammer, Dieter Fellner


The definition of similarity is a key prerequisite when analyzing complex data types in data mining, information retrieval, or machine learning. However, the meaningful definition is often hampered by the complexity of data objects and particularly by different notions of subjective similarity latent in targeted user groups. Taking the example of soccer players, we present a visual-interactive system that learns users’ mental models of similarity. In a visual-interactive interface, users are able to label pairs of soccer players with respect to their subjective notion of similarity. Our proposed similarity model automatically learns the respective concept of similarity using an active learning strategy. A visual-interactive retrieval technique is provided to validate the model and to execute downstream retrieval tasks for soccer player analysis. The applicability of the approach is demonstrated in different evaluation strategies, including usage scenarions and cross-validation tests.


  1. Abbasnejad, M. E., Ramachandram, D., and Mandava, R. (2012). A survey of the state of the art in learning the kernels. Knowledge and Information Systems, 31(2):193-221.
  2. Behrisch, M., Korkmaz, F., Shao, L., and Schreck, T. (2014). Feedback-driven interactive exploration of large multidimensional data supported by visual classifier. In IEEE Visual Analytics Science and Technology (VAST), pages 43-52.
  3. Bell, S. and Bala, K. (2015). Learning visual similarity for product design with convolutional neural networks. ACM Transactions on Graphics (TOG), 34(4):98.
  4. Bellet, A., Habrard, A., and Sebban, M. (2013). A survey on metric learning for feature vectors and structured data. arXiv preprint arXiv:1306.6709.
  5. Bernard, J., Daberkow, D., Fellner, D. W., Fischer, K., Koepler, O., Kohlhammer, J., Runnwerth, M., Ruppert, T., Schreck, T., and Sens, I. (2015a). Visinfo: A digital library system for time series research data based on exploratory search - a user-centered design approach. Int. Journal on Digital Libraries, 16(1):37-59.
  6. Bernard, J., Sessler, D., Bannach, A., May, T., and Kohlhammer, J. (2015b). A visual active learning system for the assessment of patient well-being in prostate cancer research. In IEEE VIS WS on Visual Analytics in Healthcare (VAHC), pages 1-8. ACM.
  7. Bernard, J., Wilhelm, N., Krüger, B., May, T., Schreck, T., and Kohlhammer, J. (2013). Motionexplorer: Exploratory search in human motion capture data based on hierarchical aggregation. IEEE Transactions on Visualization and Computer Graphics (TVCG), 19(12):2257-2266.
  8. Boriah, S., Chandola, V., and Kumar, V. (2008). Similarity measures for categorical data: A comparative evaluation. International Conference on Data Mining (SIAM), 30(2):3.
  9. Cha, S.-H., Yoon, S., and Tappert, C. C. (2005). Enhancing binary feature vector similarity measures.
  10. Chopra, S., Hadsell, R., and LeCun, Y. (2005). Learning a similarity metric discriminatively, with application to face verification. In Computer Vision and Pattern Recognition (CVPR), pages 539-546. IEEE.
  11. Fogarty, J., Tan, D., Kapoor, A., and Winder, S. (2008). Cueflik: Interactive concept learning in image search. In SIGCHI Conference on Human Factors in Computing Systems (CHI), pages 29-38. ACM.
  12. Frome, A., Singer, Y., Sha, F., and Malik, J. (2007). Learning globally-consistent local distance functions for shape-based image retrieval and classification. In Conference on Computer Vision, pages 1-8. IEEE.
  13. Gönen, M. and Alpaydin, E. (2011). Multiple kernel learning algorithms. Journal of Machine Learning Research, 12(Jul):2211-2268.
  14. Heimerl, F., Koch, S., Bosch, H., and Ertl, T. (2012). Visual classifier training for text document retrieval. IEEE Transactions on Visualization and Computer Graphics (TVCG), 18(12):2839-2848.
  15. Höferlin, B., Netzel, R., Höferlin, M., Weiskopf, D., and Heidemann, G. (2012). Inter-active learning of ad-hoc classifiers for video visual analytics. In IEEE Visual Analytics Sc. and Technology (VAST), pages 23-32.
  16. Janetzko, H., Sacha, D., Stein, M., Schreck, T., Keim, D. A., and Deussen, O. (2014). Feature-driven visual analytics of soccer data. In IEEE Visual Analytics Science and Technology (VAST), pages 13-22.
  17. Jeong, D. H., Ziemkiewicz, C., Fisher, B., Ribarsky, W., and Chang, R. (2009). iPCA: An Interactive System for PCA-based Visual Analytics. In Computer Graphics Forum (CGF), volume 28, pages 767-774. Eurographics.
  18. Kulis, B. (2012). Metric learning: A survey. Foundations and Trends in Machine Learning, 5(4):287-364.
  19. Malisiewicz, T., Gupta, A., and Efros, A. A. (2011). Ensemble of exemplar-svms for object detection and beyond. In Conf. on Computer Vision, pages 89-96. IEEE.
  20. Noh, Y.-K., Zhang, B.-T., and Lee, D. D. (2010). Generative local metric learning for nearest neighbor classification. In Advances in Neural Information Processing Systems, pages 1822-1830.
  21. Norouzi, M., Fleet, D. J., and Salakhutdinov, R. R. (2012). Hamming distance metric learning. In Adv. in neural information processing systems, pages 1061-1069.
  22. Prud'hommeaux, E. and Seaborne, A. (2008). SPARQL Query Language for RDF. W3C Recommendation.
  23. Salakhutdinov, R. and Hinton, G. E. (2007). Learning a nonlinear embedding by preserving class neighbourhood structure. In AISTATS, pages 412-419.
  24. Salton, G. and Buckley, C. (1997). Improving retrieval performance by relevance feedback. Readings in Information Retrieval, 24:5.
  25. Seebacher, D., Stein, M., Janetzko, H., and Keim, D. A. (2016). Patent Retrieval: A Multi-Modal Visual Analytics Approach. In EuroVis Workshop on Visual Analytics (EuroVA), pages 013-017. Eurographics.
  26. Seifert, C. and Granitzer, M. (2010). User-based active learning. In IEEE International Conference on Data Mining Workshops (ICDMW), pages 418-425.
  27. Sessler, D., Bernard, J., Kuijper, A., and Kohlhammer, J. (2014). Adopting mental similarity notions of categorical data objects to algorithmic similarity functions. Vision, Modelling and Visualization (VMV), Poster.
  28. Settles, B. (2009). Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin-Madison.
  29. Shao, L., Sacha, D., Neldner, B., Stein, M., and Schreck, T. (2016). Visual-Interactive Search for Soccer Trajectories to Identify Interesting Game Situations. In IS&T Electronic Imaging Conference on Visualization and Data Analysis (VDA). SPIE.
  30. Sneath, P. H., Sokal, R. R., et al. (1973). Numerical taxonomy. The principles and practice of numerical classification.
  31. Torresani, L. and Lee, K.-c. (2006). Large margin component analysis. In Advances in neural information processing systems, pages 1385-1392.
  32. Tversky, A. (1977). Features of similarity. Psychological review, 84(4):327.
  33. Ware, M., Frank, E., Holmes, G., Hall, M., and Witten, I. H. (2001). Interactive machine learning: letting users build classifiers. Human-Computer Studies, pages 281-292.
  34. Weber, N., Waechter, M., Amend, S. C., Guthe, S., and Goesele, M. (2016). Rapid, Detail-Preserving Image Downscaling. In Proc. of ACM SIGGRAPH Asia.
  35. Weinberger, K. Q. and Saul, L. K. (2009). Distance metric learning for large margin nearest neighbor classification. Machine Learning Research, 10:207-244.
  36. Xing, E. P., Ng, A. Y., Jordan, M. I., and Russell, S. (2003). Distance metric learning with application to clustering with side-information. Advances in neural information processing systems, 15:505-512.
  37. Yang, L., Jin, R., and Sukthankar, R. (2007). Bayesian active distance metric learning. In Conference on Uncertainty in Artificial Intelligence (UAI) .
  38. Yu, J., Amores, J., Sebe, N., Radeva, P., and Tian, Q. (2008). Distance learning for similarity estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 30(3):451-462.
  39. Zagoruyko, S. and Komodakis, N. (2015). Learning to compare image patches via convolutional neural networks. In Comp. Vis. and Pattern Recognition (CVPR). IEEE.

Paper Citation

in Harvard Style

Bernard J., Ritter C., Sessler D., Zeppelzauer M., Kohlhammer J. and Fellner D. (2017). Visual-Interactive Similarity Search for Complex Objects by Example of Soccer Player Analysis . In Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: IVAPP, (VISIGRAPP 2017) ISBN 978-989-758-228-8, pages 75-87. DOI: 10.5220/0006116400750087

in Bibtex Style

author={Jürgen Bernard and Christian Ritter and David Sessler and Matthias Zeppelzauer and Jörn Kohlhammer and Dieter Fellner},
title={Visual-Interactive Similarity Search for Complex Objects by Example of Soccer Player Analysis},
booktitle={Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: IVAPP, (VISIGRAPP 2017)},

in EndNote Style

JO - Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: IVAPP, (VISIGRAPP 2017)
TI - Visual-Interactive Similarity Search for Complex Objects by Example of Soccer Player Analysis
SN - 978-989-758-228-8
AU - Bernard J.
AU - Ritter C.
AU - Sessler D.
AU - Zeppelzauer M.
AU - Kohlhammer J.
AU - Fellner D.
PY - 2017
SP - 75
EP - 87
DO - 10.5220/0006116400750087