Exploratory Multimodal Data Analysis with Standard Multimedia Player - Multimedia Containers: A Feasible Solution to Make Multimodal Research Data Accessible to the Broad Audience

Julius Schöning, Anna L. Gert, Alper Açık, Tim C. Kietzmann, Gunther Heidemann, Peter König

Abstract

The analysis of multimodal data comprised of images, videos and additional recordings, such as gaze trajectories, EEG, emotional states, and heart rate is presently only feasible with custom applications. Even exploring such data requires compilation of specific applications that suit a specific dataset only. This need for specific applications arises since all corresponding data are stored in separate files in custom-made distinct data formats. Thus accessing such datasets is cumbersome and time-consuming for experts and virtually impossible for non-experts. To make multimodal research data easily shareable and accessible to a broad audience, like researchers from diverse disciplines and all other interested people, we show how multimedia containers can support the visualization and sonification of scientific data. The use of a container format allows explorative multimodal data analyses with any multimedia player as well as streaming the data via the Internet. We prototyped this approach on two datasets, both with visualization of gaze data and one with additional sonification of EEG data. In a user study, we asked expert and non-expert users about their experience during an explorative investigation of the data. Based on their statements, our prototype implementation, and the datasets, we discuss the benefit of storing multimodal data, including the corresponding videos or images, in a single multimedia container. In conclusion, we summarize what is necessary for having multimedia containers as a standard for storing multimodal data and give an outlook on how artificial networks can be trained on such standardized containers.

References

  1. Ac¸ik, A., Bartel, A., and K önig, P. (2014). Real and implied motion at the center of gaze. Journal of Vision, 14(2):1- 19, doi:10.1167/14.1.2.
  2. Bertellini, G. and Reich, J. (2010). DVD supplements: A commentary on commentaries. Cinema Journal, 49(3):103-105, doi:10.1353/cj.0.0215.
  3. Bresenham, J. E. (1965). Algorithm for computer control of a digital plotter. IBM Systems Journal, 4(1):25-30, doi:10.1147/sj.41.0025.
  4. Cohen, M. X. (2014). Analyzing Neural Time Series Data: theory and practice. MIT University Press Group Ltd.
  5. Delorme, A. and Makeig, S. (2004). EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. Journal of Neuroscience Methods, 134(1):9-21, doi:10.1016/j.jneumeth.2003.10.009.
  6. Einhäuser, W. and K önig, P. (2010). Getting real-sensory processing of natural stimuli. Current Opinion in Neurobiology, 20(3):389-395, doi:10.1016/j.conb.2010.03.010.
  7. Einhäuser, W., Schumann, F., Bardins, S., Bartl, K., Böning, G., Schneider, E., and König, P. (2007). Human eye-head co-ordination in natural exploration. Network: Computation in Neural Systems, 18(3):267-297, doi:10.1080/09548980701671094.
  8. GNU (2016). Basic Tar Format Version: http://www.gnu.org/software/tar/manual/html node/ %53tandard.html.
  9. ISO/IEC (1993). Information technology-coding of moving pictures and associated audio for digital storage media at up to about 1,5 Mbit/s-Part 1: Systems (ISO/IEC 11172-1:1993).
  10. ISO/IEC (2001). Information technology-multimedia content description interface-Part 3: Visual (ISO/IEC 15938-3:2001).
  11. ISO/IEC (2003). Information technology-coding of audio-visual objects-Part 14: MP4 file format (ISO/IEC 14496-14:2003).
  12. ISO/IEC (2015). Information technology-generic coding of moving pictures and associated audio informationPart 1: Systems (ISO/IEC 13818-1:2015).
  13. Martin, M., Charlton, J., and Connor, A. M. (2015). Mainstreaming video annotation software for critical video analysis. Journal of Technologies and Human Usability, 11(3):1-13.
  14. Matroska (2016). Matroska Media Container https:// www. matroska.org/ .
  15. MKVToolNix (2016). mkvmerge https://mkvtoolnix. download/doc/mkvmerge.html.
  16. Paris, C., Vialle, L., and Hammer, U. (2016). TitleVision - USF specs http://register.titlevision.dk/files/ usf-specs-html.zip.
  17. Petrovic, N., Jojic, N., and Huang, T. S. (2005). Adaptive video fast forward. Multimedia Tools Appl., 26(3):327- 344, doi:10.1007/s11042-005-0895-9.
  18. PKWARE Inc. (2016). ZIP File Format Specification Version: 6.3.4 https://pkware.cachefly.net/webdocs/ appnote/appnote-5.2.0.txt.
  19. Rackaway, C. (2010). Video killed the textbook star? use of multimedia supplements to enhance student learning. Journal of Political Science Education, 8(2):189-200, doi:10.2139/ssrn.1547142.
  20. Schöning, J., Faion, P., and Heidemann, G. (2016a). Interactive feature growing for accurate object detection in megapixel images. Computer Vision ECCV 2016 Workshops, 9913:546556, doi:10.1007/978-3-319- 46604-0 39.
  21. Schöning, J., Faion, P., Heidemann, G., and Krumnack, U. (2016b). Eye tracking data in multimedia containers for instantaneous visualizations. In IEEE VIS Workshop on Eye Tracking and Visualization (ETVIS). IEEE [InPress].
  22. Schöning, J., Faion, P., Heidemann, G., and Krumnack, U. (2017). Providing video annotations in multimedia containers for visualization and research. In IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE [InPress].
  23. SSA v4.00+ (2016). Sub Station Alpha v4.00+ Script Format http:// moodub.free.fr/ video/ ass-specs.doc.
  24. Telepool Media GmbH (2014). Colourful planet collection courtesy of www.mdr.de.
  25. The MathWorks Inc. (2014). Matlab 2014b.
  26. Vernier, M., Farinosi, M., and Foresti, G. L. (2016). A smart visual information tool for situational awareness. In Proceedings of the International Conference on Computer Vision Theory and Applications (VISAPP 2016), volume 3, pages 238-247. SCITEPRESS.
  27. WebM (2016). WebM Container Guidelines https://www. webmproject.org/docs/container/.
  28. Wylie, C., Romney, G., Evans, D., and Erdahl, A. (1967). Half-tone perspective drawings by computer. In Proceedings of the Fall Joint Computer Conference, AFIPS 7867 (Fall), pages 49-58, New York, NY, USA. ACM.
  29. Xiph.org (2016). Ogg https:// xiph.org/ ogg/ .
Download


Paper Citation


in Harvard Style

Schöning J., Gert A., Açık A., Kietzmann T., Heidemann G. and König P. (2017). Exploratory Multimodal Data Analysis with Standard Multimedia Player - Multimedia Containers: A Feasible Solution to Make Multimodal Research Data Accessible to the Broad Audience . In Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP, (VISIGRAPP 2017) ISBN 978-989-758-225-7, pages 272-279. DOI: 10.5220/0006260202720279


in Bibtex Style

@conference{visapp17,
author={Julius Schöning and Anna L. Gert and Alper Açık and Tim C. Kietzmann and Gunther Heidemann and Peter König},
title={Exploratory Multimodal Data Analysis with Standard Multimedia Player - Multimedia Containers: A Feasible Solution to Make Multimodal Research Data Accessible to the Broad Audience},
booktitle={Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP, (VISIGRAPP 2017)},
year={2017},
pages={272-279},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006260202720279},
isbn={978-989-758-225-7},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP, (VISIGRAPP 2017)
TI - Exploratory Multimodal Data Analysis with Standard Multimedia Player - Multimedia Containers: A Feasible Solution to Make Multimodal Research Data Accessible to the Broad Audience
SN - 978-989-758-225-7
AU - Schöning J.
AU - Gert A.
AU - Açık A.
AU - Kietzmann T.
AU - Heidemann G.
AU - König P.
PY - 2017
SP - 272
EP - 279
DO - 10.5220/0006260202720279