Second-Order Learning with Grounding Alignment: A Multimodal Reasoning Approach to Handle Unlabelled Data

Arnab Barua, Mobyen Uddin Ahmed, Shaibal Barua, Shahina Begum, Andrea Giorgi

2024

Abstract

Multimodal machine learning is a critical aspect in the development and advancement of AI systems. However, it encounters significant challenges while working with multimodal data, where one of the major issues is dealing with unlabelled multimodal data, which can hinder effective analysis. To address the challenge, this paper proposes a multimodal reasoning approach adopting second-order learning, incorporating grounding alignment and semi-supervised learning methods. The proposed approach illustrates using unlabelled vehicular telemetry data. During the process, features were extracted from unlabelled telemetry data using an autoencoder and then clustered and aligned with true labels of neurophysiological data to create labelled and unlabelled datasets. In the semi-supervised approach, the Random Forest (RF) and eXtreme Gradient Boosting (XGBoost) algorithms are applied to the labelled dataset, achieving a test accuracy of over 97%. These algorithms are then used to predict labels for the unlabelled dataset, which is later added to the labelled dataset to retrain the model. With the additional prior labelled data, both algorithms achieved a 99% test accuracy. Confidence in predictions for unlabelled data was validated using counting samples based on the prediction score and Bayesian probability. RF and XGBoost scored 91.26% and 97.87% in counting samples and 98.67% and 99.77% in Bayesian probability, respectively.

Paper Citation

in Harvard Style

Barua A., Uddin Ahmed M., Barua S., Begum S. and Giorgi A. (2024). Second-Order Learning with Grounding Alignment: A Multimodal Reasoning Approach to Handle Unlabelled Data. In Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART; ISBN 978-989-758-680-4, SciTePress, pages 561-572. DOI: 10.5220/0012466500003636

in Bibtex Style

@conference{icaart24,
author={Arnab Barua and Mobyen Uddin Ahmed and Shaibal Barua and Shahina Begum and Andrea Giorgi},
title={Second-Order Learning with Grounding Alignment: A Multimodal Reasoning Approach to Handle Unlabelled Data},
booktitle={Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART},
year={2024},
pages={561-572},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012466500003636},
isbn={978-989-758-680-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART
TI - Second-Order Learning with Grounding Alignment: A Multimodal Reasoning Approach to Handle Unlabelled Data
SN - 978-989-758-680-4
AU - Barua A.
AU - Uddin Ahmed M.
AU - Barua S.
AU - Begum S.
AU - Giorgi A.
PY - 2024
SP - 561
EP - 572
DO - 10.5220/0012466500003636
PB - SciTePress