Authors:
Obeida ElJundi
1
;
Mohamad Dhaybi
1
;
Kotaiba Mokadam
2
;
Hazem Hajj
1
and
Daniel Asmar
3
Affiliations:
1
American University of Beirut, Electrical and Computer Engineering Department, Lebanon
;
2
American University of Beirut, Civil and Environmental Engineering Department, Lebanon
;
3
American University of Beirut, Mechanical Engineering Department, Lebanon
Keyword(s):
Deep Learning, Computer Vision, Natural Language Processing, Image Captioning, Arabic.
Abstract:
Image Captioning (IC) is the process of automatically augmenting an image with semantically-laden descriptive text. While English IC has made remarkable strides forward in the past decade, very little work exists on IC for other languages. One possible solution to this problem is to boostrap off of existing English IC systems for image understanding, and then translate the outcome to the required language. Unfortunately, as this paper will show, translated IC is lacking due to the error accumulation of the two tasks; IC and translation. In this paper, we address the problem of image captioning in Arabic. We propose an end-to-end model that directly transcribes images into Arabic text. Due to the lack of Arabic resources, we develop an annotated dataset for Arabic image captioning (AIC). We also develop a base model for AIC that relies on text translation from English image captions. The two models are evaluated with the new dataset, and the results show the superiority of our end-to-
end model.
(More)