Authors:
Jaya Sharm
1
;
Peketi Divya
2
;
C. Vishnu
1
;
C. Reddy
3
;
B. Sekhar
4
and
C. Mohan
1
Affiliations:
1
Department of Computer Science and Engineering, Indian Institute of Technology Hyderabad, Hyderabad, India
;
2
Department of Artificial Intelligence, Indian Institute of Technology Hyderabad, Hyderabad, India
;
3
Department of Information and Communication Technology, UiA, Campus Grimstad, Norway
;
4
Department of Computer Science, Mangalore University, Karnataka, India
Keyword(s):
Deformable Network, Contextual Network, Structural Representative Network, Attention Mechanism, Multi-Level LSTM, Remote Sensing Image Captioning.
Abstract:
Remote sensing image captioning has greater significance in image understanding that generates textual description of aerial images automatically. Majority of the existing architectures work within the framework of encoder-decoder structure. However, it is noted that the existing encoder-decoder based methods for remote sensing image captioning avoid fine-grained structural representation of objects and lack deep encoding representation of an image. In this paper, we propose a novel structural representative network for capturing fine-grained structures of remote sensing imagery to produce fine grained captions. Initially, a deformable network has been incorporated on intermediate layers of convolutional neural network to take out spatially invariant features from an image. Secondly, a contextual network is incorporated in the last layers of the proposed network for producing multi-level contextual features. In order to extract dense contextual features, an attention mechanism is acc
omplished in contextual networks. Thus, the holistic representations of aerial images are obtained through a structural representative network by combining spatial and contextual features. Further, features from the structural representative network are provided to multi-level decoders for generating spatially semantic meaningful captions. The textual descriptions obtained due to our proposed approach is demonstrated on two standard datasets, namely, the Sydney-Captions dataset and the UCM-Captions dataset. The comparative analysis is made with recently proposed approaches to exhibit the performance of the proposed approach and hence argue that the proposed approach is more suitable for remote sensing image captioning tasks.
(More)