Distance-aware multilayer aggregation transformer for image captioning

Honglong Xu; Lixia Xue; Ronggui Wang; Juan Yang

doi:10.1117/12.3035618

16 August 2024 Distance-aware multilayer aggregation transformer for image captioning

Honglong Xu, Lixia Xue, Ronggui Wang, Juan Yang

Proceedings Volume 13230, Third International Conference on Machine Vision, Automatic Identification, and Detection (MVAID 2024); 132302G (2024) https://doi.org/10.1117/12.3035618
Event: Third International Conference on Machine Vision, Automatic Identification and Detection, 2024, Kunming, China

Abstract

Recently, the Transformer based on grid features has achieved great success in image captioning, but it still has some problems: the flattening operation of Transformer will destroy the positional information among visual objects, and only the output of the last layer encoder is sent to the decoder, which will lose low-level semantic information. To solve the above problems, we first introduce Distance-aware self-attention (DA), which considers the original geometric distance between visual objects in a two-dimensional image during the self-attention modeling process, and integrates distance information into attention calculation through a mapping function, better capturing the relational information among visual objects. Second, we propose the Multilayer Aggregation (MA) module, which aggregates the output of the encoder and establishes a weighted residual connection as the final output, sent to the decoder separately. It aggregates information from different encoder layers to achieve cross-layer semantic complementarity, features with rich semantics can be explored simultaneously from both low-level and high-level coding layers. To verify the validity of our proposed two designs, we applied them to a standard Transformer and conducted extensive experiments on MS-COCO, a benchmark dataset for image captioning. The experimental results demonstrate the effectiveness of our proposed Distance-aware Multilayer Aggregation Transformer (DMAT) model.

(2024) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Honglong Xu, Lixia Xue, Ronggui Wang, and Juan Yang "Distance-aware multilayer aggregation transformer for image captioning", Proc. SPIE 13230, Third International Conference on Machine Vision, Automatic Identification, and Detection (MVAID 2024), 132302G (16 August 2024); https://doi.org/10.1117/12.3035618

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
9 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Transformers

Visualization

Semantics

Performance modeling

Matrices

Information visualization

Education and training

Show All Keywords

Keywords/Phrases

Search In:

Publication Years