MCADFusion: a novel multi-scale convolutional attention decomposition method for enhanced infrared and visible light image fusion

1.   Introduction

Image fusion is a technique used to combine information from multiple images of the same scene, captured by different sensors, from various locations, or at different times ^[1]. Combining multiple imaging modalities can yield more comprehensive information about the observed world than using a single modality ^[2]. Fusion targets include digital images ^[3,4], multimodal images ^[5,6,7], and remote sensing images ^[8,9,10]. This technique can be widely used in several fields such as saliency detection ^[11,12], target detection ^[13,14], semantic segmentation ^[15,16], object tracking^[17], autonomous driving, and video surveillance by providing a clearer presentation of the target and scene ^[18].

Image fusion mainly consists of three key elements: feature extraction, fusion strategy, and image reconstruction. Existing research has focused on one or more of these processes to improve the fusion performance ^[19]. In multi-sensor image processing, infrared and visible image fusion has received widespread attention for combining the advantages of both. While infrared images utilize the difference in thermal radiation of the target to discriminate the target, visible light images have advantages in texture details ^[2]. Infrared and visible image fusion technology integrates the thermal radiation information from infrared images with the texture details of visible images, thereby significantly enhancing the clarity and interpretability of the resulting images. This technology holds significant application value in military, security monitoring, and medical fields. In recent years, numerous methods have been explored to address the challenges of fusing infrared and visible light images ^[20,21,22]. Among these, infrared and visible light image fusion methods are primarily categorized into traditional fusion methods ^[23,24] and deep learning-based fusion methods ^[25,26]. In traditional methods, multi-scale transform techniques ^[27] are commonly used to extract multi-scale features from different source images. These features are subsequently combined using an appropriate fusion strategy and reconstructed with multi-scale transform techniques to generate the final fused image. Another common approach is pixel-level image fusion ^[28], which enables rapid merging of different source images by directly applying simple fusion rules, such as averaging, maximizing, or weighted averaging, at the pixel level. There are some shortcomings in the traditional fusion methods: 1) Traditional multi-scale transform methods may fail to effectively retain all critical features from different source images, leading to incomplete information in the fused images. 2) Simple pixel-level fusion rules struggle to manage complex image information, leading to lower quality fused images. 3) Fusion rules must be meticulously crafted to suit specific application scenarios, thereby escalating design complexity and diminishing adaptability across different scenarios. Fusion method based on deep learning: In 2017, Liu et al. ^[29] introduced a convolutional neural network-based fusion method designed specifically for integrating multi-focus images. Li et al. ^[30] proposed a fusion approach based on dense blocks and auto-encoding architecture, specifically designed for integrating images from various sources. There is also a generative adversarial network (GAN) based fusion method for infrared and visible images ^[31]. An adversarial model is trained to generate a fused image containing both the thermal radiation information from the infrared image and the texture details from the visible image. Although the application of deep learning techniques to the field of image fusion has greatly improved the fusion performance. There are still the following shortcomings: 1) The same network is used for both infrared and visible light images, which cannot distinguish the unique features of different source images well, and the fusion strategy uses a manually designed scheme. The features of infrared and visible light images are not well preserved. at the same time, and the quality of the fused image is not high. 2) When using complex architectures such as generative adversarial networks (GAN), the training process may become unstable and prone to problems such as pattern collapse or unstable quality of generated images. 3) Some networks lack downsampling operators for extracting multi-scale features, resulting in underutilization of deep features ^[32]. Moreover, to fully leverage multi-scale features and integrate deep features effectively, network topology improvements and meticulous fusion strategy design are essential.

In this paper, we propose a new end-to-end fusion method for multi-scale convolutional attention decomposition to solve the above problems. First, our goal is to separately extract global and local information from different source images, while incorporating similarity constraints to their shared and unique features, so as to improve the controllability and interpretability of feature decomposition, in order to obtain the global and local feature information of different source images. Taking infrared images as an example, we initially employ a multi-scale feature extraction module to extract multi-scale features from infrared images. Subsequently, the restormer module enriches feature maps with global and local information using its multi-attention and convolutional properties. Channel attention and spatial attention modules are then used to extract global and local features from the feature maps. The training process is divided into two stages. In the first stage, we independently fuse the global and local information of infrared and visible images using a fusion strategy. In the second stage, we concatenate the global information of the infrared image with that of the visible image, and similarly concatenate the local information. Finally, we fuse the concatenated global and local information. The final fused image is reconstructed using the multi-attention and convolutional properties of the restormer module. The MCADFusion method shows high complexity in terms of technical design, implementation difficulty, computational resource requirements, and resource optimization, but through fine design and optimization, it successfully achieves efficient and accurate fusion of infrared images and visible light images. Experimental results demonstrate that the MCADFusion method surpasses existing fusion methods in both subjective visual and objective evaluations on public TNO and MSRS datasets. The contributions of this paper can be summarized in four main areas:

● Multi-scale feature extraction combined with attention mechanism: We propose combining a multi-scale feature extraction module with the restormer module to acquire and fuse global and local features from different source images using multi-head attention and convolutional properties. Specifically, the channel attention module extracts global features, while the spatial attention module processes local features, effectively preserving and fusing features from infrared and visible light images.

● Two stage training strategy: To enhance the quality of fused images, we adopt a two-stage training strategy. In the first stage, global and local information from infrared and visible images are fused separately. In the second stage, global and local information from both image types are concatenated and then fused. This strategy better preserves the unique and shared features of different source images, improving the quality and consistency of the fused images.

● Similarity constraints and controllability of feature decomposition: We introduce similarity constraints in the feature extraction and fusion process to enhance the controllability and interpretability of feature decomposition. This approach better distinguishes and preserves unique features of different source images while effectively utilizing shared features in the fusion process to generate high-quality fused images.

● End-to-end training design: The MCADFusion method employs an end-to-end training approach that minimizes human intervention, streamlines the fusion process, and significantly enhances adaptability across various scenes and lighting conditions. Using publicly accessible TNO and MSRS datasets, the approach was rigorously compared to state-of-the-art methods both qualitatively and quantitatively, consistently outperforming them in terms of visual and objective metrics. This technique reliably delivers superior performance in both visual and objective evaluations.

2.   Related works

In recent years, a variety of techniques have been developed for image fusion. These include traditional methods that fuse infrared and visible images, deep learning-based fusion methods, and feature-level fusion methods. This section will provide a concise overview of these approaches, alongside discussions on the multi-scale feature extraction module, the transformer module, and the channel and spatial attention modules.

2.1.   From traditional methods to deep learning and feature-level fusion

Image fusion techniques have witnessed significant development over the past decades, evolving from initial traditional methods to advanced deep learning techniques and feature-level fusion strategies. Traditional methods for infrared and visible image fusion primarily rely on pixel-level operations, such as averaging, maximum value selection, or multi-scale transformation techniques ^[30,31]. While these methods are simple and intuitive, they often lack adaptability to complex scenes. With advancements in computational power and data volume, deep learning-based fusion methods ^[32,33,34] have emerged, utilizing sophisticated models like convolutional neural networks (CNNs), generative adversarial networks (GANs), and others. These models effectively learn abstract features from data, enhancing the quality of fused images and expanding application versatility. For example, in ^[25], Li et al. developed an end-to-end framework for infrared and visible image fusion using an innovative deep learning architecture with CNNs and dense blocks. The framework features a dense encoder network with larger kernel convolutions to efficiently capture features over a wide receptive field. Texture details and feature contrast are then improved using a texture contrast compensation module based on gradient residuals and attention mechanisms. Finally, four convolutional layers reconstruct the fused image.

In addition, fusion methods based on feature level ^[35] further enhance the processing fineness, by directly manipulating and synthesizing the deep features of an image, these methods are able to capture and fuse the key information in the image more efficiently and achieve more advanced fusion effects. This evolution not only pushes the technological boundaries of image fusion techniques, but also expands the prospects for their application in medical imaging, military reconnaissance, autonomous driving, etc.

2.2.   Multi-scale feature extraction module

Inception blocks were first introduced by Szegedy et al. in ^[36]. The multi-scale feature extraction module primarily utilizes and modifies these inception blocks, which were initially designed to extract multi-scale features within the framework of that thesis. In this module, multi-scale feature information is extracted from different source images using 1 × 1, 3 × 3, and 5 × 5 convolutional filters, as well as max pooling methods. Batch normalization (BN) layers are added after each convolutional layer to speed up the training process and enhance the generalization ability of the model. As shown in Figure 1, the multi-scale feature extraction module is divided into four parallel paths, each extracting features at different scales. The outputs of these four paths are concatenated along the channel dimension to form a feature map containing multi-scale features. This module structure effectively combines the advantages of convolution kernels of different sizes to capture multi-scale information from various source images. Taking an infrared image and a visible image as examples, the inputs are both single-channel. After extraction through the four parallel paths, the outputs are concatenated along the channel dimension. A 1 × 1 convolution kernel is then used to adjust the number of channels, resulting in a 64-channel feature map. The results of the infrared and visible images processed by this module are shown in Figure 1.

2.3.   Transformer module in restormer

Transformer was originally proposed by Vaswani et al. ^[37] for the field of natural language processing, and subsequently, ViT ^[38] applied it to computer vision. Given the high computational cost of spatial self-attention mechanisms, Wu et al. ^[39] developed a lightweight LT architecture for mobile NLP tasks. This architecture maintains the performance while reducing the number of model parameters by implementing a long and short ranges attention mechanism and a flattened feedforward network. In addition, restormer ^[40] improved the transformer module by incorporating a gated deconvolution network and a multi-deconvolution head attention transposition module to support multi-scale local-to-global representation learning for high-resolution images. In the MCADFusion framework, the transformer module is utilized in both the encoder and decoder, with the transformer module depicted in (c) in Figure 2 below. Within the transformer module, input feature maps undergo normalization with LayerNorm (LN) layers to ensure that the inputs of each layer are standardized and not affected by input distribution. In Figure 2(c), Q (Query), K (Key), and V (Value) are employed to compute attentional weights, obtained through matrix multiplication with the input sequence. Subsequently, the dot product operation compares the Q and K vectors, with a softmax activation function applied to provide them with a nonlinear representation. These weighted vectors are then used to compute the final attention representation with the Value vector. Following this, a 1 × 1 convolution modifies the number of channels. The red connecting arrows denote shortcut connections, which add the input of a layer to its output to facilitate gradient flow. This is followed by normalization via the LN layer, and subsequently, another 1 × 1 convolution adjusts the number of channels. The tensor segmentation operation evenly splits the channels, resulting in two feature maps. One of these feature maps undergoes processing with the ReLU activation function, and then the feature maps, after activation, undergo position-wise multiplication to obtain a feature map containing global and local features. Finally, another 1 × 1 convolution is employed to alter the number of channels and yield the final resultant feature map. The transformer based on primitive blocks cannot guarantee information communication between blocks, which will hinder the extraction of attentional information from the global view ^[41]. In contrast, our transformer module effectively extracts a feature map containing both global and local features after a multi-scale feature extraction module. Global features improve the efficiency and accuracy of the model by capturing large-scale information, while local features focus on details and filter out irrelevant and redundant information. The notation $8*(\hat{H}*\hat{W})*\hat{C}$ and $8*\hat{C}*(\hat{H}*\hat{W})$ in (c) of Figure 2 represents the shape description of the four-dimensional tensor, with the reshaping primarily for realizing the matrix multiplication operation. The module combines an efficient self-attention mechanism with convolutional operations, which can effectively capture both global and long-range dependencies of the image and accurately extract local features, thus significantly enhancing the image restoration performance; at the same time, its modularized design allows the module to be easily integrated with other models, further extending its performance and application potential. This module is mainly used in the encoding and decoding stages of our method for feature extraction and image reconstruction.

2.4.   Channel and spatial attention module

In ^[33], Li et al. proposed a network based on nested connections and a spatial/channel attention model. Inspired by this, in the encoder stage of MCAD, we use the spatial attention module (SAM) and the channel attention module (CAM) to extract local and global information of different source images, respectively. Specifically, the channel attention mechanism is mainly used to further extract global features of infrared or visible images, called basic features; the spatial attention mechanism is mainly used to further extract local features of infrared or visible images, called detailed features. Among the CAMs is the shared MLP (a Multilayer Perceptron), which consists of multiple fully connected (dense) layers, each applying a linear transformation followed by a nonlinear activation function. In this process, different regions of the feature map share the same set of weights (parameters), ensuring parameter efficiency and consistency. This sharing mechanism facilitates the learning of uniform transformations that can be applied to different parts of the input data. The specific implementations of the channel attention module (CAM) and spatial attention module (SAM) are shown in Figures 3 and 4, respectively.

2.5.   Comparison with existing methods

The MCADFusion model demonstrates significant advantages over existing image fusion techniques. Compared with traditional infrared and visible image fusion methods, MCADFusion not only extracts and fuses local and long-range features from different source images more efficiently, but its structural design also makes the model intuitive and easy to understand. Additionally, the multi-scale convolutional attention fusion strategy adopted by MCADFusion, combined with the channel attention (CAM) and spatial attention (SAM) modules, enables it to excel in preserving key global information and enhancing local texture details when handling high-complexity image fusion tasks. Compared with conventional convolutional neural network (CNN)-based fusion methods, our model not only offers more fine-grained feature processing capabilities but also improves the quality of feature extraction by using an innovative loss function that effectively suppresses redundant information.

3.   Methodology

In this section, we first briefly describe the workflow of MCADFusion and its sub-modules. Then, we provide a detailed description of the implementation process for the encoder, fusion layer, and decoder. For ease of discussion, we refer to the low-frequency global features as base features and the high-frequency local features as detailed features.

3.1.   General

The MCADFusion method begins by decomposing the base and detail features of different source images using a decoder. These features are then fused through a fusion layer, and finally, the image is reconstructed using a decoder. In the first stage, the input consists of separate channels for infrared and visible light images These images pass through a multi-scale feature extraction module and the restormer module to produce shallow feature maps. These maps are then processed through the channel attention mechanism and spatial attention mechanism modules to extract basic and detailed features for both infrared and visible light images. For example, the base and detail features of the infrared image are input into the restormer module after channel merging, resulting in the recovered infrared image. In the second stage, the same inputs are used as in the first stage. To better integrate the base and detail features from both the infrared and visible light images, the training parameters from the first stage are utilized. The shallow base or detail features extracted from the infrared image via the multi-scale feature extraction and restormer modules, along with the corresponding features from the visible light image, are channel-merged. They are then passed through the channel attention module and apatial attention module, respectively, to obtain the base and detail features for the infrared or visible light images. These features are subsequently input into the restormer module after channel merging, ultimately producing the fused image of the infrared and visible light images. In both the first and second stages, the feature extraction module is referred to as the encoder stage, while the stage where the images are reconstructed after channels merging is called the decoder stage. The detailed workflow is illustrated in Figure 5.

3.2.   Encoders

The encoder stage consists of four main modules: the shared multi-scale feature extraction (MSF) module, the restormer sharing (SFE) module, the CAM module, and the SAM module. The MSF module extracts multi-scale features from different source images. Following this, the SFE module utilizes a transformer module to extract feature information that includes both global and local features. The two-branch structure employs the CAM and SAM modules to simultaneously extract base and detailed feature information from different source images. These CAM and SAM modules ensure synchronized extraction of these features.

For clear understanding, we denote the input IR and visible images as $I \in R^{H*W*1}$ and $V \in R^{H*W*3}$. where the four modules MSF, SFE, CAM, and SAM in the encoder stage are represented by M(.), SF(.), CC(.), and SA(.) representations. The purpose of shared feature encoder based on multi-scale feature extraction (MSF) is to extract multi-scale features $\{ \Phi_{I}^{M}, \Phi_{V}^{M} \}$ from different source images $\{ I, V \}$ of the input, i.e.,

The multi-scale feature extraction module is used in MSF to improve the robustness of the model, to enhance the model's adaptability to scale changes, and to help the model to distinguish between foreground and background. The purpose of SFE is to extract features containing both base and detailed features $\{ \Phi_{I}^{SF}, \Phi_{V}^{SF} \}$ from the extracted multi-scale features $\{ \Phi_{I}^{M}, \Phi_{V}^{M} \}$, i.e.,

The restormer block is chosen in SFE because it can be used to extract the global features of the source image from the high-resolution input feature maps by applying its multi-head attention across feature dimensions. Therefore, the use of restormer block can be realized to extract the cross-modal shallow feature maps. The purpose of CAM is to extract the base features from containing both base and detailed features $\{ \Phi_{I}^{SF}, \Phi_{V}^{SF} \}$, i.e.,

where $\{ \Phi_{I}^{C}, \Phi_{V}^{C} \}$ denote the base features of infrared and visible image respectively. The use of channel attention mechanism (CAM) mainly improves the performance, efficiency, flexibility, adaptability to new data, and robustness of deep learning models by emphasizing important features and suppressing minor features. Spatial attention detail feature extraction (SAM) is just the opposite of CAM, which mainly extracts the detail features from containing both base and detailed features $\{ \Phi_{I}^{SF}, \Phi_{V}^{SF} \}$, i.e.,

where $\{ \Phi_{I}^{SF}, \Phi_{V}^{SF} \}$ denotes the detail features of infrared and visible images, respectively. The use of the spatial attention mechanism (SAM) effectively enhances the ability of the deep learning model in processing image details, adapting to new environments, improving computational efficiency, and resisting interference by focusing on the key parts of the image, thus achieving better performance in the task of extracting detailed features.

3.3.   Fusion layer

The function of the base/detail fusion layer is mainly to fuse the base/detail feature information from different source images separately. Since the generalization bias in base/detail fusion is similar to the base/detail feature extraction obtained through encoder decomposition, we adopt CAM and SAM blocks as the base/detail fusion layer. The base/detail features $\{ \Phi_{I}^{C}, \Phi_{V}^{C} \}$, $\{ \Phi_{I}^{SA}, \Phi_{V}^{SA} \}$ are extracted from the encoder for infrared and visible light, where:

$\{ F_{C}, F_{SA} \}$ is base/detail fusion layer respectively.

3.4.   Decoder

In the decoder D(.), in which the features obtained from decomposition are concatenated in the channel dimension as input to the decoder. In training phase Ⅰ, the decoder outputs the original image; while in training phase Ⅱ, the output is the fused image. It is formulated as:

In the decoder stage, since the inputs for the second stage of training are base and detailed features from different modalities, in order to maintain design consistency with the SFE (feature extractor), the decoder employs a restorer block as its base building block.

3.5.   Two-stage training

The lack of field validation due to the need to extract base and detail features separately from different source images makes state-of-the-art semi-supervised and, fully supervised learning methods inapplicable. Therefore, we use a two-stage learning scheme for end-to-end training of these images, for which the MCADFusion method is proposed, and the two-stage framework is shown in Figure 5.

In the training stage Ⅰ, the infrared and visible images $\{ I \in R^{H*W*1}$, $V \in R^{H*W*3} \}$ are used as the inputs of the encoder, the multi-scale features $\{ \Phi_{I}^{M}, \Phi_{V}^{M} \}$ of the infrared and visible images are obtained by the shared feature encoder based on multi-scale feature extraction (MSF), $\{ \Phi_{I}^{M}, \Phi_{V}^{M} \}$ are used as the inputs of the encoder, and then the shallow features $\{ \Phi_{I}^{SF}, \Phi_{V}^{SF} \}$ are obtained by the shared feature encoder based on the restorer [SFE] block. The shared feature encoder based on the restorer [SFE] block obtains the shallow features $\{ \Phi_{I}^{SF}, \Phi_{V}^{SF} \}$, the shallow features are used as the inputs of the dual channel (based on the channel-attentive base feature extraction (CAM) encoder and spatial-attentive detailed feature extraction (SAM) encoder), to obtain the base and detailed features of the infrared (or visible light) image $\{ \Phi_{I}^{C}, \Phi_{I}^{SA} \}$ (or $\{ \Phi_{V}^{C}, \Phi_{V}^{SA} \}$), the base and detail features are concatenated as the input of the decoder, and finally the IR (or visible light) original image $\{ \hat{I}, \hat{V} \}$ is obtained by the decoder.

In the training phase Ⅱ, infrared and visible images ($\{ I \in R^{H*W*1}$, $V \in R^{H*W*3} \}$) are fed into the encoder pre-trained in training phase Ⅰ. We extract the image on the basis of characteristics of $\{ \Phi_{I}^{C}, \Phi_{V}^{C} \}$ and the detail characteristics of $\{ \Phi_{I}^{SA}, \Phi_{V}^{SA} \}$. These features are then fed into the corresponding fusion module $F_{C}$ or $F_{SA}$. Finally, the fused features $\{ \Phi^{C}, \Phi^{SA} \}$ are fed into the decoder to generate the fused image F.

The total loss function used in the training phase Ⅰ is:

where $L_{ir}$ and $L_{vis}$ are the similarity and error between the IR and visible images and the reconstructed image, respectively, and both $L_{ir}$ and $L_{vis}$ are known as reconstruction losses. $L_{decomp}$ is the similarity between the IR and visible images at the base and detail level for finer and more balanced feature fusion called base and detail decomposition loss, and $L_{grad}$ is the gradient of the fused image between the IR and visible images and the fused image loss. In Eq (3.7), the values of each hyperparameter are $\alpha_{1} = 1$, $\alpha_{2} = 1$, $\alpha_{3} = 2.5$, and $\alpha_{4} = 6$. These values have been experimentally verified to ensure that the model can balance the infrared image loss and visible light image loss to achieve optimal image restoration. The weights of the infrared and visible light image losses are both 1, indicating their equal importance. The weight of the decomposition loss is 2.5, emphasizing the model's image decomposition ability, while the weight of the gradient loss is 6, focusing on edge information and detail preservation. Adjusting these hyperparameters will degrade the model's performance in certain aspects, affecting image quality. By maintaining these optimized values, the model can achieve the best balance between the losses and ensure high-quality image restoration. The reconstruction loss, i.e.,

In Eq (3.8), $L_{ssim}({I, \hat{I}}) = 1-SSIM({I, \hat{I}})$ is the structural similarity index, which is used to measure the similarity of the visual effect of the two images, and $L_{ssim}$ is mainly used to evaluate the quality loss between the recovered image and the original image. $L_{MSE} = ||I-\hat{I}||_{2}^{2}$ is mainly used to calculate the average of the sum of squares of the pixel-level differences between the original image and the recovered image, which is used to accurately control the consistency of each pixel of the reconstructed image with the original image, and to improve the pixel-level accuracy. $\beta_{1}$ is set to 5 for regulating the effect of the $L_{ssim}$ loss in the total loss function. The weighted combination of loss functions using $\beta_{1}$ and $\beta_{2}$, a combination that primarily utilizes the perceived quality benefits of SSIM losses while still retaining the numerical accuracy of MSE losses. $L_{vis}$ is similar to $L_{ir}$, which is used for assessing the the reconstruction loss of the visible image with the reconstructed image after reconstruction, i.e.,

In Eq (3.9), we introduce the correlation CC(.), which is used to calculate the degree of similarity between the base or detail feature vectors $\{ \Phi_{I}^{SA}, \Phi_{V}^{SA} \}$ or $\{ \Phi_{I}^{C}, \Phi_{V}^{C} \}$ of the infrared and visible images, and is mainly used as a measure of the loss of correlation between different features. In order to make it combinable, we also introduce $\epsilon$ = 1.01, mainly to prevent the denominator from being 0 or close to 0, which leads to divergence or non-divergence of the calculation results, and to enhance the robustness and stability of the formula. The base and detail total decomposition loss equations, i.e.,

In addition, the gradient loss in Eq (3.7) is calculated as shown in Eq (3.11) below, i.e.,

Subject to ^[42,43], during training phase Ⅱ, the total loss function is:

4.   Experimental validation

In this section, we first describe the experimental setup, detailing the datasets, evaluation metrics, and training process. We then conduct quantitative and qualitative analyses on two publicly available datasets, comparing their performance with seven state-of-the-art models in both subjective and objective aspects. Finally, we validate the impact of different model components through ablation experiments.

4.1.   Experimental configuration

Datasets: we validate our MCADFusion model using two publicly available datasets, MSRS ^[44], and TNO ^[45]. We compared the fusion results with seven state-of-the-art fusion models: DIDFuse ^[7], SDNet ^[46], U2Fusion ^[6], TarDAL ^[20], DeFusion ^[47], ReCoNet ^[48], CDDFuse ^[43]. We used entropy (EN), standard deviation (SD), spatial frequency (SF), $Q^{AB/F}$, difference correlation sum (SCD), and structural similarity index measure (SSIM) as six metrics to quantitatively measure the fusion results. Detailed information on these metrics can be found in ^[49].

Implementing rules: The experiments in this study were completed on a high-performance computer configured with Intel(R) Xeon(R) Platinum 8352V CPUs, 90GB RAM, NVIDIA RTX 4090 GPU (quantity: 1), and running Ubuntu 18.04, PyTorch 1.8.1, and CUDA 11.1, which ensured efficient complex model processing and accurate reproduction of complex models. In the preprocessing stage, the training samples were randomly cropped into small blocks of 128 × 128 pixels. The whole training process consists of 120 rounds divided into two phases: 40 rounds in the first phase and 80 rounds in the second phase, with 8 samples processed in each batch. The Adam optimizer was used, and the initial learning rate was set to 0.0001, and the learning rate was reduced by 0.5 after each round of training. The hyper parameters settings of the network included: the number of restorer blocks in the SFE was 4, with 8 attention heads and 64 dimensions. The weight parameters of the loss function were set to 1 for both $\alpha_{1}$ and $\alpha_{2}$, 2.5 for $\alpha_{3}$ and 6 for $\alpha_{4}$, 5 for both $\beta_{1}$ and $\beta_{2}$, 2.5 for $\alpha_{5}$, and 10 for $\alpha_{6}$.

4.2.   Experimental results

Qualitative comparison: In a qualitative comparison, the MCADFusion methodology was scrutinized against several cutting-edge, state-of-the-art (SOTA) techniques. The assessment of these fusion approaches was conducted using the TON dataset, employing two distinct scenarios, as illustrated in Figures 6–8. Furthermore, the outcomes achieved following the integration of multiple scenes are presented on the MSRS dataset, showcasing the effectiveness of the proposed method. In Figure 6, the fused images produced by the DIDFuse, DeFusion, and DATFuse methods exhibit more noise in the red box, and the salient features are not clear. In contrast, the fused images generated by the MCADFusion method contain less noise in the red box and appear more natural. Additionally, the MCADFusion method is more natural in appearance compared to the U2Fusion, DATFuse, and CDDFuse methods. In the blue box, the human thermal imaging information generated by the DIDFuse, U2Fusion, and DeFusion methods is darker, while there is almost no visual difference in terms of human sensitivity compared to the CDDFuse method. In Figure 7, within the group 1 images, the MCADFusion method provides clearer signage information in the red box relative to the DIDFuse, DATFuse, and CDDFuse methods. In the yellow box, the MCADFusion method retains more detailed information. Meanwhile, in the blue box, there is almost no visual difference between the information retained by the MCADFusion method and the DATFuse. CDDFuse methods in terms of human perception. In the second set of images, the fusion results of the DIDFuse method in the blue basket and the red box are darker in terms of feature information, while the CDDFuse and MCADFusion methods retain more detailed feature information. In Figure 8, the results of the MCADFusion method demonstrate superior preservation of content within the red-framed area and the basket. This approach effectively retains both the thermal radiation information from the infrared image and the detailed feature information from the visible image, resulting in a more natural visual effect.

Quantitative comparison: In a quantitative comparison, the MCADFusion method was compared with eight existing SOTA methods. Six metrics, namely entropy (EN), standard deviation (SD), spatial frequency (SF), $Q^{AB/F}$, difference correlation (SCD), and structural similarity index (SSIM), were used to quantitatively assess the fusion results. The quantitative comparison results on the publicly available TNO dataset are shown in Table 1 below. It is obvious from Table 1 that the MCADFusion method has better performance in all four metrics, EN, SD, SF, and SCD, compared to the eight existing SOTA methods. The higher these four metrics are, the better the fused image is in terms of information, contrast, and similarity of detail information to the original image. Meanwhile, the two metrics, $Q^{AB/F}$ and SSIM, have little difference between the metrics in compared with the other eight methods, while the higher of these two metrics indicates that the fused image is better in terms of quality and detail retention, and the structural similarity with the original image. It indicates that the MCADFusion method does not differ much from the other eight methods in terms of image quality and degree of detail retention. The quantitative comparison results on the publicly available MSRS dataset are shown in the following Table 2. It is obvious from Table 2 that the MCADFusion method has comparable or better performance compared to the seven existing SOTA methods in the five metrics of EN, SD, SF, SCD, and SSIM. It can be shown that the fused image obtained by the MCADFusion method performs better in terms of information, contrast, detail information, similarity of the original image, and structural similarity compared to the other 8 SOTA methods. Meanwhile, the $Q^{AB/F}$ metrics are not much different from the other methods, indicating that they are also not much different in terms of the quality of the fused images. The evaluation metrics in Figures 3 and 4 clearly demonstrate that removing various modules from our MCADFusion model results in decreased performance across nearly all metrics. This substantiates the importance and effectiveness of each module within our model.

4.3.   Ablation experiments

We conducted a large number of ablation experiments to validate the plausibility of the different modules in the MCAD model. In these experiments, we used six evaluation metrics such as entropy (EN), standard deviation (SD), spatial frequency (SF), $Q^{AB/F}$, difference correlation (SCD), and structural similarity index (SSIM) to assess the validity and reasonableness of our method. The results are shown in Tables 3 and 4.

First, removing the multi-scale feature extraction module and keeping the other hyperparameters and model framework unchanged showed that not adding the multi-scale feature extraction module was less effective. Second, simplifying the training phase to use only the second phase and performing 120 rounds of training, the results show that two-stage training is better than single-stage training. Third, comparing the effect of using four and three restorer blocks, the experiment shows that the configuration of four restorer blocks is superior to three. Finally, replacing the channel attention module with a module using a 3 × 3 convolutional kernel, the results demonstrate that the channel attention module achieves better fusion results.

We conducted ablation experiments on the hyperparameters $\alpha_{1}$, $\alpha_{2}$, $\alpha_{3}$, $\alpha_{4}$, $\beta_{1}$, and $\beta_{2}$. Specifically, $\beta_{1}$ and $\beta_{2}$ are the weight parameters between the SSIM and MSE losses for the infrared (or visible) image and the reconstructed image, trained during the first stage of the total loss functions. For comparison purposes, we set $\beta_{1}$ and $\beta_{2}$ to 2, 4, or 5. The results indicated that all evaluation indexes were relatively better when $\beta_{1}$ and $\beta_{2}$ were set to 5. This proves that when the weight of the structural loss function is 5, the quality and detail retention of the IR and visible images in the fused images are better.

Additionally, we compared the different weights of the modified parameters $\alpha_{3}$ and $\alpha_{4}$. We concluded that the evaluation indexes were relatively better when $\alpha_{3}$ and $\alpha_{4}$ were set to 5 and 6, respectively. The results of these ablation experiments on the TNO and MSRS datasets are presented in Tables 5 and 6.

4.4.   Extensibility of the MCADFusion method

The MCADFusion method proposed in this paper not only excels in infrared and visible image fusion tasks but also shows significant potential for applications in other areas, such as RGB-D image segmentation and the fusion of point clouds and images. Specifically, MCADFusion can be extended to RGB-D image segmentation by leveraging multi-scale convolution and attention mechanisms. This approach draws on the strategies of depth-aware convolutional neural networks ^[51] and cascaded feature networks ^[52] to enhance segmentation accuracy. More likely, since the MCADFusion method ultimately produces a fused image that incorporates the thermal radiation information and detailed texture features of the infrared and combinable light images, there is a considerable advantage for the method to be used in the future for preprocessing of small target detection ^[53]. Meanwhile, the multi-scale feature extraction module and restormer module proposed in this paper can be easily used to migrate to other methods, especially multimodal image fusion techniques ^[54,55], which can be used to extract features and recover fused medical images of different morphologies from different source images. Additionally, literature ^[56] provides a comprehensive review of RGB-D semantic segmentation, presenting various deep learning methods and their effectiveness in different application scenarios. This offers valuable insights for further optimizing the MCADFusion method.

Moreover, MCADFusion performs well in tasks involving the fusion of point clouds and images, which are relevant to autonomous driving and robot navigation. For instance, in the RGB-IR pedestrian re-identification task ^[57], the method improves re-identification accuracy by fusing RGB and infrared image features. Similarly, in the 3D pedestrian re-identification task ^[58], combining global semantic guidance with local feature aggregation strategies further enhances the fusion of different modal data. Future research can explore the application of the MCADFusion method in more practical scenarios, aiming for breakthroughs across various fields.

5.   Conclusions

This paper presents a novel multi-scale convolutional attention decomposition method for enhanced infrared and visible light image fusion in computer graphics (MCADFusion). The method fuses the thermal radiation information from infrared images and the detail information from visible images by fusing multi-scale convolution, attention mechanism, and decomposition techniques to achieve high-quality fusion of infrared and visible images. Among the infrared and visible image fusion tasks, significant advantages are demonstrated, and experiments on publicly available TNO and MSRS datasets show that the method outperforms existing methods in both subjective visual assessment and objective assessment.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

This study was financially supported by the following projects: Science, and Technology Research Project of Henan Province (No. 242102211110, No. 242102210217); the Key Research Project of Higher Education Institutions of Henan Province (No. 24A510013); and the Open Project of Key Laboratory of Microsystems Technology (No. 6142804231002).

W.Z.W and M.H.D proposed and conceptualized the innovations of this paper. M.H.D and B.Z prepared the experimental data and verified the experimental results in the paper. M.H.D and C.H.W were responsible for the overall framework, implementation, etc., and evaluated the performance of the proposed methodology in this paper. All authors reviewed the manuscript.

The data in this study are available upon request to the second author.

Conflict of interest

The authors declare that there are no conflicts of interest.