Exploring ginseng's potential role as an adjuvant therapy in COVID-19

1.   Introduction

As an image enhancement technology, image fusion is utilized to combine images captured by different types of sensors or under distinct shooting settings, aiming to obtain images with more comprehensive scene representation information ^[1]. Among various applications of image fusion, infrared and visible image fusion serves as a typical example. Infrared images are captured using infrared sensors, relying on thermal radiation. They are characterized by prominent targets, minimal environmental influence, high imaging noise and blurred details ^[2]. On the other hand, visible images exhibit rich texture details, high resolution and sensitivity to lighting conditions ^[3]. Image fusion enables the integration of unique and shared information from both modalities to generate fused images with enhanced texture and salient targets. These fused images play a crucial role in subsequent high-level vision tasks such as semantic segmentation ^[4] and nighttime vehicle target detection ^[5].

The research on infrared and visible image fusion has led to the development of various traditional methods that have been proposed ^[6,7,8,9,10]. These methods are often highly interpretable but rely on hand-designed fusion rules, which can limit their performance when dealing with more complex scene fusion tasks. However, with the advancements in deep learning, an increasing number of deep learning methods are being applied to image fusion tasks ^{[11,12,13,14,15]}. Deep networks exhibit strong capabilities in characterizing image features, surpassing the limitations of traditional feature extraction methods. They adopt a data-driven approach, enabling end-to-end generation of fused images.

In order to enhance the performance metrics of fusion, many existing deep learning-based fusion methods incorporate complex network modules that require more storage and computational resources to achieve improved performance metrics. For instance, Long et al. ^[16] proposed a network that aggregates residual dense blocks, combining dense connected blocks with residual connected blocks. Pu et al. ^[17] introduced a complex contextual information perceptual module for image reconstruction. Xu et al. ^[18] employed dissociative representation learning in an auto-encoder-based approach. These methods have demonstrated performance improvements in fusion results; however, they also introduce greater computational complexity due to the inclusion of complex modules in the network.

Furthermore, existing fusion algorithms often employ fusion layers that incorporate intricate fusion modules or fusion rules, with the primary aim of improving evaluation metrics. However, these algorithms often overlook the characteristics of different modalities. Notably, auto-encoder-based methods ^{[18,19,20,21]} utilize hand-designed fusion strategies for combining depth features. The use of such hand-designed fusion strategies may not assign proper weights to the depth features, leading to limitations in the performance of the fusion methods.

Currently, there is a lack of research on lightweight fusion models, which aim to reduce model parameters and convolutional depth channels. One example is PMGI ^[22], which performs information extraction through gradient and intensity scale preservation. It achieves this by reusing and fusing features extracted with fewer convolutional layers. Another lightweight model, FLFuse ^[23], generates fused images using a weight sharing encoder and feature swapping training strategy to ensure efficiency. However, FLFuse fails to fully extract and fuse image features due to its shallow network channel dimension and simplistic implicit fusion strategy, resulting in subpar visual effects and performance metrics.

We focus on exploring lightweight fusion methods based on structural re-parameterization. Existing structural re-parameterization methods have demonstrated high performance in training and fast inference speeds, making them effective for advanced vision tasks ^{[24,25,26,27]}. They are likely to be crucial in addressing the imbalance between fusion performance and computational resource consumption. However, directly applying these structural re-parameterization blocks designed for high-level vision tasks provides limited improvement for infrared and visible image fusion. Specific structural re-parameterization blocks tailored for fusion tasks are required to efficiently extract richer information from different modal features.

To address the limitations of existing image fusion methods, this paper proposes a novel approach that combines edge operators with structural re-parameterization. This approach enables the rapid generation of fused images with enhanced edge texture information and prominent targets, effectively addressing the imbalance between fusion performance and computational resource consumption. The major contributions of this paper are outlined as follows:

● A fast edge convolution fusion network (FECFusion) for infrared and visible images is proposed, which combines edge operations with structural re-parameterization for the first time to rapidly generate fused images with rich edge texture information and salient target, solving the problem of imbalance between fusion performance and computational resource consumption.

● A structural re-parameterization edge convolution block (RECB) is proposed, which can deeply mine the edge information in the source images and improve the performance of the fusion model without introducing additional inference burden.

● An attention fusion module (AFM) is designed to sufficiently fuze the unique and common information of different modal features to effectively integrate the feature information of the source images with less computational effort.

2.   Related works

2.1.   Infrared and visible image fusion

In the current literature, there are numerous works that primarily focus on preserving texture details in images ^[28,29,30]. In contrast, our work aims to address the challenge of balancing lightweight design and performance in image fusion networks. One approach is IFCNN ^[31], where both the encoder and decoder components employ only two convolutional layers for feature extraction and image reconstruction. Additionally, the fusion rules are adjusted based on the source image type, resulting in a unified network capable of handling various fusion tasks. Another method, SDNet ^[32], tackles the fusion task by incorporating the generated fused image reconstruction into a squeezed network structure of the source image. This forces the fused image to contain more information from the source images. SeAFusion ^[33] utilizes dense blocks with gradient residuals for feature extraction and employs the semantic segmentation task loss to guide the training of the fusion network. Recently, FLFuse ^[23] achieves feature extraction implicitly through a weight sharing encoder and feature swapping training strategy, enabling the generation of fused images in a lightweight and fast manner.

However, existing methods for infrared and visible image fusion only reduce the network model's parameters through conventional lightweight network design approaches, which can lead to a degradation in fusion performance.

2.2.   Structural re-parameterization

Many existing methods for infrared and visible image fusion rely on attention mechanisms or multi-scale feature extraction to enhance network performance, but these approaches often come at the cost of increased computational complexity. Finding networks that effectively extract image features while maintaining high computational efficiency is challenging. In ACNet ^[34], Ding et al. proposed a method to convert multi-branch structures into a single-branch structure, thereby improving the performance of convolutional networks. In another work by Ding et al. ^[35], a concise VGG-like backbone network called RepVGG was introduced. RepVGG utilizes structural re-parameterization as its core technique, enabling efficient feature extraction while reducing network computation. RepVGG has demonstrated excellent performance in target detection tasks. Building upon this work, Ding et al. ^[36] proposed six network structures that can be structurally re-parameterized. The authors also explained the underlying reasons why the structural re-parameterization method is effective. Given the success of structural re-parameterization in various vision tasks, this approach holds promise for addressing the challenge of balancing fusion performance and computational resource consumption in infrared and visible image fusion tasks.

Unfortunately, the direct use of those structural re-parameterization blocks designed for advanced vision tasks provides little improvement for infrared and visible image fusion tasks. They still require the structural re-parameterization block specifically designed for the image fusion task to quickly extract the full wealth of information from the different modal features.

Therefore, we propose a new image fusion method, FECFusion, which substantially reduces computational resource consumption while maintaining high fusion performance through a well-designed structural re-parameterization technique.

3.   Methods

3.1.   Network architecture

In this paper, FECFusion utilizes end-to-end convolutional neural networks to perform feature extraction, feature fusion and image reconstruction, enabling efficient and straightforward fusion tasks. The network architecture, as depicted in Figure 1, consists of three main components: an encoder, a fusion layer and a decoder. In the encoder, a two-branch structure is employed, comprising one convolutional layer and two structural re-parameterization edge convolution blocks. This setup allows for the extraction of depth features from both the infrared and visible images. The fusion layer combines these extracted features, leveraging the complementary information present in the two modalities. Subsequently, the decoder, consisting of three structural re-parameterization edge convolution blocks, reconstructs the hybrid features obtained from the fusion layer to generate the final fused image. Overall, FECFusion offers a simple and efficient solution for infrared and visible image fusion, utilizing end-to-end convolutional neural networks for image fusion of the two modalities.

To ensure that the fused images better retain the edge feature information of the source images, FECFusion has designed a structural re-parameterization edge convolution block(RECB) for improving the performance in infrared and visible image fusion tasks. In addition, we use an attention fusion module (AFM) to better fuse the feature information of different modal images extracted from different branches.

3.1.1.   Structural re-parameterization edge convolution block (RECB)

Although it has some effects to use standard convolution to extract infrared and visible image feature information for fusion networks, it is inferior to complex models in terms of fusion performance. However, replacing standard convolution with complex blocks would make the network consume more computational resources. Therefore, the structural reparameterization technique is introduced in this paper to enrich the characterization capability of the network without increasing the computational resource consumption of the network in the inference stage.

To ensure that the fused images are better able to retain the edge feature information of the source images, we design a structural re-parameterization edge convolution block(RECB) for improving the performance in infrared and visible image fusion task. The specific structure of RECB is shown in Figure 2.

In particular, the RECB consists of four elaborated operators as follows.

1) The branch of standard 3 $\times$ 3 convolution

To guarantee the basic performance of the module, we use a standard 3 $\times$ 3 convolution. This convolution is represented as:

where $F_{n}$ represents the output feature of 3 $\times$ 3 standard convolution. $K_{n}$ represents the convolution kernel weight of 3 $\times$ 3 standard convolution. $X$ represents the input feature. $B_{n}$ represents the offset of 3 $\times$ 3 standard convolution.

2) The branch of feature expansion convolution

The representational power of the fusion task is improved by expanding the channels of the features, which helps to improve the extraction of more feature information. Specifically, the branch uses 1 $\times$ 1 convolution to expand the channel dimension of the features and 3 $\times$ 3 convolution to extract the feature information, which is expressed as:

where $F_{e}$ represents the output feature of the feature expansion convolution branch. $K_{e}$ represents the convolution kernel of 1 $\times$ 1 convolution. $B_{e}$ represents the offset of 1 $\times$ 1 convolution.

3) The branch of Sobel filter

Edge information is tremendously helpful for the performance improvement of the fusion task. Since it is usually difficult for the network model to learn the weights of the edge detection filters through training, a pre-defined Sobel edge filter is embedded in this branch for extracting the first-order spatial derivatives and learning the scaling factors of the filters. Specifically, the input features are firstly scaled by 1 × 1 convolution, then the edge information is extracted by horizontal and vertical Sobel filters, which are processed as follows:

where $K_{x}$, $B_{x}$ and $K_{y}$, $B_{y}$ are the weights, bias of the 1 $\times$ 1 convolution in the horizontal and vertical directions. $S_{D_{x}}$, $B_{D_{x}}$ and $S_{D_{y}}$, $B_{D_{y}}$ are the scaling parameters and bias with the shape of $C \; \times$ 1 $\times$ 1 $\times$ 1. $\otimes$ and $*$ represent DWConv and normal convolution. ($S_{D_{x}} \cdot D_{x}$), ($S_{D_{y}} \cdot D_{y}$) are in the shape of $C \; \times$ 1 $\times$ 3 $\times$ 3.

4) The branch of Laplacian filter

In addition to the Sobel operator for extracting the first-order spatial derivatives, this branch employs a more stable Laplacian edge filter that is more robust to noise to extract the second-order spatial derivatives of the image edge information. Similarly, this branch also uses 1 $\times$ 1 convolution for scaling and then uses the Laplacian operator to extract the edge information, processed as:

where $K_{lap}$, $B_{lap}$ are the weights, bias of the 1 $\times$ 1 convolution. $S_{lap}$, $B_{lap}$ are scaling factors and bias of DWConv, respectively.

In addition, the BN layer is not used in the RECB, unlike the structural re-parameterization block designed for advanced vision tasks, because the BN layer would hinder the performance of the fusion network. Finally, the output features of these four branches are summed and mapped to the nonlinear activation layer:

where $F$ is the output characteristic of RECB. The nonlinear activation layer used in this experiment is LeakyRelu.

The above RECB is the structure of the training phase. After the training is completed, the parameters of the four branch structures are equivalent to a 3 $\times$ 3 convolution parameter through the structure re-parameterization technique, so that the same effect can be obtained only through the 3 $\times$ 3 convolution processing after the structure re-parameterization in the inference phase.

3.1.2.   Attention fusion module (AFM)

Since these two features come from source images of different modalities, they have object focus of different scenes, with complementary information and public information of each other. Thus, the fusion module should have to focus on the fusion of complementary information and public information of different modalities.

In Figure 3, it is evident that detecting pedestrians in visible images at night can be challenging due to inadequate lighting conditions. However, in infrared thermal images, pedestrians are clearly highlighted. Therefore, the key challenge lies in fusing these two features by leveraging the complementary information that exists in only one of the modalities. In the case of a well-illuminated thermal target, such as the vehicle in Figure 3, both cameras are capable of sensing it. During the fusion process, it is important to enhance both features simultaneously. If a method is used that focuses solely on processing the complementary information, there is a risk of weakening one of the features. In order to effectively fuse the infrared and visible features, it is crucial to devise a fusion approach that preserves and enhances the salient information from both modalities. This will ensure that both the infrared highlights and the visible details are effectively integrated, leading to improved detection results.

In order to better solve the problem of fusion of different modal information, the element-by-element addition method for extracting complementary information of heteromodal images and the element-by-element multiplication method for extracting common information of heteromodal images are employed here. The element-by-element addition and element-by-element multiplication methods are expressed as:

where $X_{vi}$ and $X_{ir}$represent the depth features of infrared and visible images extracted by the encoder, respectively. Element-by-element addition $X_{add}$ represents the addition of elements to visible image features and thermal target features to accumulate complementary information of different modes, while element-by-element multiplication $X_{mul}$ represents the multiplication of elements to visible image features and thermal target features to enhance common information of different modes.

As in the simple example in Figure 4, 0 represents that the target is not sensed by the sensor and 1 represents that the target is sensed by the sensor. Suppose there are two cases of sensed and un-sensed target for infrared and visible sensors, respectively, which are illustrated by 0 and 1. Therefore, four cases are generated, and the goal is to retain all the information as long as possible. Element-by-element addition preserves the target information to the maximum extent possible, the left in Figure 4, with the target being sensed by at least one of the sensors. Element-by-element multiplication allows filtering out the common information, the right image in Figure 4, with the target needing to be sensed by two sensors at the same time. Once the two types of information are obtained separately, they are preserved by feature concatenation, combining the unique information from both modalities. This approach allows for the retention of crucial information while effectively combining the features extracted from each sensor.

Therefore, the attention fusion module (AFM) based on element-by-element addition and element-by-element multiplication is designed to better fuse the information of these two depth features. The specific structure of the AFM is shown in Figure 5.

The upper branch in AFM is used to enhance the common information of different modal features, while the bottom branch enhances the feature information of different modal features through the attention module and aggregates the complementary information of different modal features by feature summation, and then cascades the common and complementary information to allow both feature information to be retained as much as possible.AFM is processed as follows:

where $Cat$ represents concatenation; $A$ represents the attention module, and CBAM is used in this article.

3.2.   Loss function

The loss function is the key to guide the training of deep neural networks to achieve the desired results. The loss of structural similarity is often used to maintain a clear intensity distribution of the fused image. However, in the fusion task, the fused image needs to be similar to the two source images at the same time, and there is more complementary information between the two source images, which will weaken the complementary information region and lead to a decrease in fusion performance.

In order to better promote the recovery of texture details, this paper uses content loss $L_{content}$ and traditional loss $L_{tradition}$ to jointly constrain the training of the network. The total loss $L_{fusion}$ formula is as follows:

where $\lambda$ is the weight coefficient to balance these two losses.

3.2.1.   Traditional loss

The design of traditional loss function can enhance the similarity between the fused image and the two types of source images, guide the network to generate the fusion result with complete information faster, and avoid the single and incomplete information in the fusion result. The traditional loss $L_{tradition}$ calculation formula is :

where $I_{f}$ represents the fused image. $I_{ir}$ and $I_{vi}$ represent the infrared and visible images. $\left\| \cdot \right \|$ refers to the $l_{1}$-norm.

3.2.2.   Content loss

In order to promote the model to fuse more meaningful information, retain the saliency in the infrared image and the edge texture information of the source image, the content loss $L_{content}$ with bilateral filtering is designed in this paper. The content loss consists of two parts: the intensity loss $L_{in}$ and the edge gradient loss with bilateral filtering $L_{grad}$. The formula is as follows:

where $\mu_{1}$, $\mu_{2}$ are the weighting coefficients for balancing these two losses.

Among them, the intensity loss $L_{in}$ constrains the overall apparent intensity of the fused image. In order to better retain the salient target, the pixel intensity of the fused image should be biased towards the maximum intensity of the infrared and visible images. The formula of strength loss is as follows:

where $Max(\cdot)$ stands of the element-wise maximum calculation.

In addition, in order to make the network model better preserve the edge texture details of the fused image, the existing methods use the maximum edge gradient of the source image to constrain the training of the network, but this loss is easily affected by noise in the infrared image. To this end, this paper uses a bilateral filter that preserves the edge gradient to denoise the infrared image, thereby reducing the noise of the fused image. The following is the calculation formula of edge gradient loss of bilateral filtering:

where $\nabla$ is the gradient operator for measuring image texture information, and the $\mathit{Sobel}$ operator is used to calculate the gradient in this paper. $|\cdot|$ indicates the absolute operation. $\mathit{Bila}$ represents a bilateral filter.

4.   Experimental analysis

4.1.   Experimental details

In this paper, FECFusion is trained with the MSRS dataset ^[37]. Since the existing infrared and visible image fusion dataset is small, the MSRS training set part of the infrared and visible images common dataset is expanded from 1083 to 26,112 pairs of images, and the size of the training set image pairs after data enhancement is 64 pixel × 64 pixel, which can basically meet the training requirements. In order to evaluate the effectiveness of FECFusion, it is tested on the test set of the MSRS dataset and picks 361 pairs of images as the test subject. In addition, to more comprehensively evaluate the generalization performance of FECFusion, it selected 42 and 300 pairs of images on the TNO^[38] and M3FD^[39] datasets, respectively, for generalization comparison experiments.

Since none of the current public datasets for infrared and visible image fusion have reference images, the quality of the fusion result images cannot be directly evaluated by ground truth, therefore, we evaluate the visualization image effects of different algorithms by human subjective visual perception as a qualitative assessment, and by objective generic image quality evaluation index results as a quantitative assessment.

In this paper, standard deviation (SD) ^[40], mutual information (MI) ^[41], visual information fidelity (VIF) ^[42], sum of correlation differences (SCD) ^[43], entropy (EN) ^[44] and $Q_{abf}$ ^[45] are used. SD evaluates the contrast and distribution of the fused images from a statistical point of view. MI measures the amount of information from the source image to the fused image. VIF reflects the fidelity of the fused information from a human visual point of view. SCD measures the difference between the source image and the fused image. EN measures the amount of information contained in the image. $Q_{abf}$ evaluates the amount of fused edge information from the source image. All the above metrics are positive metrics, and higher values mean better fusion results.

FECFusion is compared with seven fusion algorithms, including DenseFuse^[46], FusionGAN^[47], IFCNN^[31], SDNet^[32], U2Fusion^[48], FLFuse^[23] and PIAFusion^[37]. All the compared algorithms are experimented in public code, where the relevant settings of the experiments are kept constant. In the superparameter settings of the proposed network, the network optimizer uses Adam, epoch = 10, batch size = 64, learning rate is 1 $\times \; 10^{-4}$, loss function parameters are $\lambda = 10$, $\mu_{1} = 12$, $\mu_{2} = 45$. The parameters of bilateral filtering are $\sigma_{d} = 0.05$, $\sigma_{r} = 8.0$, and window size is 11 $\times$ 11. The training process of FECFusion is summarized in Algorithm 1.

Besides the comparative and generalization experiments, the effectiveness of RECB and AFM is verified by ablation experiments in this paper. In addition, FECFusion is verified to be helpful for the advanced vision task through segmentation experiments. Finally, we have compared the operational efficiency of FECFusion with other methods and compare the computational resource consumption with and without structural re-parameterization. Our experiments are all conducted on a GeForce RTX 2080Ti 11GB and an Intel Core i5-12600KF, with PyTorch of a deep learning framework.

4.2.   Comparative experiment

4.2.1.   Qualitative results

It is an important challenge for the image fusion algorithm to generalize the performance of different scenes. In the MSRS dataset, we have chosen two daytime and two nighttime images to evaluate the subjective visualization performance, and the comparison results are shown in Figures 6 and 7. We mark the texture detail information with green boxes and the highlighted target information with red boxes.

In the daytime scene depicted in Figure 6, we can observe the performance of different fusion methods. DenseFuse, SDNet and U2Fusion fail to effectively highlight the infrared target and do not fully utilize the background information present in the visible image. FusionGAN manages to highlight the salient target to some extent, but it can be seen from the green box that it blurs the background. In contrast, IFCNN and FLFuse weaken the texture details of the background, as evident from the green box. Only PIAFusion and the method proposed in this paper successfully integrate the relevant information, effectively preserving both the infrared target and the background texture details. Therefore, it is evident that the method proposed in this paper exhibits superior performance in the daytime scene, achieving a balanced fusion result that highlights the target while retaining the background information.

In the night scene depicted in Figure 7, the visible image contains limited texture information, while the infrared image contains both background texture details and a salient target. Many existing fusion methods tend to overemphasize the information from one modality, making it challenging to achieve satisfactory results across different scenes.

Among the fusion methods examined, DenseFuse, SDNet and U2Fusion exhibit a bias towards infrared images, and weaken infrared targets. FusionGAN introduces artifacts into the fused images. Only IFCNN, PIAFusion, and the method proposed in this paper are capable of generating fused images with higher contrast in black night scenes. FLFuse, which is also a lightweight method, performs poorly in this scenario. It fails to fully leverage the characteristics of both modal images, leading to a degradation in both the background and the infrared target. Therefore, the method proposed in this paper demonstrates good performance in night scenes as well. It effectively captures the characteristics of both modal images and achieves better contrast, thereby preserving the infrared target while retaining background details.

4.2.2.   Quantitative results

In this section, we perform quantitative evaluation on the MARS dataset and select six metrics for evaluation. The comparison of the metrics of different methods is shown in Table 1, where red represents the best result and blue represents the second one.

From Table 1, it is clear that our method shows significant advantages in four metrics, SD, VIF, SCD and EN, while its performance in MI and Qabf is second only to PIAFusion. The value of SD is the best indicating that the fusion result of this paper method Shencheng achieves high contrast between infrared target and background; the highest value of VIF indicates that the fused image generated by this paper method is more in line with The highest value of VIF indicates that the fused images generated by this method are more consistent with the human visual system; the highest values of SCD and EN indicate that this method can generate fused images with more edge details and contain more realistic results than other methods. In conclusion, the quantitative experimental results show that this method can generate fused images with more information while reducing the computational effort.

4.3.   Generalization experiment

In the fusion task, it is required that the fusion model has a stronger generalization capability, which is applicable in different scenes. Therefore, we selected 20 and 300 pairs of images in the TNO and M3FD datasets, respectively, to evaluate the generalization ability of FECFusion. A qualitative comparison of the different algorithms on the TNO and M3FD datasets is presented in Figures 8 and 9.

From the figures, it is evident that DenseFuse, SDNet, U2Fusion and FLFuse tend to blend the background and the target together, making it difficult to distinguish the salient infrared target. FusionGAN, on the other hand, exhibits high overlap with the infrared image and lacks the inclusion of background information. In comparison, IFCNN and PIAFusion show performance similar to the proposed method in this paper. However, it is important to note that these methods may not match the inference speed of the proposed method, which offers faster processing capabilities. Therefore, based on objective evaluation and considering the faster inference speed, the proposed method in this paper demonstrates competitive performance and provides a promising solution for infrared and visible image fusion tasks.

The results of quantitative metrics for the generalization experiments are shown in Table 2. The metrics performance of our method is the best or the second best on both datasets, which indicates that our method can both preserve the texture details of the source image and improve the contrast of the target. In conclusion, the qualitative and quantitative results show that FECFusion performs excellently in generalization. In addition, the method in this paper effectively maintains the intensity distribution of the target region and preserves the texture details of the background region, benefiting from the proposed RECB and AFM.

4.4.   Ablation experiment

In order to verify the effectiveness of adding RECB and AFM to our FECFusion, ablation experiments are designed in this section to further analyze the role of these two proposed modules in the network model. First, RECB is a structure that equates multiple branch structures into a single branch structure by structural re-parameterization; therefore, in the ablation experiments, the RECB part is directly replaced with the structure after structural re-parameterization for training, i.e., the structure of a single ordinary convolution. For the ablation experiments of AFM, the network is trained with a direct feature cascade instead of AFM. This experiment is performed on the MSRS dataset, and the experimental results are shown in the Figure 10, where the background texture is marked with a green solid box and the infrared salient targets are marked with a red solid box.

From the experimental results, it can be seen that without RECB, the fused images are blurred at the edges to some extent, which proves that the module contributes to maintaining the edge information of the fusion results. Without AFM, the saliency of the fusion results decreases. If both RECB and AFM are not available, the fusion results have a decreased target significance and blurred edge texture. The evaluation metrics for this ablation experiment are shown in Table 3. We observe that the absence of both RECB and AFM leads to a decrease in the evaluation metric values to different degrees, proving the effectiveness of each part of our FECFusion.

4.5.   Efficiency comparison experiment

To verify the execution efficiency of the proposed algorithm, the average processing time of forward propagation of each fusion method is tested on the MSRS dataset in this paper, and the comparison results are shown in Table 4 where red represents the best and blue represents the second best. It can be seen that our method is more efficient than most methods, while FLFuse is faster than our method, the method in this paper works better, so these differences in running efficiency are acceptable.

In addition, with reference to the image size used in the MSRS data set, the data of 640 $\times$ 480 $\times$ 1 is used as the input of the network forward propagation, and the parameters and weights of the network model are calculated by the TorchSummary library. The forward propagation time, running storage space, parameter quantity, weight size and the cumulative deviation of the fusion results pixel by pixel before and after the structural re-parameterization are compared experimentally. The comparison results are shown in Table 5.

By comparing the results, it can be seen that there is almost no difference in the fusion results of the network before and after the structural re-parameterization, indicating that the structural re-parameterization can effectively reduce the running time, running storage space, parameter quantity and weight size in the case of very low deviation.

4.6.   Segmentation experiment

Semantic segmentation algorithms are an important general-purpose computer vision method whose performance reflects well on the semantic information of the fused resultant image. To verify that the fused images can be helpful for subsequent vision tasks, DeepLabV3+^[49], a semantic segmentation model pre-trained on the Cityscapes dataset^[50], is also used in this section to evaluate the performance of the fused images, and the semantic segmentation results are shown in the Figure 11.

From the experimental results, the semantic segmentation results of the fused result images are all a little better than the infrared and visible images, especially at night when the lighting conditions are poorer, the visible sensors have difficulty capturing enough information, and the semantic segmentation models often have difficulty detecting hot targets such as travelers, so to some extent, it can be shown that the image fusion has an enhanced effect on subsequent vision tasks.

5.   Conclusions

In this paper, we propose FECFusion, an infrared and visible image fusion network based on fast edge convolution. The network consists of several key components. First, the main part of the network employs the RECB to extract features, including detailed texture features and salient image features. These extracted features are then fused using the AFM, and the fused image is reconstructed. After the completion of training, the network undergoes a structural re-parameterization operation to optimize the inference speed and storage space required while preserving the original training effectiveness. Through subjective and objective experimental results, we demonstrate that FECFusion achieves superior fusion results compared to other algorithms. It offers better real-time performance and requires less inference memory footprint, making it more suitable for practical engineering applications that involve the design of custom hardware accelerated circuits. In future research, we will explore specific applications of FECFusion on mobile devices and further optimize its performance. This includes enhancing the network's ability to learn multi-scale image features and achieve better fusion results with lower computational resource consumption.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

This work is supported by National Natural Science Foundation of China(NO.62266025).

Conflict of interest

The authors declare there is no conflict of interest.