ACRnet: Adaptive Cross-transfer Residual neural network for chest X-ray images discrimination of the cardiothoracic diseases

1.   Introduction

Cardiothoracic diseases mainly include lung disease and heart disease. The lung is connected with the outside world and it is the main organ of the respiratory system, and the blood of the whole body flows through it. The heart plays a major role in the body's circulatory system, and it pushes blood around the body and carries away metabolic waste. Both lung and heart are important organs for us. According to the data from World Health Organization, lung disease is the third most lethal disease in the world, while heart disease is the first one. All over the world, cardiothoracic diseases cause more than ten million deaths every year.

There are many kinds of cardiothoracic diseases, among which emphysema and cardiomegaly are two common diseases. The characteristic pathological symptoms of emphysema are shown as the increase of the lung volume, the excessive dilation of the bronchial, the decrease of the bronchial airway elasticity and the destruction of the airway walls. The disease is usually caused by smoking, industrial dust, chronic bronchitis and pulmonary inflammation. Emphysema can be sub-divide into senile one, compensatory one, interstitial one and obstructive one so on. The emphysema physiological symptoms include chronic cough, sputum, tightness of chest and shortness of breath. In the chest radiograph of emphysema, bronchial swelling or lung bubble, can be obviously observed. Cardiomegaly is shown as an increase of the cardiac load caused mainly by circulatory and respiratory diseases, and it can be subdivided into physiological one and pathological one. Cardiomegaly symptoms are typically similar to ones of emphysema, including cough, chest tightness, heart palpitations, and dyspnea. The coronary heart disease, hypertension and corpulmonale, are the main pathological causes of cardiomegaly. The chest radiograph of cardiomegaly is often shown as enlargement of the right heart with pulmonary hypertension. In fact, cardiomegaly is often secondary to lung diseases including emphysema, and the symptoms of them are similar, so there is a close relationship between emphysema and cardiomegaly. While emphysema and cardiomegaly are two diseases that require different medications and treatment regimens, so it is important to accurately diagnose them.

Chest X-ray is the most commonly used medical technology for the diagnosis of cardiothoracic diseases, including emphysema and cardiomegaly. It is very effective in identifying and detecting cardiothoracic, pulmonary and interstitial diseases, and plays an important role in the treatment of cardiothoracic diseases ^[1]. Accurately analyzing patients' condition is a huge challenge for radiologists, and computer-aided diagnosis can reduce differences of the radiologists' conclusions and provide the valid reference for clinicians in abnormal cases. In recent years, we have been using deep learning in many fields which includes medical research and computer vision, and have gotten significant progress ^[2,3,4,5,6]. Related studies include the detection of pneumonia ^[7,8,9], pneumothorax ^[10], tuberculosis ^[11], pulmonary nodules ^[12,13], lung cancer ^[14,15,16], and COVID-19 ^[17,18,19]. In this work, we construct a valid machine learning model to recognize cardiomegaly, emphysema and normal by the chest X-ray images. Our contributions are as follows.

1) We construct an adaptive cross-transfer residual neural network to identify three kinds of chest X-ray images. To identify better, we alternately transmit the feature information extracted by adaptive structure and residual block to different layers.

2) We use VGG16, InceptionV2, ResNet101 and CliqueNet to compare the recognition ability with ACRnet. The results show that the recognition performance of ACRnet is better than that of the four classical models. In addition, compared with state-of-the-art convolutional neural networks, ACRnet has better recognition ability too.

3) To further improve the recognition ability of ACRnet, we use DCGAN to expand the original dataset. Compared with the previous results, the recognition ability of ACRnet is improved further, when ACRne is trained by the expanded dataset.

The paper is structured as follows: in Section 2, related works about the deep learning model for the X-ray images recognition are introduced. In Section 3, the design and tuning of the ACRnet are illustrated. Experiment result and analysis are given in Section 4. Finally, conclusions are stated in Section 5.

2.   Related works

Here we summarize the current literature on deep learning technology for medical image detection. In the early researches, lacking data is a big problem for the application of deep learning in chest X-ray image diagnosis. To solve this problem, Qasim et al. ^[20] used data expansion technology to increase the chest X-ray images of COVID-19 from 219 to 1000, and the new image dataset was used to train models to recognize chest X-ray images. By using these synthetic images to expand the existing data, the diversity of training data was increased and the classification performance was improved. Loey et al. ^[21] used AlexNet, GoogleNet and ResNet18 to detect chest X-ray images of COVID-19, normal, bacterial and viral pneumonia. In order to improve the recognition accuracy, they used the GAN network to expand the original dataset. In their research, AlexNet reaches the highest detection accuracy of 85.2%. Although GAN was helpful to improve the detection accuracy of the model, for the shallow and simple networks, it was hard to get a satisfactory performance. In order to improve the identification accuracy, Lu et al. ^[22] used the image feature fusion technology to diagnose COVID-19 in lung window computed tomography (CT). Initially, ResNet18 and ResNet50 were selected as the backbone deep networks to generate corresponding image representations from the CT images. Secondly, the representative information extracted from the two networks was fused by discriminant correlation analysis to obtain refined image features. Thirdly, three randomized neural networks (RNNs): extreme learning machine, Schmidt neural network and random vector functional-link net were trained using the refined features, and the predictions of the three RNNs were integrated to get a more robust classification performance. In another study, Lu et al. ^[23] utilized transfer learning to obtain the image-level representation (ILR) based on the backbone deep convolutional neural network. Then, a novel neighboring aware representation (NAR) was proposed to exploit the neighboring relationships between the ILR vectors. To obtain the neighboring information in the feature space of the ILRs, an ILR graph was generated based on the k-nearest neighbor algorithm, in which the ILRs were linked with their k-nearest neighboring ILRs. Afterward, the NARs were computed by the fusion of the ILRs and the graph. On the basis of this representation, a novel end-to-end COVID-19 classification architecture called neighboring aware graph neural network (NAGNN) was proposed and achieved good results. To detect better tuberculosis, Duong et al. ^[24] conceptualized a framework by adopting three recent deep neural networks as the main classification engines, and the three networks were called modified EfficientNet, modified original Vision Transformer, and modified Hybrid EfficientNet with Vision Transformer. Moreover, they empowered the learning process by various augmentation techniques. To further study the proposed approach, they compared it with two state-of-the-art systems. The results showed that their method had the best detection effect. Ferreira et al. ^[25] used the DenseNet model to detect cardiomegaly in chest X-ray images and they used a public dataset to train the neural network, and a private dataset network to verify the generalization ability of the model. DenseNet obtained an AUC value of 0.818 on the public dataset and 0.809 on the private dataset. In the detection of lung diseases, some scholars made more efforts to achieve better detection results, some advanced methods were used, and some classical neural network models were transformed into more complex structures, unfortunately, the results were still dissatisfactory. To further improve the ability to identify lung diseases, Wu et al. ^[26] proposed a vision transformer (ViT) model to classify the emphysema subtypes via CT images. The average accuracy of the pre-trained ViT model achieves 95.95% in their lab's own dataset, which is higher than that of AlexNet, Inception-V3, MobileNet-V2, ResNet34, and ResNet50. In some recent studies, Rajpurkar et al. ^[27] developed the CheXNeXt network model by using Chest X-ray 8 dataset, and compared the AUC values obtained by the model with those obtained by cardiothoracic radiologists. The AUC values of cardiomegaly and emphysema obtained by radiologists were 0.888 and 0.911 respectively, while those obtained by CheXNeXt were 0.831 and 0.704. CheXNeXt's performance in differentiating cardiomegaly and emphysema is far from that of radiologists. For the Chest X-ray 14 dataset, Angelica et al. ^[28] proposed GraphXNET, and achieved good results on it. Wang et al. ^[29] proposed TieNet. Compared with the previous classical neural networks, TieNeT significantly improved the recognition ability of chest X-ray images (average AUC increase by 6 %). Zhao et al. ^[30] proposed AM_DenseNet for chest X-ray image classification. This model used a dense connection network and added a attention module at the end of each dense block to optimize the feature extraction ability. They used the Focal Loss function to solve the data imbalance problem. The average AUC of AM_DenseNet in detecting the chest diseases reached 0.8537. For different types of lung diseases, some scholars had proposed their own state-of-the-art methods, but the detection effect still was expected to further increase. Further research found both the poor model structure and the noise of the data might lead to low accuracy of the detection of lung diseases. Some scholars have mentioned that although the total number of the Chest X-ray 14 dataset exceeds 100,000, a large number of chest X-ray images have more than two diseases ^[31]. If the data is directly classified without screening, it will cause great interference to the model and seriously affect the discriminant result. So in this work, we select only cardiomegaly, emphysema and normal from the Chest X-ray 14 dataset for research. In previous studies, scholars have proposed a large number of methods and models to identify chest X-ray images of cardiothoracic diseases and achieved good results. While due to strong relation and similarity between emphysema and cardiomegaly diseases, it is difficult to better distinguish them, so it is still necessary to build a more high precision recognition model in order to decrease the misdiagnosis rate. In this work, we construct a network model, ACRnet, for the identification of cardiomegaly, emphysema and normal. Compared with previous research, ACRnet improves comprehensively the identification performance of cardiomegaly and emphysema in a variety of evaluation indexes.

3.   Proposed approaches

3.1.   Design

This section describes the overall process of using the expanded dataset to train ACRnet. As shown in Figure 1, we firstly use DCGAN to synthesize cardiomegaly and emphysema chest X-ray images, and then, send the expanded dataset into ACRnet. After several rounds of feature extraction, the input images will be finally recognized by the model and the category label will be output. We call module A as a block, and ACRnet is composed of multiple blocks in series.

3.2.   Data description

The images in our study come from the Chest X-ray 14 dataset (Download address: https://www.kaggle.com/nih-chest-xrays/data), and each image has single or multiple pathological markers. Radiological reports show that the accuracy of pathological markers is more than 90% ^[31]. We select images with cardiomegaly or emphysema from Chest X-ray 14 dataset, and add normal images as the dataset. We divide all data into 80% for training and 20% for test. Table 1 shows the number of chest X-ray images screened from the Chest X-ray 14 dataset for training and test. To get better performance, we used DCGAN to produce 2000 images of cardiomegaly and emphysema, respectively. Since there are more than 60,000 normal images in the chest X-ray 14 dataset, we select 2000 normal images together with the synthesized images to expand the original train set. In order to meet the requirements of the model for the input size, we resize both the original 1024 × 1024 pixels image and the 256 × 256 pixels image synthesized by DCGAN into 224 × 224 pixels image.

3.3.   Deep convolution generative adversarial network

Deep convolutional generative adversarial network (DCGAN) is a special deep learning model which combines convolutional neural network (CNN) and generative adversarial network (GAN) ^[32]. DCGAN consists of two different types of networks, one is a generator network for synthesizing images, the other is a discriminator network for identifying real and synthetic images. In the process of training DCGAN, the discriminator and generator are trained simultaneously. The generator network used in the study consists of 7 transposed convolution layers, 6 Rectified Linear Units (ReLU) layers, 3 batch normalization layers and 1 Tanh layer on the end. The discriminator network consists of 7 convolution layers, 6 ReLU layers, 3 batch normalization layers and a sigmoid layer on the end. Each convolutional layer has 16 filters which are the same window size of 4×4 pixel. DCGAN network can solve the problem of insufficient training caused by small dataset. DCGAN is trained by using all the chest X-ray images of emphysema and cardiomegaly from the original dataset. After 1000 iterations, we use DCGAN to produce 2000 chest X-ray images of emphysema and cardiomegaly to expand the training dataset. Table 2 shows the model structure and parameters of generator and discriminator in DCGAN.

3.4.   Overview of convolutional neural networks

All of the CNNs are trained in a feedforward & feedback fashion. Input information is transmitted feedforward from the initial layer to the output layer, and errors are propagated feedback from the last layer. In this section, we will introduce 4 neural networks used in the experiments. VGG is a typical and effective neural network and it has 6 different structures, in which VGG16 and VGG19 are commonly used ^[33]. The VGG model usually consists of 5 blocks, and each contains several convolution layers and a pooling (Pooling Function) layer. VGG16 includes 13 convolution layers, 3 fully connected layers and 5 pooling layers. The convolution layer of VGG16 adopts 3 × 3 convolution kernels, and the pooling layer adopts 2 × 2 pooling kernels.

InceptionV2 is an upgraded version of GoogleLeNet (InceptionV1) ^[34]. InceptionV1 has a 22-layer network structure with four channels, and convolution kernels of 1 × 1, 3 × 3 and 5 × 5 sizes are used at the same time. Since convolution kernels with different sizes have different receptive fields, InceptionV1 can better learn different scales features. In addition, in InceptionV1, visual information is aggregated at different scales, so that the subsequent network level can extract different scales features. InceptionV2 firstly use batch normalization to accelerate network training and prevent gradient from disappearing ^[35]. InceptionV2 also replaces every convolution kernel of 5 × 5 in InceptionV1 with two 3 × 3 convolution kernels, and the change reduces the parameters and enhances the nonlinear processing capability when the receptive field is kept the same.

ResNet is a widely used neural network with 5 structures of 18-layer, 32-layer, 50-layer, 101-layer and 152-layer ^[36]. The residual block in ResNet uses the jump connection structure to alleviate the gradient disappearance. Using residual structure not only improves the accuracy of the model, but also makes the model to converge faster.

In the block of CliqueNet ^[37], each layer is the input, is also the output of other layers, and all these layers constitute a ring structure and update alternately. This structure makes high-level visual information be sent back to the previous level for feature reuse. In addition, the introduction of attention mechanism inhibits the irrelevant neurons representing background and noise, which further improves the recognition ability of the model.

3.5.   Constitution of ACRnet

Compared with the shallow neural network model, the deep model can better extract image features and has better recognition ability. However, with the increase of network layers, it becomes difficult to train, and the accompanying gradient disappearance problem will reduce the accuracy of the model, while the residual structure is a good chance to solve this problem. In this study, we construct a deep neural network model with residual structure, and at the same time, we add adaptive structure to the model. In order to avoid the reduction effect of residual structure on adaptive structure when two different structures are put together, we adjust the feature transfer mode of residual and adaptive, and construct a residual and adaptive alternate transfer structure. The structure not only achieves high accuracy but also can converge faster. The experimental results show that our model structure has a good ability to recognize emphysema and cardiomegaly. When we compared ACRnet with four classical neural network models and state-of-the art methods, it is found that ACRnet has the best recognition effect in the identification of emphysema and cardiomegaly.

This section introduces the structure of ACRnet, which is a 97-layer neural network. In ACRnet, we construct an adaptive cross-transfer residual structure to extract feature information to improve the recognition efficiency of the model. Adaptive learning is to obtain the importance of different features through autonomous learning, and then suppress secondary features and enhance main features according to importance ^[38]. In the adaptive module, the adaptive global average pooling layer (Adaptive Avg Pool) is used to compress the feature along the spatial dimension, and each two-dimensional feature is transformed into a figure which has global sensing ability and reflects the feature distribution. And then we use 1 × 1 convolution kernel (input channel is a, output channel is a/4) for dimension reduction, then use ReLU activation function to increase nonlinearity, and finally use another 1 × 1 convolution kernel (input channel is a/4, output channel is a) for dimension increase, so as to reduce the computational cost. In the last layer of the adaptive structure, the sigmoid function is used to generate a figure between 0 and 1 as the output result. The adaptive structure generates a characteristic adaptive coefficient between 0 and 1, and the residual structure generates a matrix. When the adaptive figure is passed to the later network hierarchy, it means that the eigen-matrix is multiplied by a weight coefficient. If both adaptive and residual structures transfer feature information from the same layer to the same subsequent network layer, it means that the feature matrix generated by the residual structure will be added to a matrix multiplied by the adaptive coefficient, so the result of the adaptive function will be weakened by the residual structure. In order to extract image feature better, we combine adaptive and residual structure to construct a deep neural network model. In ACRnet, the residual structure transmits the features of layer 0 and layer 1 to the end of layer 3 and 4. The adaptive structure transmits the features of layer 0 and layer 2 to the end of layer 2 and layer 4. This structural design is valid to avoid the reduction effect of residual structure on adaptive function. Figure 2 demonstrates ACRnet's adaptive cross-transfer residual block.

3.6.   Hyper-parameters tuning

In our work, we try to estimate the influence of various hyper-parameters on the performance of the depth model. We divide all the parameters into external parameters and internal parameters and we fine-tune the parameters respectively. The external parameters of the model include optimization algorithm, learning rate and batch size, which affect model performance very much. We find that Adam is significantly better than Ada-delta or SGD, so it is selected as the optimization algorithm. Learning rate affects the convergence speed of the model, but excessive learning rate will reduce the accuracy of the model. We tried seven different learning rates {1e-3, 1e-4, 1e-5, 3e-3, 3e-4, 3e-5, 3e-6} and find it's better on the condition of 1e-4 or 3e-4. Through experimenting repeatedly, we select 32 as batch size from {16, 24, 32, 48, 64}. The internal parameters in the neural network include the size of convolution kernel, stride, the size of pooling layer window and channel ^[39]. In the VGG16, InceptionV2, ResNet101 and CliqueNet, we choose the original internal parameters. Table 3 shows the structure and parameters of ACRnet model.

4.   Experiments

4.1.   Performance evaluation

We choose Inception Score (IS) ^[40] and Fréchet Inception Distance (FID) ^[41] as indexes to evaluate the quality of synthetic images. For the universal standard, both of IS and FID should be gotten by the InceptionV3 model pre-trained on ImageNet dataset ^[42,43]. The higher IS, the lower FID, and the better image quality. We introduce accuracy (ACC) ^[44], Precision, Recall and F1-score ^[45] as indicators to evaluate the performance of the detection model. The formulas for these indicators are as follows.

Here TP, FP, TN and FN represent the number of true positive, false positive, true negative and false negative, respectively. In addition, in order to further evaluate the model performance, we use the confusion matrix ^[46,47,48], plot the receiver operating characteristic (ROC) curve and get the area under the curve (AUC). The higher the AUC, the better performance.

4.2.   Results

This section introduces the result of recognizing cardiomegaly, emphysema and normal by ACRnet and other four classical neural networks: VGG16, InceptionV2, ResNet101 and CliqueNet. In order to improve the recognition ability of ACRnet, we use DCGAN to expand the original train-dataset. The original images are all 1024 × 1024 pixels, and the images generated by GAN are all 256 × 256 pixels. Before we use DCGAN to generate the image, the original image is converted into 256 × 256 pixels. We convert all images into 224 × 224 pixels before feeding them into the networks for final disease identification. In addition, due to the high quality of the original image, we do not carry out de-noising, sharpening or clipping before the original image is input into the neural network. All of our experiments are done on a personal computer with an NVIDIA GTX2080Super (8 G) GPU and we use the PyTorch framework. The training process of The DCGAN and all of the CNNs is finished on the 1000 training epochs. The size of the trained ACRnet model is 11,780 K. Saving the trained model can make it easier to import the model into a computer for disease detection in practical applications. Figure 3 shows the original images from the dataset and the synthetic images generated by DCGAN, which are very similar. We calculate the IS scores of them respectively, and find that the scores are very close, and it means that the quality of the synthetic images is good. This conclusion is proved after calculating the FID scores, because we sent them into the detection model simultaneously and obtain a relatively low FID value. Table 4 shows the IS and FID scores.

Table 5 shows the experimental results of different neural networks. Models are trained on all the available data in each training set, and global assessment is presented by 5-fold cross validation on the whole dataset. When using four classical neural networks to identify cardiomegaly, the recall index obtained by CliqueNet is at a high level. The precision index obtained by InceptionV2 and ResNet101 are better than that obtained by VGG16 and CliqueNet. The F1-scores obtained by the four classical neural networks are close. The AUC index of InceptionV2 is lower than the other three networks. When using four classical neural networks to identify emphysema, CliqueNet achieves the optimal precision score. The recall and F1-score obtained by InceptionV2 and ResNet101 are significantly better than CliqueNet and VGG16. The recall index obtained by CliqueNet is the lowest. The AUC index obtained by InceptionV2 and CliqueNet are higher than the other two neural networks. When identifying normal, the F1-score and AUC index obtained by the four classical neural networks are relatively close. VGG16 achieves the best precision index and ResNet101 achieves better recall index than other networks. Overall, the existing four classical neural networks: VGG16, InceptionV2, ResNet101 and CliqueNet have their respective advantages in recognizing cardiomegaly, emphysema and normal chest X-ray images. But none of them are very good at identifying all three types of chest X-ray images. ACRnet is a deep network model with cross residual structure and the adaptive mechanism, and can extract image features better. Compared with the four classical network structures, ACRnet has higher precision, recall, F1-Score and AUC indexes in recognizing the three types of chest X-ray images. After data expansion by DCGAN, the recognition ability of ACRnet is further improved. Figure 4 shows intuitively the detection result of all models for three types of chest X-ray images.

Table 6 shows mean ACC and average-AUC index obtained by ACRnet and four classical neural networks using 5-fold cross validation. The result shows that the ACC and average-AUC values of four classical neural networks in identifying cardiomegaly, emphysema and normal are close. The overall recognition performance of ACRnet trained by the original dataset is better than that of other 4 classical neural networks. The recognition ability of ACRnet is further improved on the expanded dataset. Figure 5 shows the confusion matrixes obtained by neural networks using 5-fold cross validation. Comparing the confusion matrixes, we find that VGG16 has better recognition ability for cardiomegaly than InceptionV2 and ResNet101, and worse recognition ability for cardiomegaly than InceptionV2 and ResNet101. Compared with the other three classical neural networks, CliqueNet has the best recognition ability for cardiomegaly, but the worst recognition ability for emphysema. Confusion matrix shows that ACRnet has a good and relatively balance recognition ability in identifying three types of chest X-ray images: cardiomegaly, emphysema and normal. The overall performance of ACRnet is better than that of the existing four classical neural networks. On the expanded dataset, the recognition ability of ACRnet is further improved. Figure 6 shows the ROC curves obtained by neural networks using 5-fold cross validation. The results show that the recognition ability of ACRnet for emphysema is better than the existing four classical neural networks, and the recognition ability for cardiomegaly and normal is also at a high level. After data expansion, ACRnet's ability to recognize cardiomegaly and emphysema is significantly improved.

Analyzing furtherly the confusion matrix in Figure 5, for emphysema and cardiomegaly, in (a) the overall misdiagnosis rate is calculated as 15.1%, (b) 21%, (c) 30%, (d) 27.9%, (e) 29.2%, (f) 33.6%. It is obviously that the misdiagnosis rate of ACRnet is lower than the other four classical neural networks. When the expanded dataset is used, the misdiagnosis rate of ACRnet has been declined to 15.1%. With the decrease of the misdiagnosis rate, patients will have gotten more valid and timely treatment. So ACRnet has good practical applied value.

By analyzing ROC curves, it is not difficult to find that the ROC curves obtained by ACRnet are all relatively smooth. Compared with other models, ACRnet obtains the best AUC index for recognizing three types of chest X-ray images. The result further prove that ACRnet has good recognition ability. In addition, the ACRnet's identification performance will be further improved when the expanded dataset is used to train the model.

To comprehensively evaluate the performance of our method for identifying cardiomegaly, emphysema and normal, we compare our method with several most advanced methods. Table 7 shows the AUC indices of our method and the other 4 state-of-the-art convolutional neural networks: CheXNet ^[27], GraphXNet ^[28], TieNet ^[29] and AM_DenseNet ^[30]. We can find that our method has excellent performance in identifying the three types of chest X-ray images.

5.   Conclusions

Cardiothoracic diseases are a serious threat to human health, and chest radiography interpretation is very important for timely detection of cardiothoracic diseases. However, this time-consuming work is usually completed by radiologists, and subjective differences among different people will lead to missed diagnosis and misdiagnosis. In this way, using deep learning technology to build neural network model can reduce clinicians' subjective errors, which is of great significance for disease diagnosis and treatment. In this paper, we construct ACRnet to recognize cardiomegaly, emphysema and normal chest X-ray images, and achieve good recognition effect. Using DCGAN technology to expand the existing dataset also can further improve the accuracy of the model. In future research, we hope that our method can be used to distinguish more types of cardiothoracic diseases, so as to help clinicians diagnose cardiothoracic diseases more accurately.

Acknowledgments

The work was supported by the Natural Science Foundation of Liaoning Province of China (No.20180551011).

Conflict of interest

The authors declare no conflict of interest.