TCN-Attention-BIGRU: Building energy modelling based on attention mechanisms and temporal convolutional networks

1.   Introduction

Placenta accrete spectrum (PAS) disorder is defined as the abnormal invasion of trophoblast cells into the myometrium at different depths of infiltration. It occurs mainly in patients with placenta previa or previous cesarean section ^[1]. Common complications of PAS include catastrophic perinatal hemorrhage and injury of bladder, rectal and urethral ^[2]. There are many high-risk factors for PAS, including the history of cesarean section, placenta previa, multiple miscarriages and curettage, history of other uterine-related surgeries such as myomectomy, and advanced maternal age ^[3,4]. With the increase of various risk factors such as cesarean section and abortion, the incidence of PAS is also increasing year by year ^[5,6,7,8]. China is a country with a relatively high cesarean section rate ^[9]. With the implementation of the three-child policy, the number of late marriages and childbearing has increased. It can be speculated that the incidence of PAS in China will further rise. Therefore, the prenatal prediction of PAS is of important practical significance.

The traditional detection methods of PAS based on MRI generally include three consecutive steps: region of interest segmentation, image feature extraction and PAS detection ^[10]. A recent study showed that experienced radiologists performed significantly better than junior radiologists (90.9% of sensitivity and 75% of specificity for senior attending physicians, 81.8% of sensitivity and 61.8% of specificity for primary attending physicians) ^[11]. In order to reduce the reliance on the clinical experience of doctors and improve the diagnostic level of PAS, some scoring systems for the diagnosis of placental invasion have been proposed in recent years, which have not been widely tested ^[12]. Some scholars ^[13,14] used machine learning methods to detect PAS based on radiomics features and clinical factors (such as whether it was a scarred uterus, whether there was a history of cesarean section, and whether there was a history of miscarriage, etc.). There are also a few studies ^[15] using deep neural networks to learn powerful visual representations in MR images to predict PAS, but detection of PAS based on MRI is still a challenging task, there are several problems:

(1) Radiomics features are explicitly designed or handcrafted. Although the number of features can reach tens of thousands, these features are shallow and low-level image features. The heterogeneity within the placenta and the relationship between the placenta and adjacent tissues may not be fully characterized, thus limiting the predictive potential of the model.

(2) In clinical practice, doctors usually diagnose PAS based on the placental signal reflected by T2WI MR images supplemented by the bleeding conditions reflected by T1WI MR images. Traditional MR image analysis methods are difficult to extract multiple sequences of MR image features at the same time and assign corresponding weights to predict PAS according to their significance. Figure 1 shows T2WI and T1WI MRI slices of a patient with PAS at the same location.

Among them, Figure 1(a) is a sagittal view of T2WI, which shows PAS in the posterior and lower part of placenta (white arrow), and a strip of low signal on T2WI (red arrow) is seen in the placenta; Figure 1(b) is a sagittal view of T1WI, and a mass of hyperintense hemorrhage is seen in the placenta (white arrow).

In order to solve the above problems, we proposed a dual-path neural network fused with a multi-head attention module, which includes a dual-path neural network and a multi-head attention module. Dual-path neural network can extract T2WI and T1WI MR image features. Specifically, the presence of low-intensity bands or small patches in the placenta on T2WI sequences may indicate PAS; Intraplacental hemorrhage is usually seen in the placenta of T1WI MR images with patchy slightly high signal intensity, which may also suggest PAS. The multi-head attention module assigns different weights to the features of different sequences through learning and fuses them to better measure the importance of different sequence features.

The main contributions of this paper are: (1) A dual-path neural network is designed to extract the features of T2WI and T1WI sequences of MR images; (2) A multi-head attention module is proposed to learn the weights of different features to generate final features with stronger discrimination. Experimental results on an independent validation dataset show that the detection accuracy achieved by our method is superior to the methods using only a single sequence of MR images. The comparative experimental results also show the effectiveness of the multi-head attention module proposed in this paper.

2.   Related work

In this section, we discuss the work most related to our work: detection of PAS based on MRI. MRI is less affected by intestinal gas and bones, has high tissue resolution and can be imaged at any angle in multiple directions, so it is especially recommended for cases with unclear posterior placenta and ultrasound results and/or high clinical suspicion ^[16,17,18]. In practice, MR images can provide important information for doctors to predict and diagnose the type of PAS. Recently, there have been many studies with promising performance. These algorithms typically use hand-designed or measured features to detect PAS. For example, Zheng et al. ^[19] used the observed imaging features on MR images to diagnose PAS, such as whether it is placenta previa, whether the placenta is thickened, etc. However, manually designing and measuring features is time-consuming and labor-intensive. To overcome this difficulty, several radiomics or deep learning-based methods have been proposed to automatically extract MRI features. For example, Romeo et al. ^[13] used a machine learning algorithm to predict the types of PAS based on radiomics features. Li et al. ^[15] used an auto-encoding network to extract features of MR images to predict the types of PAS. However, most of the current studies are based on a single sequence of MRI for the detection of PAS. In clinical practice, doctors usually diagnose PAS based on the placental signal reflected by T2WI MR images supplemented by the bleeding conditions reflected by T1WI MR images. Therefore, we propose a dual-path neural network fused with a multi-head attention module to detect PAS. The model first uses a dual-path neural network to extract T2WI and T1WI MR image features separately, and then combines these features. The multi-head attention module learns multiple different attention weights to focus on different aspects of the placental image to generate discriminative final features.

3.   Materials and method

3.1.   Data set

This retrospective study was approved by the Ethics Committee of The Affiliated Hospital of Medical College of Ningbo University, and all patients' identities were de-identified to protect patient privacy. The MR images were collected from The Affiliated Hospital of Medical College of Ningbo University and Ningbo Women & Children's Hospital from January 2018 to May 2021.

All MRI examinations were performed by radiologists with more than 5-year of work experience using 1.5 Tesla units to perform 8 or 16-channel array sensitivity-coded abdominal coil scans. The imaging equipment of The Affliated Hospital of Medical College of Ningbo University is Ge signa twinspeed 1.5T superconducting dual gradient magnetic resonance scanner with 8-channel body phased array coil. The imaging equipment of Ningbo Women & Children's Hospital is Philips Achieva Noval Dual 1.5T superconducting dual gradient magnetic resonance scanner, using a 16-channel body phased array coil. In this study, we chose the supine sagittal image of the conventional T2WI and T1WI sequence (side-lying imaging is prone to curling artifacts due to the bulge of the abdomen) as the experimental sequence.

Inclusion criteria are as follows: (1) Patients who underwent T2WI and T1WI MRI after 30 weeks of gestation; (2) Those with clear placenta invasion or pathological records after cesarean section; (3) Image quality is good. The exclusion criteria are as follows: (1) Patients without T2WI or T1WI MRI data; (2) Patients with mismatched number of T2WI and T1WI MRI slices; (3) Patients without clinical or surgical pathological confirmation; (4) Patients with severe image artifact.

Based on the above criteria, we collected a total of 321 cases, including 142 normal cases and 179 cases of PAS (including accrete, increta and percreta). The degree of invasion in all patients was determined based on surgical findings, intraoperative diagnosis and pathological examinations. Table 1 is the distribution table of our dataset.

3.2.   Dual-path neural network

In order to extract the features of dual-sequence MR images, we designed a dual-path neural network to extract T2WI and T1WI MR image features. The dual-path neural network consists of two independent backbone networks, and finally combines the features extracted from the two backbone networks. Taking ResNet-50 as the backbone network as an example ^[20], the structure of the dual-path neural network is shown in Figure 2.

As shown in Figure 2, the dual-path neural network consists of two independent backbone networks. The backbone network ResNet-50 includes 5 convolution blocks. The last layer of each convolution block outputs a feature map of a specific scale as the input of next convolution block, and the size of the feature map output by the previous convolution block is 1/2 of the size of the feature map output by the next convolution block. The final output of each backbone network is a 128-dimensional feature vector, and the outputs of the two backbone networks are spliced as the extracted combined features.

3.3.   Multi-head attention module

We obtained the combined features that fused the two sequences of MR images by the dual-path neural network. To assign different weights to different features and perform further fusion, we proposed a multi-head attention module. The output of the attention module usually focuses on a specific part of the image, such as: intraplacental heterogeneity information in T2WI MR images or hemorrhage information in T1WI MR images. A single attention unit can usually only reflect one aspect of an image. However, it may be more effective to have multiple attention units focus on different features in different sequences of images and to describe the images of the entire case together as the MRI data contains multiple sequences. Therefore, to be able to represent multiple aspects of an image, multiple attention units are needed to focus on different aspects of the image. Multiple attention units focus on the same input, but the parameters between multiple units are independent of each other.

Based on the above ideas, this paper proposed a multi-head attention module, which learns multiple sets of parameters to focus on different aspects of the final feature as shown in Figure 3.

Specifically, multiple sets of paired scalars are learned. The first scalar scales the global features linearly, and the second scalar acts as a bias to introduce nonlinear factors. The above process can be expressed as:

where ${\omega }_{i}$ and ${b}_{i}$ represent the scalar parameter pair learned by the i^th attention unit. After obtaining the feature $\mathit{\boldsymbol{V}}_{i}$ output by the attention unit, $L2$ regularization is performed on it, and the features obtained by all units are spliced together, and after nonlinear transformation, it is used as the output of the multi-head attention module. The process can be expressed as:

where $\mathit{\boldsymbol{V}}$ represents the final feature, $\oplus$ represents the splicing operation, and the nonlinear transformation uses the $Relu$ activation function. In subsequent experimental sections, we will compare the detection accuracy of attention module networks with different head counts.

4.   Experiments and results

4.1.   Data preprocessing

The dataset we collected included 142 normal cases and 179 cases of PAS (31 accrete, 136 increta and 12 percreta). The number of T2WI and T1WI sequence slices in each case is equal, including 24–48 slices. Both sequences scan the same part of the patient, so the slices of the two sequences can be in one-to-one correspondence. We excluded 5 slices from the head and tail of the two sequences in each case because there was no uterine area in these slices. To expand the dataset, we treat each T2WI image and its corresponding T1WI image as a slice group ^[21,22].

Background information occupies a large proportion in MR images and has a large impact on subsequent feature extraction and classification ^[23,24]. We crop the center region of the image, and the cropped size is 256 × 256. We randomly split the dataset into a training set and an independent validation set in a 4:1 ratio. Table 2 is the data set split table.

4.2.   Experimental settings

All models are implemented in PyTorch. Batch normalization ^[25] is used for all models. All networks are trained using one RTX 2080Ti GPU with 50 training epochs for the dual-path neural network and the multi-head attention module. We used the Adam Optimizer with a small learning rate ${10}^{-4}$. In addition, the two backbone networks in the dual-path neural network are separately trained on T2WI and T1WI sequence image data respectively. Based on the trained model, the features are extracted and spliced to obtain the combined features which are used for the training of the multi-head attention module.

Evaluate the performance of classification methods using a confusion matrix. The true positive (TP), true negative (TN), false positive (FP) and false negative (FN) values are obtained from the confusion matrix to calculate four performance evaluation metrics as follows:

Among them, accuracy represents the percentage of samples that exactly match the real situation; precision is the proportion of true positives in samples predicted as positive examples; Recall is the proportion of true positives in true positive examples; F1-score is the harmonic mean of precision and recall. In order to avoid dividing by 0 in the calculation process, when TP is 0, the F1 score is recorded as 0.

4.3.   Evaluation of different features

In order to prove that the features of dual-sequence MR images are effective for the detection of PAS, we use three machine learning methods to compare the detection accuracy of the three features on an independent validation set. The three machine learning methods are: decision tree (DT), random forest (RF), and support vector machine (SVM); the three features are: features extracted from T2WI MR images, features extracted from T1WI MR images, and combined features obtained by splicing the above two features. Table 3 shows the comparison results.

It can be seen in Table 3 that the method using SVM achieves the highest accuracy and F1 score on combined features, which are 0.869 and 0.882 respectively, which are higher than the accuracy of the same method on T2WI features and T1WI features. Based on DT and RF, the accuracy and F1 score of T2WI features and combined features are almost equal, and the accuracy and F1 score of T1WI features are lower than those of the above two features. In summary, simply splicing the features of the two sequences can improve the detection accuracy of PAS, but the effect is not obvious. Based on the SVM method, we used T2WI features and combined features to draw receiver operating characteristic (ROC) curves and performed the significance test (DeLong's Test) of the area under the ROC curves (AUC). The experimental results are shown in Table 4.

As can be seen from Table 4, the P-value is less than 0.0001, which indicates that there is a significant difference between the AUCs of the two kinds of features. To further improve model performance, we design a multi-head attention module to pay attention to different features in different sequences of MR images. The multi-head attention module can assign their respective weights according to their importance to improve the detection accuracy.

4.4.   Comparison of different head attention modules

In order to compare the detection accuracy of attention modules with different numbers of heads in the detection of PAS, we calculated the detection accuracy and F1 score of attention modules with different numbers of heads on the independent validation set. Table 5 shows the comparison results. The case where the number of attention heads is 0 means that the attention mechanism is not used. It only uses one Relu layer and fully connected layers (the output dimension of the fully connected layer is 2, which means normal or invasion respectively) to detect PAS.

As can be seen from Table 5, the detection accuracy and F1 score of using the attention module are improved compared with those without the attention module. The highest accuracy and F1 score were achieved with 8 attention heads, 0.886 and 0.899 respectively. When the number of heads is less than 8, the fitting ability of the model is insufficient due to insufficient parameters. When the number of heads is greater than 8, because the model has too many parameters and the size of the training set is not large, the model is over-fitted, and the accuracy rate decreases to a certain extent. Therefore, an attention module with 8 heads is used in the experiment. To further compare the model performance with and without the attention module, we plot the ROC curves. Figure 4 shows the ROC curves with and without the 8-head attention module.

As shown in Figure 4, the performance of the model with the 8-head attention module is significantly higher than that of the model without the attention module. To compare the performance of the two models more intuitively, we calculated the AUC. The results are shown in Table 6.

In order to objectively evaluate the performance of the model and eliminate the impact of data leakage on the experimental results, we select 40 samples that do not appear in the training set from the validation set to form a new test set (20 for normal, 20 for PAS), and verify the performance of the proposed model with 8-head attention module from the accuracy and F1 score. The experimental results show that the model can also achieve ideal results on the test set, reaching an accuracy of 0.825 and an F1 score of 0.837.

5.   Conclusions

This paper proposed a method to detect PAS by extracting and fusing dual sequence placental MR image features using a dual-path neural network. The proposed model mainly includes a dual-path neural network and a multi-head attention module. The dual-path neural network is used to extract the features of the two sequences of MR images and perform feature fusion; the multi-head attention module learns the corresponding weights for different features in different sequences to generate more discriminative final features. Experimental results on an independent validation set demonstrate the effectiveness of each module in our method, with clear advantages over methods that only use a single sequence of MR images. This method may assist physicians in clinical diagnosis, help physicians make perinatal planning and improve maternal outcomes.

Acknowledgments

This work was supported by the Key Talents of Ningbo City Health Technology under Grant 2020SWSQNGG-06 and Zhejiang Province Medicine and Health Project under Grant 2022KY1149.

Conflict of interest

The authors declare that there are no conflicts of interest.