Design and analysis of a robust breast cancer diagnostic system based on multimode MR images

1.   Introduction

Breast cancer is the most prevalent type of cancer among women worldwide and it also threatens men's health ^[1]. More than 1.7 million new cases of breast cancer are detected around the world each year ^[2]. Early detection of breast cancer is an important step to reduce negative effects. The main medical methods for breast cancer detection are X-ray mammography (mammography), Computed Tomography (CT), Doppler ultrasonography, and Magnetic Resonance imaging (MRI). Mammography has a low detection rate for dense breasts ^[3]. The diagnostic radiation of CT is larger than it of ordinary X-ray machines. Ultrasound devices have poor penetration into bone and it is difficult for obese patients to produce effective images with high quality.

MRI is a technique for clinical imaging of the mammary glands. Breast MRI is employed for 1) screening of breast cancer in high-risk subjects, 2) evaluation of tumor extent in specific groups of breast cancer patients (e.g., patients with invasive lobular carcinoma), 3) evaluation of tumor response to chemotherapy treatment, and 4) troubleshooting in case of inconclusive findings from other modalities ^[4]. Breast MRI can be effective at finding unsuspected cancer in the opposite breast ^[5], which can facilitate early treatment of both tumors and reduce the death rate of breast cancer. Thus, we use MRI for the diagnosis of breast cancer in this paper.

Computer-Aided Diagnosis (CAD) software has been developed and used since the last century. Unfortunately, commercial CAD systems have not brought significant improvements in performance ^[6], especially in the last decade. The actual diagnosis is still more dependent on the experience of clinicians. There is always a difference in the diagnostic results of radiologists and CAD. Many radiologists either fail to use CAD as a second reader or only rely on the output of CAD ^[7]. The clinician's observation may influence the diagnostic results to some extent. Our work is to develop a system for better classification of Magnetic Resonance (MR) images and localization of tumors in MR images. As an auxiliary reader, this system can reduce radiologists' image reading time.

In previous studies, most researchers relied heavily on single-sequence images. But in the actual diagnostic process, radiologists and clinical experts observe and analyze multi-sequence images simultaneously to diagnose the patient accurately. Therefore, we use several sets of images with different imaging modes for robust breast image classification. According to the radiologists' suggestions, we chose T1W_TSE, T2W_SPARI, DWI_SSh, and DYN_eTHRIVE+C. They are written as T1W, T2W, DWI, and DYN in the following. We use the characteristics of these four sequence images to determine whether an image contains a tumor. In addition to image classification, radiologists also need to locate tumors to lay the foundation for subsequent diagnosis and treatment. Nevertheless, the poor signal-to-noise-ratio and faint edges caused by partial volume effects increase the difficulty of tumor segmentation ^[8]. Different radiologists may determine different tumor locations following their own experiences. To assist radiologists in locating the tumor edge better, we propose an optimized segmentation method. The classification-segmentation process becomes the basis for radiologists to analyze and interpret MR images and clinicians to implement treatment.

Deep-learning algorithms ^[9,10,11] are capable of directly learning discriminative features from the given data, and convolutional neural networks (CNNs) have been proven to be effective models for tackling visual tasks ^[12,13,14]. Deep-learning has also produced positive results in medical disease analysis ^[15,16]. In the medical image classification and segmentation task, CNNs also have extraordinary performance^[17] and may perform better with the expansion of data sets and the improvement of network structure. Some scholars have focused on the robustness ^[18,19] and the optimization scheme ^[20] of models to obtain superior and steady performance on the Internet of things. In any case, technology is increasingly tied to life. We investigate CNNs and optimization methods to produce a better solution for breast cancer diagnosis.

The main contributions of our work are:

1) We design and analyze a Robust Breast Cancer Diagnostic System (RBCDS) based on multimode MR images.

2) We propose a four-mode convolutional neural network (FMS-PCNN) model to distinguish whether there is a tumor in an image. This model combines four types of MR images comprehensively to improve experimental performance and simulate the classification process of radiologists. The spatial pyramid pooling (SPP) ^[21] and principal components analysis (PCA) ^[22] are used to utilize the original size images and remove redundancy of fused features.

3) We use the improved U-Net architecture for the segmentation task ^[23]. The U-Net achieves excellent performance on biomedical segmentation applications ^[24]. The important modification is applying the sub-pixel convolution method ^[25] in the upsampling process.

The rest sections of this paper are conducted as follows. Section II reviews the related work of breast cancer classification, segmentation based on MRI and CNNs in image processing. Section III describes the system model and relevant definitions. Sections IV and V show the details of classification and segmentation methods used in RBCDS. Section VI exhibits the performance of our proposed model and analyzes the comparisons across different methods. And section VII summarizes the full paper and future work.

2.   Related works

2.1.   Breast cancer classification

The automatic image processing for cancer diagnosis has been explored as a topic of research for more than 40 years, but it is still challenging due to the complexity of the images ^[26]. The state of the art in classification problems ^[27] on vision communities, such as action ^[28], object ^[29,30,31], and scene ^[32,33], have been significantly improved and produced remarkable results these years.

Liu et al. ^[34] used the cell shape and the bare nuclei as the significant features for breast cancer diagnosis and utilized a Bayesian network (BN) modeling approach to support clinicians in making diagnostic decisions. And the BN modeling method was built by the K2 learning algorithm and statistical computation methods. They reported that the combination of all features showed the best performance, with a diagnostic accuracy of 0.934. Their model could help clinicians diagnose diseases according to important characteristics, which could be used in other healthcare data analytics. Sayed et al. ^[35] extracted features from breast MR images and then used K-Nearest Neighbor (KNN) and Linear Discriminant Analysis (LDA) to differentiate between benign and malignant tumors. They stated that benign cases were misclassified with an average specificity reduction percentage of about 3%. They also reported that their method may become a noninvasive diagnostic tool for breast cancer. Drukler et al. ^[36] assessed the performance of radiologic signatures by using a support vector machine (SVM) and 10-fold cross-validation based on a dataset consisting of 64 lesions. They calculated the 95% confidence interval of the area under the receiver operating characteristic (ROC) curve (AUC) to confirm that texture enhancement played a dominant role in the diagnosis of breast cancer. In their report, the radiologic enhancement-texture signature performed best for ultrafast MRI and achieved the AUC of 0.71.

However, in previous work of classification, only a single imaging mode was mainly investigated. Few studies involved in the analysis of multiparametric images ^[7,23] and the dependencies among features in a single experiment. Kyono et al. ^[7] introduced a Man and Machine Mammography Oracle (MAMMO) to support radiologists in their diagnosis using mammography datasets. They stated that multi-task learning enabled CNNs to capture more features related to cancer and their model reduced the number of radiologist reading by 42.8%. The second part of MAMMO could determine which decisions output by classification networks was correct and confident. In addition, most studies adapted the resized image to the network structures, which would obviously affect the learning of image features and experimental results. We apply SPP to classify images with original sizes, avoiding loss caused by resizing them. Most importantly, images produced by four imaging modes are used in our proposed classification model to simulate actual image reading, gather features of different sequences and improve the accuracy of classification.

2.2.   Segmentation based on MRI

Breast tumors behave differently across several imaging modes in quantitative analysis. Assessment of tumor size, texture and morphology strongly relies on accurate segmentation, and it has been shown difficult even for expert radiologists.

Gubern-Mérida et al. ^[4] developed a method to calculate the density and segment the breast and fibro-glandular tissue, using an atlas-based approach and Expectation-Maximum (EM) algorithm. They achieved an average dice similarity coefficient (DSC) value of $0.94\; \pm$ 0.04 for breast segmentation and $0.80\; \pm$ 0.13 for fibro-glandular tissue segmentation. Maicas et al. ^[37] introduced a new fully automated breast tumor segmentation model. The method was based on the globally optimal inference in a continuous space (GOCS) and used a shape prior computed from a semantic segmentation. The salient advantage of their approach was that it did not require precise initialization. They achieved the mean dice coefficient (DC) of 0.77 for the test set in dynamic contrast-enhanced Magnetic Resonance imaging (DCE-MRI). Zhang et al. ^[38] proposed a mask-guided hierarchical learning (MHL) framework by extending the fully convolutional network (FCN), which can decompose the problem into several simpler sub-problems. This framework avoided time-consuming manual annotation for tumor regions. The segmentation was based on both segmentation probability maps and landmarks defined on nipples and achieved the mean DSC of 0.72. The most important part of the segmentation task is the localization of the tumor edge. We pay more attention to the precise location of the tumor edge by optimizing the upsampling process. Furthermore, we use DC as the evaluation coefficient of segmentation method, referring to the above studies.

2.3.   Convolutional neural networks in image processing

CNNs have achieved great success in large-scale image recognition and won the large scale visual recognition challenge (ILSVRC) since 2012 ^[10]. Due to lack of data, however, a large number of parameters and multiple trainable layers in CNNs cannot reasonably be trained.

Hence, Ronneberger et al. ^[24] used data augmentation and training strategy to build the U-Net structure. The network consisted of a contracting path and an expansive path, which gave it the u-shaped architecture. Their network can be trained end-to-end from very few images. They marked a very significant achievement and won the ISBI challenge for the segmentation of neuronal structures in electron microscopic stacks in 2015.

Alom et al. ^[39] proposed a Recurrent Convolutional Neural Network (RCNN) and a Recurrent Residual Convolutional Neural Network (RRCNN). The new models were all based on U-Net, which had great performance on medical image detection tasks. The recurrent residual convolutional layers could improve the feature representation and parameter design for medical image segmentation. Their method performed better on lung lesion segmentation, blood vessel segmentation in retina images, and skin cancer segmentation. The new model provided a testing accuracy of 0.9424 with a higher AUC, which was 0.9419 on skin cancer segmentation.

Zhou et al. ^[40] developed a powerful architecture for medical image segmentation based on a deeply-supervised encoder-decoder network, which utilized skip pathways to reduce the differences between the two symmetrical ways. They achieved better performance on nodule segmentation in the low-dose CT scans of the chest. Isensee et al. ^[41] modified the structure of the U-Net slightly and improved the results by a Dice loss function. They trained the network with the regions of original images and additional training data. This method resulted in the highest Hausdorff Distances (95th percentile) of 6.03 for the whole tumor on the test data.

3.   System model and definitions

We propose an RBCDS based on CNNs and multimode MR images to simulate the real clinical diagnostic process. The complete architecture of RBCDS is shown in Figure 1. The proposed classification model is modified with SPP and PCA. We compare the effects of different dimensionality reduction ratios on the results. Some data augmentation techniques, transfer learning, and fine-tuning methods are also used in this study. This model achieves the highest accuracy rate of 94.6%. For the optimized segmentation method, we compare the performance by evaluating the DC values of experiments. Apart from this, the classification network is used to initialize the segmentation network. Here we achieve an average DC value of 0.867.

As a comparison, the traditional classification method commonly used is shown in Figure 2. This approach is a common process in the current diagnostic process.

In our study, our MRI dataset includes T1W, T2W, DWI and DYN of 90 breast examinations from high-risk patients. The dataset is built at the radiology department of the Tianjin Center Obstetrics and Gynecology Hospital. In this work, 7000 images (2500 images with tumors and other images without tumors) are used for classification, and 2450 images with tumors are used for segmentation. The images are divided into training, verification, and test by 8:1:1. Radiologists use a 1.5 T scanner (PHILIPS MR Systems Achieva, version 3.2) for bilateral MRI.

The specific imaging methods of the four types of images are exhibited as following: T1W_TSE SENSE data are obtained in the transverse plane, scanning sequence is the spin-echo (SE) for 4.8 mm slice thickness, having $512 \times 512$ resolution, and $100 \times 100$ mm field of view (FOV), with 10 ms echo time (TE), and 456.6 ms repetition time (TR). T2W_SPAIR data are obtained in the transverse plane, using the SE for 4.8 mm slice thickness, each slice image has a $352 \times 352$ resolution, and $100 \times 100$ mm FOV, with 90 ms TE, and TR is 4858.8 ms. DWI_SSh SENSE data are obtained in the transverse plane, scanning sequence is SE for 4.8 mm slice thickness, $320 \times 320$ resolution, and $100 \times 100$ mm FOV, TE is 68.4 ms, and TR is 5869.2 ms. DYN_eTHRIVE+C data are obtained in the transverse plane and sagittal plane, using the gradient recalled (GR) for 340 mm slice thickness, $352 \times 352$ resolution, and $100 \times 100$ mm FOV, with 3.4 ms TE, and 6.9 ms TR.

For the segmentation task, images are preprocessed by a semi-automated data labeling system. A code written in Python is used for punctuation and connection to denote the boundaries of tumors in the images. The manual segmentation has been performed by three experienced radiologists, which can avoid the inaccuracy of decisions by a single radiologist. The radiologists used the code that we wrote to open each image containing the tumor and record the points on the edge around the tumor with the mouse. After the marking is completed, the completion button is triggered, and the tumor is displayed in a single image.

4.   Our proposed classification model in breast cancer detection

In this study, images from four imaging modes are used to form the data set. In order to enable the network to process all images smoothly, we use SPP to optimize the network structure. The FMS-PCNN is also optimized by PCA. We make four comparative studies with different networks to verify that the proposed model is effective. The analyses intend to prove whether our proposed model is superior to the decision-level fusion method in every network. We also demonstrate the effectiveness of transfer learning ^[42] in our study through experiments. The algorithm of the classification task in breast cancer detection is explained in Algorithm 1. The features that entered into the SPP layer are extracted by partial CNN in Figure 1. SPP plays a role in organizing features and learning them. Through convolution and pooling, our model can grasp important and detailed features of images. The extracted features are the basis of the following steps.

4.1.   Preprocessing

Image quality directly affects the design and accuracy of the classification algorithm. Therefore, in addition to method optimization, preprocessing technology also plays an important role in the whole project.

We apply two standard preprocessing steps to the images. The entire image contains invalid information that interferes with the observed objects. We segment effective area related to the tumor from each image and then normalize the image by global contrast normalization (GCN), which is also named zero-centering. In general, normalization can eliminate or weaken the influence of some factors or transformation on some properties of the image. We remove the lung areas, leaving the breast tissue, the subphrenic region and the wall of the thoracic cavity that often adhered to the tumor. GCN attempts to deal with changes and differences brought by external conditions like illumination levels. We compute the mean of intensities for each image and subtract it from each pixel of the image. The new tensor of the normalized image is defined as

where $M_{i, j}$ denotes the tensor of an image and $\bar{M}$ stands for the mean of the intensities of $M_{i, j}$. The image $M$ satisfies $M \in \mathbb{R}^{r\times r}$. $\bar{M}$ is defined as:

4.2.   Data augmentation

Data augmentation in computer vision is the artificial introduction of prior knowledge for visual invariance. Data augmentation is one of the commonly used skills in deep learning. It is mainly used to increase the amount of training data and make the dataset as diverse as possible so that the training model will have stronger generalization ability.

We augment the dataset by rotating the images 90,180 and 270 degrees, and then applying random horizontal flip. Multiple patches are selected in the images. For baseline networks without SPP and PCA, the images are resized to the appropriate input sizes for different networks. Four types of images are grouped together according to their imaging sequence and position. Finally, images are labeled with $0, 1$, as the dichotomous variable. The label $0$ means that there is no tumor in the image, the label $1$ stands for the opposite.

4.3.   Experimental details

In the process of image classification, we use five CNNs. Different experimental methods are employed in each network to examine the superiority of FMS-PCNN. In this way, we can avoid experimental specificity due to the particularity of specific structures.

4.3.1.   Decision-level fusion

As shown in Figure 2, features of each single image are extracted through CNN. Next, they are forwarded to the softmax layer for two-category classification, and then the classification results of four images are summarized. If more than half of the images suggest the tumor in one group, the corresponding subject is considered to have a tumor, otherwise, there is no tumor. This method does not introduce the deep correlation of different imaging modes. Meanwhile, because of the complexity and diversity of the mammary glands, different effects appear across various types of images, leading to confusion in the observation of tumors.

The differences between four types of images in the same position can be seen in Figure 3. The images from left to right in the first row are T1W and T2W, in the second row are DWI and DYN. The tumors marked with yellow circles in DWI and DYN are distinguished from surrounding tissue, but they are not clear in T1W and T2W.Both images are from the same location on the same subject's breast. In DWI, the breast tumor typically appears more intense to surrounding tissues ^[43]. This observation suggests that the radiologists may not be able to identify the tumor with a single-sequence image. Thus, we propose the detection model with connection of four sequential images. Among the numerous MR imaging sequences, we choose these four sequences for the following reasons: 1) a single image can not determine the diagnostic results; 2) radiologists often use four imaging modes in Figure 3 to analyze whether a subject has breast cancer. Combining multiple images for diagnosis can simulate the real diagnostic process more reliably.

4.3.2.   SPP-networks

The popular networks are ended with pooling layers and fully connected layers followed by a $C$-way softmax as output, where $C$ is the number of classes. They require a fixed size of an input image. Cropping and wrapping become the main methods to produce images of arbitrary sizes. Changing sizes of the input images may cause image losses and affect the performance of classification models. SPP layer ^[21] in the CNNs allows networks to take images of any size as input. It can generate fixed-length representations of features and maintain spatial information by pooling in local spatial bins. The number of bins is fixed regardless of different image sizes. Max pooling is used in SPP.

We replace the last pooling layer with an SPP layer. An SPP layer has an $x$-level pyramid pooling layer. Figure 4 shows a network with a 3-level SPP layer, which is applied in our model. Let $k$ denote the number of filters in the last layer before the SPP layer and $N$ denote the total number of bins. The outputs of the SPP layer are $k\times N$-dimensional features. If the feature map of the last layer before the SPP layer is $m\times m$ and a pyramid level of all SPP layers is $n\times n$. The size of each bin is computed by

and the corresponding stride is computed by

For example, when we use a 3-level SPP layer and the feature map of the last layer is $7\times7$, ${ pool }_{4 \times 4}$ will have bins with size of $2\times2$ and stride of 1.

4.3.3.   Feature fusion

Images produced by a single imaging method cannot be used as the only basis for diagnosing a patient. For the classification task, we combine the features of the four types of images to achieve better classification performance. As shown in Figure 1, in the classification process, after images of four sequences pass through the neural networks, the learned features are fused to form more representative features. This step can deeply take advantage of all the features from the parameters of four sequences and overcome the contingency of a single imaging mode. The $n$-th feature of the images output by previous layers is defined as $t_{n}$. We fuse the other features with $t_{n}$ and compute the new $n$-th feature of images by

where $t_{c}$ is the feature of the $c$-th image in an imaging mode, $K_{S}$ can transform the feature to a lower dimension, the maximum value of $x$ is 4. The weight of linear sum is $\varphi^{xy}$, denoting the impact from other images, which can be computed as

The weight $\varphi_{Q}^{xy}$ is computed as dot product and can be calculated as

where $K_{U}$, $K_{V}$ are matrices with similar function like $K_{S}$. And $d_{k}$ is the dimension of the transformed feature. For the $n$-th feature, there are four transformed features, $t_{n}^{'(1)}$, $t_{n}^{'(2)}$, $t_{n}^{'(3)}$ and $t_{n}^{'(4)}$. Finally, all the $t_{n}^{'}$ are concatenated together and the $n$-th feature $t_{n}^{''}$ is rephrased as

After transformation, four new features $t_{1}^{''}$, $t_{2}^{''}$, $t_{3}^{''}$ and $t_{4}^{''}$ can be concatenated as a new feature ^[44]. The features corresponding to the four imaging parameters are learned by networks. We first combine features that belong to different imaging modes to produce an output. The different outputs, $t_{n}^{'(1)}$, $t_{n}^{'(2)}$, $t_{n}^{'(3)}$ and $t_{n}^{'(4)}$ are then combined with the original feature $t_{n}$. Lastly, all the calculated features of four modalities are concatenated as the final feature for image classification. On the basis of combining multiple sequence parameters, the classification model can be more reliable.

4.3.4.   Dimensionality reduction by PCA

Differences between parameters of multiple sequences can be combined by feature fusion. But the similarities among multi-parameter modes may lead to repetitive information. In other words, fused features may contain redundant information, which could weaken the model's performance in the experiments. PCA ^[45,22] can remove the information redundancy. Suppose that the feature needs to be reduced from $a$-dimension to $b$-dimension: PCA first finds $b \; a$-dimensional vectors, and then projects the features into $b$-dimensional space. It finally ensures that the projection error is small enough. Let $F = \left(f^{(1)}, f^{(2)}, \ldots, f^{(z)}\right)$ denote all the original components. We can compute new feature $g^{(i)}$ by

where ${f^{(i)}}$ denotes the $i$-th vector and $U_{\rm {reduce}}$ stands for a matrix of retained singular vectors. They can be calculated by

In the above formulas, ${\mu^{(i)}}$ is the mean of feature $i$ and ${s^{(i)}}$ is the standard deviation of feature $i$. And $u^{(1)}, u^{(2)}, \cdots, u^{(b)}$ are the retained singular vectors. After the feature is normalized, the covariance matrix of it can be expressed as

Finally, we define the feature statistics rate (FSR) as the ratio of the number of principal components retained by PCA to all the original components. According to the proportion, we can obtain new features with less redundant information for classification. We first select FSR with values of 50, 70 and 90% in the experiments. Setting FSR to 90% means that the top 90% of principal components are selected. We secondly set FSR to 80% according to the highest accuracy of the above experiments. Preserving the most representative features can reduce the computational complexity and improve classification accuracy.

In the training, we select ResNet-50 ^[29], AlexNet ^[12], VGG 16 ^[31], Inception V3 ^[46] and DenseNet ^[47] as basic networks. All the networks are trained using the stochastic gradient descent (SGD) method. We use a batch size of 32 for 100 epochs. The weight decay is set to 1e-5 and a Nesterov momentum ^[25] is set to 0.9. The initial learning rate is set to 0.01 and is divided by 10 times at 50 and 75% of the total number of training epochs. We evaluate the models by classification accuracy and the AUC of the best model.

5.   The segmentation model in RBCDS

For the segmentation task in our RBCDS, we introduce the SU-Net. We use the sub-pixel convolution in the upsampling process of U-Net ^[24] and the classification network to initialize the SU-Net. Based on FCN, U-Net is modified and extended to process fewer training images and produce more accurate segmentation. It is a completely symmetrical architecture that only uses the valid part of each convolution. The skip connection promotes the gradient flow of the network. The algorithm of segmentation method ^[23] used in RBCDS is shown in Algorithm 2.

5.1.   Data preprocessing and augmentation

The images, including major areas related to the tumors, are normalized by GCN. Normalization can avoid image differences caused by light conditions and subtle human operation differences. We augment the data by rotating the images 90,180 and 270 degrees. The random horizontal flip is also adopted for the segmentation task. We resize all the images to $256 \times 256$ as input to the segmentation network.

5.2.   Experimental methods

The U-Net ^[24] model is improved for the segmentation task. Since the upsampling process of U-Net only uses a simple method called the nearest neighbor method to copy the neighborhood points. This method for upsampling is too rough, especially for the segmentation of pixel-level tasks. A rough upsampling method may lead to less clear edges of the segmentation results.

For the above reasons, we specifically use the sub-pixel convolution method to optimize the upsampling process in U-Net. The sub-pixel convolution is a standard convolution that can learn an array of upscaling filters to upscale the low resolution (LR) feature maps into the high resolution (HR) output ^[25]. In image processing for medical imaging, the recovery of an HR image from its LR counterpart refers to as super-resolution (SR) ^[25,48]. The sub-pixel convolution method can contribute to better localization of edges through image upsampling and it can remove the blocking artifact in image SR task. The trained model in the classification is used to initialize the left half of SU-Net (namely the encoder process).

All the networks are trained using the SGD method. The batch size of 16 for 80 epochs is employed. The network uses a non-linear transformation (ReLU). The initial learning rate is set to 0.001, divided by 10 times at 40 and 60 epochs. The weight decay is set to 1e-5 and Nesterov momentum ^[25] is set to 0.9, which is the same as the classification model.

We evaluate the performance of the SU-Net by dice coefficient, which can reflect the similarity between ground truth and segmentation result of the model. The value of DC can be computed by

where $X_{\mathrm{gt}}$ and $X_{\mathrm{ans}}$ represent the ground truth and auto-segmentation of a given image.

6.   Performance analysis

6.1.   Classification results

This section describes the results of the FMS-PCNN model with five CNNs. The start-up training with the pre-trained model on ImageNet is also compared. The benefits of using multimode MR images for training are analyzed based on the experimental results. Using transfer learning and fine-tuning strategies can improve the classification accuracy.

Figure 5 shows the results using the decision-level fusion method. Figure 6 reveals better performance based on SPP, PCA, and fused features. Using three training strategies (transfer learning, fine-tuning and data augmentation) simultaneously improves the results. The data can strongly demonstrate that using SPP and PCA in the networks produces higher accuracy. This is due to the use of original-sized images and the reduction of superfluous information in the fused features. The accuracy achieves 92.6% without PCA. In the case of multiple images, each method uses four paths. The model does not use multiple fully connected layers containing a large number of parameters, but directly maps to the fully connected layers activated by classification at the end, which compensates for the parameter increase caused by fusion. Our best result of this step is: FMS-PCNN reaches the accuracy of 92.9% with FSR set to 50%.

Table 1 reveals the comparisons of classification accuracy values across different FSR values. Encouragingly, the accuracy improves to 94.6% when FSR is set to 80%, indicating redundancy between the raw fused features. However, when we further increase the FSR to 90%, the accuracy drops to 92.8%. And higher FSR results in greater computational complexity. The retained top 90% of components of all components still have invalid information for classification. If fewer features are retained, the classification accuracy will be lower because of the incompleteness of features. In brief, we should find the most appropriate proportion to achieve the best classification performance, instead of reducing the capacity of feature too much.

Figure 7 shows that AUC of the best classification model achieves 0.928 with an accuracy of 0.946. In addition, we compare our results with other methods, as shown in Table 2. Multi-parameter images are the most important foundation for outstanding results. The other methods use dynamic contrast-enhanced MRI (DCE-MRI) and mammogram images. MR images are clearer than other images and can provide more details of the soft tissue. The feature fusion of multi-sequence images can greatly improve the classification accuracy and the AUC is also improved. At the same time, the use of multi-sequence images is more consistent with the real diagnostic process.

6.2.   Segmentation results

Figure 8 exhibits the DC values of using the basic U-Net, only using initialization, and using both initialization and SU-Net. The best mean DC value of segmentation achieves 0.867 by utilizing initialization and the proposed SU-Net model concurrently. Segmentation results can help clinicians further observe the lesions and formulate follow-up treatment plans.

To verify the effectiveness of the proposed approach, Table 3 compares the performance obtained by some literature proposals. The 2D Fuzzy C-Means (2D-FCM) ^[52], Fully Convolutional Networks (FCN) ^[53], and SegNet ^[54] are mainly compared. Our segmentation model obviously achieves better results.

Finally, the accurate segmentation process can be integrated with the classification model to form a system for the diagnosis of breast cancer. The multimode fusion method in the classification task prevents the confusion between observation and breast tumors and simulates the clinical diagnosis process. The improved segmentation network provides more detailed tumor edge information. These results can help radiologists and clinicians make inspection reports and determine the therapeutic plans.

7.   Conclusions and future works

We have proposed an RBCDS based on multimode MR images in this paper. Firstly we develop the FMS-PCNN model for image classification in the breast cancer diagnostic system. This model fuses MRI-based features to combine multiple parameters of four imaging modes. The model also avoids resizing images and retains the most representative features. Secondly, we optimize the upsampling process of U-Net and initialize SU-Net by classification network. The performance of the proposed model surpasses the traditional U-Net model. In the future, the segmentation results can be correlated with the blood flow information and density characteristics of the tumor area for diagnosis. Finally, the proposed classification and segmentation models form a complete robust diagnostic system, which can solve the confusing problem that tumors are difficult to distinguish. Instead of a single image, the system can diagnose four images, making the results more robust. This system can be used as an auxiliary system in the actual diagnostic process to provide information for clinicians.

In the future, more targeted and complex feature fusion methods and sequences can be used for breast cancer detection and achieve excellent performance. With the development and innovation of imaging technology, the researchable images and imaging modes available will be more comprehensive. Additional parameters of other imaging sequences of MRI will be helpful in improving breast cancer detection methods and systems. Further research on the extent of disease and the presence of multifocal cancer in patients with invasive carcinoma and ductal carcinoma in situ (DCIS) will rely heavily on MR images.

Acknowledgments

This work was supported by National Natural Science Foundation of China (No.61876131, U1936102) and the National Key R & D Program of China (No.2018YFC0806802).

Conflict of interest

The authors declared that they have no conflicts of interest to this work.