Modified UNet++ with atrous spatial pyramid pooling for blood cell image segmentation

1.   Introduction

With the combination of clinical medicine and information science, medical image processing technology has gradually developed and been applied in various medical laboratories, which can reduce the workload of doctors and improve detection accuracy. In traditional medical examination, professional doctors mainly rely on the eyes to observe the diseased part or tissue of the patient, and judge the disease condition according to their own experience. This method is easily affected by the professional knowledge of doctors, and different doctors may draw different inspection conclusions for the same patient, so it lacks objectivity. When examining specimens containing cells, such as pathological sections, blood smears, etc., it will be a time-consuming task to examine or count them due to a large number of samples. It can provide an objective and reliable reference for doctors to judge diseases by automatically segmenting and discriminating the lesion location images through medical imaging technology. Therefore, it is of great practical significance and social value to carry out automatic blood smear cell image segmentation based on deep learning theory and has broad application prospects in the medical field.

At present, the convolutional neural network (CNN) ^[1,2,3,4,5] has been widely used in the field of medical image segmentation. The method uses a large amount of medical image data to train the network so that the desired features in the images can be extracted. The most popular deep learning network for nuclear segmentation is the U-Net model ^[6], which combines shallow, low-level, fine-grained features from the encoder, and deep, semantic, coarse-grained features from the decoder via long skip connections to effectively improve the accuracy of nuclear detection. With the deepening of the research, various extended frameworks have emerged ^{[7,8,9,10,11,12]}. Considering the complex background of white blood cell images and the appearance changes of tissues, Lu et al. ^[13] proposed a deep learning segmentation framework based on UNet++ ^[14] and ResNet ^[15]. Firstly, a contextual feature sensing module with residual function is designed to extract multi-scale information. Then, dense convolution blocks are introduced to obtain more features on multi-scale channels. Using U-Net as the backbone architecture, Chan et al. ^[16] proposed a deep architecture for image segmentation with better performance, showing strong competitiveness. Jumutc et al. ^[17] proposed an enhanced version of the U-Net approach, which introduces the concept of a single receptive field path to obtain better coarse, fine-grained information. To overcome the interference of light conditions and other external factors for the cell nucleus, Thi Le et al. ^[18] adopted fuzzy pooling operation to maintain the salient features of the image, thus overcoming the problem of noise generation. By choosing image enhancement algorithms such as Fourier transform and mean-shift clustering, Makem et al. ^[19] proposed a deep-based network framework for white blood cells, and the experimental results proved the effectiveness and robustness of this method.

Although various algorithms have been proposed, blood cell segmentation in pathological images still faces great challenges for several reasons: 1) The complex structure, diverse shape, irregular boundary and other factors cause high variability of blood cell image. 2) Accurate segmentation of blood cells from microscope images requires expert knowledge and can be labor-intensive, resulting in limited cell image databases for specific diseases. 3) The sample data of abnormal cells are relatively small, resulting in the imbalance of distribution among classes, which affects the segmentation effect of the deep learning model on rare data. Figure 1 shows different types of cell images from the BCISC dataset. It can be seen that the shapes, sizes and boundaries of cells in different images are different, which may lead to difficulties in nucleus segmentation. Therefore, the existing methods can only deal with a specific image, and there is no general method that can automatically and effectively segment the cell images of all modes.

Inspired by the UNet++ model, we apply the end-to-end deep learning method to blood cell image segmentation. By adding the ASSP module to UNet++ framework, the cell segmentation accuracy is improved without significantly increasing the number of parameters. Then the model structure of the jump connection is redesigned to ensure a stronger learning ability and feature extraction effect. On this basis, MSOF strategy is used to fuse the feature information from different semantic levels to further improve the accuracy of detection and recognition. Finally, the Dice loss function is utilized for training the network.

The remainder of this paper is divided as: Section 2 reviews related work and Section 3 gives the proposed method. The results and discussion of our algorithm on the BCISC dataset are illustrated in Section 4. Finally, the conclusion is discussed in Section 5.

2.   Related works

2.1.   U-Net structure

U-Net ^[6] network is mainly designed to solve the problems of a small amount of data, unclear boundaries and large gray ranges in medical image analysis. Its structure is symmetric and similar to a U-shape, as shown in Figure 2. In the left part, 5 convolutional layers and 4 pooled down-sampling layers are used to decompose the image into features of different levels, which can capture the pixel information of the context. The right side is basically symmetric with the left side, and 5 convolution layers and 4 up-sampling layers are used to restore the input size of the image, which has the function of extracting the shallow, low-level and fine-grained features of the image. Another characteristic of U-Net is a skip connection, which completes the channel connection between the deep, semantic and coarse-grained feature maps from the decoder sub-network and the shallow, low-level and fine-grained feature maps from the encoder sub-network, so as to reduce the information loss in the process of feature extraction and achieve the purpose of accurate positioning.

The simple structure of U-Net is just suitable for the task of medical image segmentation, and it has an excellent performance in cell image segmentation. However, due to the fixed structure, it has two limitations: 1) The deeper the network structure, the larger the computational load, but the results could not be necessarily better. 2) Different data sets determine the different optimal depths of the network.

2.2.   UNet++ structure

Zhou et al. proposed a new neural network structure for semantic and instance segmentation, named UNet++ ^[14]. On the basis of maintaining the U-Net network structure, UNet++ improves the feature connection method between the encoder and the decoder, and its structure is shown in Figure 3. Take the node ${X}^{\mathrm{1, 2}}$ as an example, it receives all previous convolution units of the same level (${X}^{\mathrm{1, 0}}$ and ${X}^{\mathrm{1, 1}}$) and the corresponding up-sampled output of ${X}^{\mathrm{2, 1}}$, and then gets it through a convolution and nonlinear correction unit. In this way, the semantic level of the encoder feature map is closer to the corresponding decoder part, which is more suitable for optimizer optimization. Assuming ${x}^{i, j}$ is the output of node ${X}^{i, j}$, the calculation of ${x}^{i, j}$ can be computed:

where $i$ represents the ith down-sampling layer in the encoder; $j$ represents the jth convolution layer in skip connection, $H(\cdot)$ is the convolution operation with activation function, $u(\cdot)$ and $[\cdot]$ indicate the up-sample and concatenation operations, respectively.

The most obvious difference between UNet++ and U-Net is the new skip connection, which integrates different levels of semantic features of network output in network decoding. This method is very flexible, which can well remove unnecessary features and speed up the training. At the same time, UNet++ also introduces more parameters, and the memory occupied is greatly increased, which slows down the training speed and the convergence speed of the loss function.

3.   Proposed methods

The improvement of UNet++ network mainly focuses on two parts: the feature extraction module and the feature output layer, as shown in Figure 4. Firstly, in order to better match blood cell features at different levels, the ASSP module is introduced to carry out the convolution of different cavities during feature extraction and connect feature map channels at different levels. Secondly, to make full use of the features of each level, the skip connection is redesigned and the multi-task learning module is introduced to optimize the segmentation results. The detailed technical and implementation of our modified Unet++ are explained below.

3.1.   Redesigned skip connection

UNet++ and its variant ^{[20,21,22,23,24,25]} have a great advantage in obtaining multi-scale feature maps because of their nested dense jump paths. However, since each node in the encoder and decoder is connected through the intermediate connection, this kind of dense connection leads to many model parameters and high computational complexity. Therefore, we present an improved UNet++ architecture, which only preserves the skip connections between the decoder and each node, the details are shown in Figure 4. The redesigned skip connection can ensure that the parameters are reduced without losing any information, and achieve better segmentation effect. The modified ${x}^{i, j}$ can be represented as:

3.2.   ASSP module

In deep learning, convolution is usually used for feature extraction. However, when the number of convolutions is too large, it is easy to lead to too many parameters and difficult weight optimization. By designing multiple parallel convolution kernels with different cavity rates, the size of the receptive field can be enlarged, and more spatial information can be retained while the number of parameters is unchanged. Figure 5 is the schematic diagram of dilated convolution, where (a) is a standard 3 × 3 convolution with the dilated factor of 1, and its receptive field size is 9; (b) is a 3 × 3 convolution with the dilated factor of 2, and its receptive field size is 49; (c) is a 3 × 3 convolution with the dilated factor of 3, and its receptive field size is 225. It can be seen that the applicability of the network to multi-scale objects can be increased by setting the inflation factor to obtain filters with different receptive fields.

The atrous spatial pyramid pooling module uses dilated convolution with different cavity rates for sampling, as shown in Figure 6. The ASPP module is mainly composed of eight convolution layers and one global average pooling (GAP) ^[26,27] layer in parallel. Specifically, a 3 × 3 convolution operation is performed on feature images ${X}^{\mathrm{0, 0}}$, ${X}^{\mathrm{1, 0}}$, ${X}^{\mathrm{2, 0}}$ and ${X}^{\mathrm{3, 0}}$ with stride of 16, 8, 4 and 2, respectively, while ${X}^{\mathrm{0, 0}}$ is subjected to one 1 × 1 convolution, pooling pyramid (three 3 × 3 dilated convolution) and global average pooling layer (1 × 1 convolution). After that, the corresponding results are concatenated. Therefore, the network not only realizes the acquisition of feature maps of different scales and the integration of cross-channel information, but also expands the regional and contextual features of the received information.

3.3.   Multi-sided output fusion

For the UNet ++ backbone network structure, its depth increases from left to right, and the feature map corresponding to each output result becomes more and more refined from left to right. At the rightmost end of the network, the segmentation effect of the cell region is the best because of the increase of convolutional layers. When making a change prediction, the deepest network output is generally selected as the final result. However, there is useful information in relatively shallow levels of output, even if there are error areas in the deepest output predictions. Therefore, our model combines the shallow features with the deepest output features, which is beneficial to improve the prediction and segmentation accuracy of the overall model. As shown in Figure 4, the feature maps generated by the five convolution units $\{{X}^{\mathrm{0, 4}}, {X}^{\mathrm{1, 3}}, {X}^{\mathrm{2, 2}}, {X}^{\mathrm{3, 1}}, {X}^{\mathrm{4, 1}}\}$ are calculated by a 1 × 1 convolution and sigmoid function to get the final segmentation result.

3.4.   Loss function

The Dice loss ^[28,29,30] is a region-based function, which can effectively solve the problem of imbalance between positive and negative samples because it only calculates a part of the region intersecting with the label without considering the background region. The loss function based on Dice coefficient can be expressed as:

where ${L}_{Dice}$ represents Dice loss, $N$ denotes the total pixel, ${y}_{i}$ represents the probability that the network predicts that voxel $i$ belongs to the foreground target, and ${\widehat{y}}_{i}$ is the corresponding value of voxel $i$.

4.   Experiments and results

In order to verify the high performance of the proposed method and its superiority for blood cell image segmentation, BCISC is used as the data set to test and discuss the experimental results. Next, we employ Mcc, Dice and Jaccard as evaluation indicators to compare with the current classical learning methods.

4.1.   Experimental settings

4.1.1.   Dataset description

The BCISC dataset ^[31] was provided by the Third People's Hospital of Fujian Province, which contains 400 training sets and 100 test sets. These images were from neutrophils, eosinophils, basophils, monocytes, and lymphocytes, to ensure the diversity of nuclear morphology. All images were taken by a physician for routine examination of the subject, and the ground truth was marked by a junior annotator and verified by an experienced radiologist. Taking into account the limitations of computer memory usage, we resized all images to 256 × 256 pixels.

4.1.2.   Evaluation metrics

Considering the specificity of the nucleus segmentation task, the Mcc ^[32,33], Dice coefficient ^[34,35] and Jaccard index ^[36,37] are introduced as evaluation index. They are defined as:

where TP, TN, FP, and FN denote numbers of true positives, true negatives, false positives, and false negatives on the pixels set, respectively.

4.1.3.   Implementation details

All algorithms use the open-source deep learning framework Keras to write the network structure and realize the training of the network. The hardware and environment of the experiment are Windows10 workstation configured with an Intel Xeon Gold 6248R CPU@3.00 GHz processor, 128 GB of 3200MHz DDR4 ECC RDIMM, NVIDIA Quadro RTX A6000 GPU memory of 48 GB. During model training, the original images and labels were sent to the network, and the Adam optimizer was used to optimize the network. The initial learning rate was set to 0.001, the number of training iterations was set to 400, and the batch size was set to 16.

4.2.   Quantitative and qualitative results

Figure 7 shows the convergence curve of the loss function and accuracy. It can be seen from Figure 7(a) that the number of trainings in this experiment is 400. During the first 20 training sessions, the gradient descends faster and the value of the loss function decreases faster. After that, the decline speed of the loss function becomes slower than before, which can avoid missing the optimal value due to the large gradient descent pace. When the loop reaches a certain level, the model optimization is no longer obvious. Figure 7(b) shows the changing trend of the accuracy index during the training process. As can be seen from the graph, the value of accuracy increases rapidly in the early stage of training, and then gradually becomes stable, with only a small range of fluctuation.

To verify the performance superiority of our proposed approach on blood cell segmentation task, it is compared with DenseUnet ^[38], FCN ^[39], Segnet ^[40], U-Net+++ ^[41], UNet++ ^[14] and U-Net ^[6] methods. The main evaluation indicators included Mcc, Dice and Jaccard, and the results are listed in Table 1. It can be seen from the table that after multiple pooling, the feature map size of the FCN method is very small. Although it contains more semantic information, there are serious false detections and missed detections. Although it contains a lot of semantic information, there is still serious false detection and missed detection phenomena, so all indicators are the lowest. The U-Net and Segnet models allow the decoder to learn the relevant feature information lost in the encoder pooling at each stage through the skip-connected architecture, so the performance of each evaluation index is better than FCN. The DenseUnet structure greatly expands the actual depth of the network and improves feature utilization. Therefore, it has great advantages over ordinary convolutional networks. UNet++ and UNet+++ have nested dense jump paths, making the semantic level of the encoder feature graph closer to the level of the corresponding encoder part, so it has been significantly improved on the basis of U⁃Net, and the phenomenon of false detection and missed detection is reduced. After the introduction of the ASSP module and multi-sided output fusion strategy, the Dice loss function is adopted to reduce the impact of sample imbalance. When compared with the U-Net, the Mcc, Dice and Jaccard values of our model are increased to 3.30%, 3.25% and 5.48%, respectively. The comprehensive evaluation index shows that our network has great advantages in cell localization and extraction.

To further visually compare the differences in blood cell segmentation results by different network models, some results with obvious contrast were selected from the BCISC dataset for intuitive qualitative analysis and then compared with the segmentation results manually annotated by experts, as shown in Figure 8. According to the visualization results, the segmentation effect of FCN, Segnet and U-Net in the complex background is not very ideal, and the nucleus cannot be completely segmented. Through the multi-scale feature extraction techniques, U-Net+++ and UNet++ can improve the integrity of complex cell segmentation, but the edge details are obviously insufficient. However, the convolution kernel of the proposed model can extract information at different scales and suppress useless feature information, so it can better demonstrate the superiority of its segmentation performance.

5.   Conclusions

In this study, based on the UNet++ architecture, we describe a new end-to-end method for blood cell segmentation on pathological images, which achieves good results in both localization and boundary segmentation. This is due to the introduction of the ASSP module, which can obtain blood cell image information features at each layer from different receiving fields. On the other hand, the multi-sided output fusion strategy can fuse feature information of different semantic levels to further improve the accuracy of detection and recognition. The quantitative and qualitative results on cell images indicate that, compared with other advanced deep learning-based methods, our model has great advantages in Mcc, Dice and Jaccard values. In the future, we will apply our technology to other cell images and medical images, and try to build an automated diagnostic system that can more accurately distinguish or predict benign and malignant lesions.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Nos. 62102227, 51805124, 62101206), Zhejiang Basic Public Welfare Research Project (Nos. LZY22E050001, LZY22D010001, LGG19E050013, LZY21E060001), Science and Technology Major Projects of Quzhou (2021K29, 2022K56).

Conflict of interest

The authors declare there is no conflict of interest.