Three-dimensional image reconstruction based on improved U-net network for anatomy of pulmonary segmentectomy

1.   Introduction

Lung cancer has the highest morbidity and mortality ^[1,2]. With the popularization of low-dose computed tomography (CT), the detection rate of early lung cancer with ground glass opacity as the primary imaging manifestation has increased significantly ^[3]. Anatomic pulmonary segmentectomy is a good choice for some patients with early lung cancer who are not suitable for pulmonary lobectomy. At present, its clinical application in thoracic surgery has been gradually promoted. The critical technology of segmentectomy is to distinguish the anatomical structure of pulmonary vessels and bronchus. However, the anatomical structure pulmonary segments are relatively complex, leading to high risk and operation difficulty. For standardization training residents, it is challenging to master the anatomical points of pulmonary segmentectomy in a short period.

According to our department's routine diagnosis and treatment, patients with pulmonary nodules manifested by ground glass shadow need to receive CT angiography before operation. After three-dimensional (3D) reconstruction of pulmonary vessels and bronchus, the clinicians can accurately locate the mass and analyze its relationship with segmental pulmonary vessels and adjacent structures ^[4,5,6]. Simultaneously, the video-assisted thoracic surgery (VATS) was used to perform the operation and video recording, conducive to fine dissection of the pulmonary segment and intraoperative and postoperative teaching ^[7,8].

To obtain high-quality 3D reconstruction images efficiently, the automatic and accurate segmentation of complete pulmonary parenchyma has become an important research topic ^[9,10]. Many corresponding methods have been proposed. Hu et al. ^[11] proposed the threshold segmentation method, but the segmentation task's result with a similar gray value is not satisfactory. Shelhamer et al. ^[12] give up the full connection layer with huge parameters and replaces it with a convolution layer. The U-net network contains the same number of upsampling layers and downsampling layers. It can obtain high accuracy segmentation results in a small number of data sets ^[13,14]. The residual network allows the original input information to be directly transferred to the following layers, which provides a new idea for applying neural networks in image segmentation. The 3D convolutional neural network structure for lung parenchyma segmentation also was used for medical image segmentation and exposed the defects such as large amount of calculation and long time consuming ^[15,16].

Therefore, based on the classical convolutional neural network structure, this study adds two modules (dilated convolution and parallel pooling) between the upper and lower sampling to realize the fast and effective lung tissue segmentation and reconstruct the pulmonary segment. We applied this method to the anatomy teaching of thoracic surgery and achieved satisfactory results. It is reported as follows.

2.   Materials and methods

2.1.   Research objects

From January 2019 to December 2020, 40 standardization training residents of thoracic surgery in the First Affiliated Hospital of Anhui Medical University were selected as the research objects. They were randomly divided into observation and control groups. There was no significant difference between the two groups in gender and age (P > 0.05). The teacher group is a double tutor system, with one associate professor of anatomy and one deputy chief physician of thoracic surgery.

Six patients who had undergone pulmonary segmentectomy were selected. All patients had preoperative thin slice CT examination and intraoperative video. The observation group was taught by 3D image reconstruction based on the improved U-net network, and conventional thin-slice CT images taught to the control group. The VATS video was used to validate pulmonary segmentectomy related anatomical knowledge.

2.2.   Improved U-net network structure

The improved U-net network structure used in this study was shown in Figure 1. In this new network, two modules, dilated convolution and parallel pooling were added between the downsampling and up a sampling of the classic U-net network.

2.2.1.   Dilated convolution module

The dilated convolution module designed in this paper includes five different branches of the original feature graph, and seven dilated convolution operations with varying rates of convolution (Figure 2). With the increase of dilated convolution rate, the receptive field of feature extraction increases gradually. The global information of the image was added to make up for the semantic information loss caused by the previous deep convolution.

2.2.2.   Parallel pooling module

The parallel pooling module designed in this paper is similar to spatial pyramid pooling (Figure 3). Four pooling cores of different sizes were used to check the input feature images for information integration. Other pooling cores extract information of various practical fields of vision. While reducing parameters, the different area of vision features extracted by the dilated convolution module are roughly retained.

2.3.   Lung parenchyma segmentation based on improved U-net network

Firstly, the chest CT images were sliced continuously along three axes (transverse, sagittal, and coronal plane) according to the image size, and the window was adjusted. Then, the processed slice images of different axes are input into the improved U-net network for training, and three optimal segmentation models corresponding to different axes are obtained. Next, three axial slices of each complete image were put into the trained model for testing. Then, the corresponding results of three axial slices are stacked into the original size to obtain three 3D segmentation results corresponding to different axial directions. Finally, the 3D segmentation results of each image corresponding to different axes were obtained according to the voting rule of voxels at the same position with the same weight, and the final pixel value of each voxel was obtained to form the final segmentation result. The flow chart of the specific method was shown in Figure 4.

The implementation process of 3D is as follows: firstly, stack the segmentation results of three different axial lung parenchyma, then calculate the equal weight voting result $\boldsymbol{f}\left(\boldsymbol{x}\right)$ of the voxel pixel prediction value corresponding to the three axial stacking results, and determine that the point belongs to the lung parenchyma or background through the calculation result of $\boldsymbol{f}\left(\boldsymbol{x}\right)$. The function expression of $\boldsymbol{f}\left(\boldsymbol{x}\right)$ is as follows:

In formula, ${\boldsymbol{x}}_{\bf{1}}$, ${\boldsymbol{x}}_{\bf{2}}$ and ${\boldsymbol{x}}_{\bf{3}}$ represent the predicted pixel values of voxels corresponding to the stacking results of transverse, sagittal, and coronal plane, respectively. If $\boldsymbol{f}\left(\boldsymbol{x}\right)$ ≥ 0.5, this voxel point is judged as lung parenchyma; otherwise, it is judged as background.

2.4.   Effect evaluation

Objective and subjective assessment methods evaluate the teaching effect. After the training process was completed, the preoperative and intraoperative video data of five patients with pulmonary nodules who underwent pulmonary segmentectomy were selected as the objective assessment data. According to the preoperative image and intraoperative video to determine: 1) location of pulmonary nodule in pulmonary segments; 2) surgical plan; 3) segmental pulmonary artery, 4) segmental pulmonary vein, 5) segmental bronchus. Take the average score of five patients, 20 points for each item, a total of 100 points.

The subjective assessment was reflected by self-designed questionnaire. Each group of students had a questionnaire survey when they finished the course. The questionnaire involved learning interest, content understanding, clinical thinking mode, understanding of operation process, clinical knowledge recall, language expression ability, etc.

2.5.   Statistical analysis

The data were processed by SPSS 25.0 statistical software. The t test was used to compare the differences between groups in objective scores. The chi-square test was used to compare the differences in questionnaires.

3.   Results

3.1.   Comparison of objective scores for the 3D reconstruction

The individual and total scores of objective scores were showed in Table 1. The observation group's total score was 87.4 ± 4.9, which was higher than the score in the control group (69.1 ± 7.8, P < 0.001). For the individual score, the results were similar.

3.2.   Comparison of the questionnaire for the 3D reconstruction

The results of the questionnaire were shown in Table 2. The satisfaction of learning interest, content understanding, clinical thinking mode, and understanding of operation process in the observation group was higher than that of the control group (all P < 0.05). There was no significant difference between the two groups in recall of clinical knowledge and language expression ability (P > 0.05).

4.   Discussion

Clinical medicine is a highly practical subject, and thoracic surgery has higher professional and practical requirements, which can be well mastered only through long-term practice accumulation. However, at present, there are some limitations in the standardized training of residents in thoracic surgery. For the regular trainees in cardiothoracic surgery, the students only come to the department of thoracic surgery for half a year in the second or third year of standardized training. For students of other majors, the rotation time in thoracic surgery is only 1–2 months. As a provincial-level first-class hospital in Anhui Province, many standardization training residents from primary hospitals come to our hospital every year. How to let young training students master the latest technology and development direction of thoracic surgery quickly is a crucial problem to be solved in clinical teaching.

With the popularity of low-dose CT scans in our country, the detection rate of lung cancer is significantly improved, especially for early lung cancer diagnosis ^[17,18,19]. At present, for early lung cancer treatment with ground glass opacity as the primary imaging manifestation, the original standard pulmonary lobectomy has gradually turned to anatomical pulmonary segmentectomy. Anatomic segmentectomy has become a hot spot in the treatment of early lung cancer. However, the pulmonary segment's anatomic structure is complex, and its arteries, veins, and bronchial trees vary and cross each other ^[20,21,22]. It is difficult to determine the attribution of some nodules near the septum between pulmonary segments. Conventional CT scans before operation can provide limited information, making it very difficult for students to learn the anatomical structure of segmentectomy. Even if VATS is routinely used for surgery, assistants or visiting doctors can see the operation process directly from the screen, there are still many feedbacks from the standardization training residents that they cannot understand and master the process of segmentectomy.

The current imaging system can achieve the simple 3D reconstruction of lung tissue, which can be used for preoperative 3D image reconstruction of patients. After reconstruction, the location of the nodules in the pulmonary segment, the blood supply, and the anatomy of bronchus can be visually displayed on the computer, which helps beginners master the anatomical structure of pulmonary segments. In our department, the patients who want to accept pulmonary segmentectomy need conventional 3D reconstruction before the operation to understand the anatomical structure of individualized pulmonary segments. To ensure and improve the quality of standardized training of residents in our department, we try to apply 3D image reconstruction technology to these students' clinical teaching.

In the past, we mainly used thin-slice CT images combined with operation videos to teach students related knowledge of pulmonary segmentectomy. Because the regular CT scanning is a two-dimensional image, the students need to construct the adjacent relationship of pulmonary nodules and the pulmonary segment by imagination, which is very difficult for the junior students. Therefore, we randomly selected some students to receive training by 3D image reconstruction based on the improved U-net network. This U-net network has two new modules that can extract the image's features more completely and filter out the adequate information of different visions to the greatest extent. Thus, the pulmonary segments in 3D images can show more accurately and concretely. After the students in the observation group entered the department, they will complete the 3D reconstruction based on the improved U-net network under the teacher's guidance. The generated image can rotate 360° freely on the computer screen, which helps students observe the anatomical structure of the pulmonary segments from all angles. After that, we used the data of five patients who underwent segmentectomy to evaluate the teaching effectiveness.

The teaching effect is evaluated by objective and subjective assessment methods. For the objective evaluation, we asked students to identify the location of nodule in pulmonary segment, and the best surgical plan through the preoperative image data. Also, through the VATS video, the segmental pulmonary artery, vein, and bronchus should be distinguished. The results showed that both the individual and total scores in the observation group were higher than those in the control group. The questionnaire indicates that most of the students agree with this teaching model. The 3D reconstruction teaching group students have significantly improved in four aspects: learning interest, content understanding, clinical thinking mode, and understanding of operation process.

5.   Conclusions

In conclusion, our 3D image reconstruction based on the improved U-net network combined with VATS video has achieved good results in the clinical teaching of standardization training residents in thoracic surgery. It deepens students' understanding of segmental structure and operation and arouses students' learning enthusiasm and interest, improves the cultivation and application of students' clinical thinking mode, and enhances students' understanding of the surgical process of segmental resection. To a certain extent, it alleviates the contradiction between short teaching time and complex learning content.

Acknowledgments

The study was supported by the Provincial Quality Engineering Project of Colleges and Universities in Anhui Province (2019mooc084) and Key Project of Anhui Province University Outstanding Young Top Talent Support Plan (gxyqZD2018022).

Conflict of interest

All authors declare no conflicts of interest in this paper.