Search Advanced

Mathematical Biosciences and Engineering

2024, Volume 21, Issue 1: 1228-1248. doi: 10.3934/mbe.2024052
Previous Article Next Article
Research article Special Issues

Entity recognition of railway signal equipment fault information based on RoBERTa-wwm and deep learning integration

  • 1.
    School of Automation and Electrical Engineering, Lanzhou Jiaotong University, Lanzhou 730070, China
  • 2.
    The Center of National Railway Intelligent Transportation System Engineering and Technology, China Academy of Railway Sciences Corporation Limited, Beijing 100081, China
  • 3.
    Signal and Communication Research Institute, China Academy of Railway Sciences Corporation Limited, Beijing 100081, China
  • Received: 31 October 2023 Revised: 11 December 2023 Accepted: 18 December 2023 Published: 26 December 2023

  • The operation and maintenance of railway signal systems create a significant and complex quantity of text data about faults. Aiming at the problems of fuzzy entity boundaries and low accuracy of entity recognition in the field of railway signal equipment faults, this paper provides a method for entity recognition of railway signal equipment fault information based on RoBERTa-wwm and deep learning integration. First, the model utilizes the RoBERTa-wwm pretrained language model to get the word vector of text sequences. Second, a parallel network consisting of a BiLSTM and a CNN is constructed to obtain the context feature information and the local attention information, respectively. Third, the feature vectors output from BiLSTM and CNN are combined and fed into MHA, focusing on extracting key feature information and mining the connection between different features. Finally, the label sequences with constraint relationships are outputted in CRF to complete the entity recognition task. The experimental analysis is carried out with fault text of railway signal equipment in the past ten years, and the experimental results show that the model has a higher evaluation index compared with the traditional model on this dataset, in which the precision, recall and F1 value are 93.25%, 92.45%, and 92.85%, respectively.

    Citation: Junting Lin, Shan Li, Ning Qin, Shuxin Ding. Entity recognition of railway signal equipment fault information based on RoBERTa-wwm and deep learning integration[J]. Mathematical Biosciences and Engineering, 2024, 21(1): 1228-1248. doi: 10.3934/mbe.2024052

    Related Papers:

    [1] Xi Lu, Zejun You, Miaomiao Sun, Jing Wu, Zhihong Zhang . Breast cancer mitotic cell detection using cascade convolutional neural network with U-Net. Mathematical Biosciences and Engineering, 2021, 18(1): 673-695. doi: 10.3934/mbe.2021036
    [2] Santisudha Panigrahi, Ruchi Bhuyan, Kundan Kumar, Janmenjoy Nayak, Tripti Swarnkar . Multistage classification of oral histopathological images using improved residual network. Mathematical Biosciences and Engineering, 2022, 19(2): 1909-1925. doi: 10.3934/mbe.2022090
    [3] Qun Xia, Yangmei Cheng, Jinhua Hu, Juxia Huang, Yi Yu, Hongjuan Xie, Jun Wang . Differential diagnosis of breast cancer assisted by S-Detect artificial intelligence system. Mathematical Biosciences and Engineering, 2021, 18(4): 3680-3689. doi: 10.3934/mbe.2021184
    [4] Thaweesak Trongtirakul, Sos Agaian, Adel Oulefki . Automated tumor segmentation in thermographic breast images. Mathematical Biosciences and Engineering, 2023, 20(9): 16786-16806. doi: 10.3934/mbe.2023748
    [5] Yuanhong Jiang, Yiqing Shen, Yuguang Wang, Qiaoqiao Ding . Automatic recognition of white blood cell images with memory efficient superpixel metric GNN: SMGNN. Mathematical Biosciences and Engineering, 2024, 21(2): 2163-2188. doi: 10.3934/mbe.2024095
    [6] Xinlin Liu, Viktor Krylov, Su Jun, Natalya Volkova, Anatoliy Sachenko, Galina Shcherbakova, Jacek Woloszyn . Segmentation and identification of spectral and statistical textures for computer medical diagnostics in dermatology. Mathematical Biosciences and Engineering, 2022, 19(7): 6923-6939. doi: 10.3934/mbe.2022326
    [7] Yali Ouyang, Zhuhuang Zhou, Weiwei Wu, Jin Tian, Feng Xu, Shuicai Wu, Po-Hsiang Tsui . A review of ultrasound detection methods for breast microcalcification. Mathematical Biosciences and Engineering, 2019, 16(4): 1761-1785. doi: 10.3934/mbe.2019085
    [8] Bo An . Construction and application of Chinese breast cancer knowledge graph based on multi-source heterogeneous data. Mathematical Biosciences and Engineering, 2023, 20(4): 6776-6799. doi: 10.3934/mbe.2023292
    [9] Junjing Cai, Qiwei Wang, Ce Wang, Yu Deng . Research on cell detection method for microfluidic single cell dispensing. Mathematical Biosciences and Engineering, 2023, 20(2): 3970-3982. doi: 10.3934/mbe.2023185
    [10] S. Dinesh Krishnan, Danilo Pelusi, A. Daniel, V. Suresh, Balamurugan Balusamy . Improved graph neural network-based green anaconda optimization for segmenting and classifying the lung cancer. Mathematical Biosciences and Engineering, 2023, 20(9): 17138-17157. doi: 10.3934/mbe.2023764

  • The operation and maintenance of railway signal systems create a significant and complex quantity of text data about faults. Aiming at the problems of fuzzy entity boundaries and low accuracy of entity recognition in the field of railway signal equipment faults, this paper provides a method for entity recognition of railway signal equipment fault information based on RoBERTa-wwm and deep learning integration. First, the model utilizes the RoBERTa-wwm pretrained language model to get the word vector of text sequences. Second, a parallel network consisting of a BiLSTM and a CNN is constructed to obtain the context feature information and the local attention information, respectively. Third, the feature vectors output from BiLSTM and CNN are combined and fed into MHA, focusing on extracting key feature information and mining the connection between different features. Finally, the label sequences with constraint relationships are outputted in CRF to complete the entity recognition task. The experimental analysis is carried out with fault text of railway signal equipment in the past ten years, and the experimental results show that the model has a higher evaluation index compared with the traditional model on this dataset, in which the precision, recall and F1 value are 93.25%, 92.45%, and 92.85%, respectively.


    1.   Introduction

    Mitosis counting on Hematoxylin and Eosin (H & E) [1,2] stained breast histopathology slides is one of the important prognostic markers that indicate the proliferative activity of breast carcinoma. The Nottingham Histopathology Grading (NHG) is a scoring system to assess the grade of breast carcinoma [2]. The NHG system was developed based on the following three morphological criteria: mitosis count, degree of tubule formation, and nuclear pleomorphism [2]. Each criterion of the NHG system is scored from 1 to 3 and the summation of these scores provides the grade. The score of breast carcinoma is determined manually by the histopathologist. There are three grades for breast carcinoma: scores 3–5 for Grade 1; scores 6 and 7 for Grade 2; scores 8 and 9 for Grade 3. Distinct prognosis and treatment planning are developed according to the obtained grade.

    Conventional assessment of mitosis count is done through visual examination method under a light microscope. For each histopathology slide, multiple images/frame (approximate 2000 frames) is created corresponding to different areas on a slide [3]. The slide is examined at a high-power field (HPF) under 40x magnifications. As the histopathology slides are complex and heterogeneous [4,5,6], visual examination requires the slide to be examined frame by frame. During the slide examination, the histopathologist needs to identify a defined area express in mm2 with the highest mitosis activity to provide scores based on the NHG system [2]. In many hospitals, histopathologist typically handles around 100 grading cases per day [7]. Thus, conventional visual examination of breast carcinoma tissues is cumbersome and tedious [1,8]. During the past few decades, the emergence of Whole-Slide-Imaging (WSI) scanner has made a giant leap in the pathology laboratory [9,10,11]. This provides a significant benefit over the conventional histopathology slides. WSI scanner takes the analogue histopathology slides as input and produces digital images as output.

    Mitosis detection is very challenging [12]. Most of the challenges arise from the variability of the mitosis cell appearance and the imperfection of H & E staining [13,14]. Mitosis cell is formed through a process known as mitosis [15,16]. Mitosis is a process where the chromosome is replicated and separated into two new nuclei. This process involves five phases [16]: interphase, prophase, metaphase, anaphase, and telophase (Figure 1, top row). The appearance of the mitosis cell varies from a round shape with rough texture (interphase and prophase) into a narrow oval shape with hyperchromatic properties (metaphase). It then becomes a separate-like nuclei with rough texture and hyperchromatic properties (anaphase and telophase) [16]. Apart from this, there are few artefacts such as karyorrhectic and apoptotic nuclei that have similar appearance with the true mitosis cells (Figure 1, bottom row). Imperfection in the H & E staining, variability of the mitosis cells and artefacts could influence the mitosis count and breast carcinoma grading.

    Figure 1.  The top row shows samples of mitosis cells in different phases, whereas the bottom row shows the artefacts with similar appearances to the mitosis cells.
    DownLoad: Full-Size Img PowerPoint

    In recent years, the research interest in mitosis detection is mainly driven by the open challenges and availability of public access datasets. These datasets allow the researchers to test and validate the respective methods and benchmark the results with other re-searchers across the globe. To date, the MITOS [17], AMIDA [18], MITOS-ATYPIA [19], and TUPAC [20] are four public access datasets, specifically for mitosis detection in breast carcinoma. Table 1 summarizes the public access datasets for mitosis cells detection.

    Table 1.  Public access datasets for mitosis cells detection.
    Datasets Year Resolutions Magnification Number of
    images    		 Number of
    mitosis cells
    MITOS [17] 2012 Aperio scanner: 2084 × 2084
    Hamamatsu scanner: 2250 × 2252
    Multispectral images: 1360 × 1360    		 40 x 50 327
    AMIDA [18] 2013 2000 × 2000 40 x 311 550
    MITOS-ATYPIA [19] 2014 Aperio scanner: 1539 × 1376
    Hamamatsu scanner: 1663 × 1485    		 40 x 1200 749
    TUPAC [20] 2016 2000 × 2000 40 x 321 Unlabeled
     | Show Table
    DownLoad: CSV

    The recent works in mitosis cells detection can be characterized into three different approaches: handcrafted feature-based approach, deep learning approach, and fusion approach. In mitosis cells detection, the unbalanced data between the mitosis cells and non-mitosis cells are significant. Therefore, to ensure a fair comparison, results benchmarking is usually performed by observing the percentage of F1-score obtained from a proposed method. Tables 2–4 show the recent works in mitosis detection using the public access datasets with handcrafted feature-based, fusion, and deep learning approaches, respectively.

    Table 2.  Recent works in mitosis cell detection using handcrafted feature-based approach.
    Authors Year Datasets Methods F1-score (%)
    Sommer et al. [32] 2012 MITOS [17] Mitosis detection using hierarchical learning workflow 62.9
    Irshad [33] 2013 Mitosis detection using the multi-channel features computation framework where the nuclei feature extraction is performed in selected channels of color spaces at a fixed image scale 63.0
    Tek [34] 2013 Mitosis detection via application of generic features and an ensemble of cascade adaboosts 58.0
    Irshad et al. [35] 2013 Mitosis detection using texture, sift features, and HMAX biologically inspired approach 76.0
    Tashk et al. [25] 2013 Mitosis detection using objective and pixel-wise textural features 70.1
    Khan et al. [36] 2013 Mitosis detection using Gamma-Gaussian mixture model 75.4
    Irshad et al. [21] 2014 Mitosis detection via the selection of spectral bands and focal plane 74.0
    Nateghi et al. [37] 2014 Mitosis detection using genetic algorithm 78.4
    Irshad et al. [21] 2014 Mitosis detection using multispectral band selection and spatial characterization 57.0
    Lu and Mandal [22] 2014 Mitosis detection using three main stages: discriminative image generation, mitosis cell candidate detection and segmentation, and followed by a classification 47.9
    Tashk et al. [23] 2015 Mitosis detection using a combination of textural, statistical, and innovative mathematical features 82.7
    Nateghi et al. [25] 2017 Mitosis detection using Maximized Inter-Class Weighted Mean method 88.4
    Nateghi et al. [25] 2017 AMIDA [18] Mitosis detection using Maximized Inter-Class Weighted Mean method 75.3
    Paul and Mukherjee [24] 2015 MITOS-ATYPIA [19] Mitosis detection by implementing the Relative Entropy Maximized Scale Space using area morphological opening and closing 73.4
    Nateghi et al. [25] 2017 Mitosis detection using Maximized Inter-Class Weighted Mean method 83.7
     | Show Table
    DownLoad: CSV
    Table 3.  Recent works in mitosis cell detection using fusion approach.
    Authors Year Datasets Methods F1-score (%)
    Malon and Cosatto [46] 2013 MITOS [17] Mitosis detection using the convolutional neural networks and seeded blob features 55.7
    Wang et al. [28] 2014 Mitosis detection using a combination of handcrafted and convolutional neural network features 73.4
    Mahmood et al. [31] 2020 Mitosis detection using a multistage mitosis-cell-detection method based on faster region convolutional neural network and deep convolutional neural network 85.8
    Beevi et al. [29] 2017 MITOS-ATYPIA [19] Mitosis detection using a multi-classifier system 76.7
    Das and Dutta [30] 2019 Mitosis detection using the deep convolution neural network with wavelet decomposed patches 55.9
    Saha et al. [12] 2018 Mitosis detection using a supervised model of deep learning architecture with handcrafted features. 90.0
    Mahmood et al. [31] 2020 Mitosis detection using a multistage mitosis-cell-detection method based on faster region convolutional neural network and deep convolutional neural network 69.1
     | Show Table
    DownLoad: CSV
    Table 4.  Recent works in mitosis cell detection using deep learning approach.
    Authors Year Datasets Methods F1-score (%)
    Ciresan et al. [38] 2013 MITOS [17] Mitosis detection using the deep max-pooling convolutional neural networks 78.2
    Chen et al. [39] 2016 Mitosis detection using the deep regression network 79.0
    Wahab et al. [40] 2017 Mitosis detection using the two-phase deep convolutional neural network 79.0
    Alom et al. [27] 2020 Mitosis detection via an end-to-end multi-task learning system from pathological images which is termed as "MitosisNet" 87.8
    Sebai et al. [10] 2020 Mitosis detection using multitask deep learning framework for object detection and instance segmentation Mask RCNN 86.3
    Xi et al. [41] 2020 Mitosis detection using the cascaded convolutional neural network (CNN) based on UNet. 83.6
    Albarqouni et al. [42] 2016 AMIDA [18] Mitosis detection using the convolutional neural network (CNN) via additional crowdsourcing layer (AggNet) 74.2
    Wollmann and Rohr [43] 2017 Mitosis detection using deep residual Hough voting 60.9
    Romo-Bucheli et al. [44] 2019 Mitosis detection using deep learning strategy with gene expression derived risk categories in estrogen receptor-positive breast carcinomas 55.6
    Chen et al. [45] 2016 MITOS-ATYPIA [19] Mitosis detection using the deep cascaded convolutional neural network 48.2
    Li et al. [8] 2018 Mitosis detection using the multi-stage deep learning framework 43.7
    Cai et al. [26] 2019 Mitosis detection using the modified regional convolutional neural network 58.5
    Alom et al. [27] 2020 Mitosis detection via an end-to-end multi-task learning system from pathological images which is termed as "MitosisNet" 75.9
    Sebai et al. [10] 2020 Mitosis detection using multitask deep learning framework for object detection and instance segmentation Mask RCNN 60.8
    Xi et al. [41] 2020 Mitosis detection using the cascaded convolutional neural network (CNN) based on UNet. 57.1
    Wahab et al. [40] 2017 TUPAC [20] Mitosis detection using the two-phase deep convolutional neural network 55.0
    Cai et al. [26] 2019 Mitosis detection using the modified regional convolutional neural network 73.6
     | Show Table
    DownLoad: CSV

    In handcrafted feature-based approaches, Irshad et al. proposed a mitosis cells detection framework that implements multispectral band selection and spatial characterization [21]. The selected spectral bands are meant to reduce the computation load and storage complexity, whereas the multi-features are meant to discriminate between the mitosis cells and non-mitosis cells. Lu and Mandal proposed a multi-stage mitosis cells detection framework that involved: discriminative image generation stage, mitosis candidate detection, and segmentation stage, followed by the classification stage [22]. Tashk et al. performed mitosis cells detection by using a combination of different features: Completed Local Binary Pattern, Statistical Moment Entropy, and Stiffness Matrix which includes textural, statistical, and innovative mathematical features [23]. Paul and Mukherjee proposed a mitosis cells detection framework that is driven by domain knowledge in mitosis cells [24]. Based on the fact that the intensity pattern of the mitosis remains similar in breast histopathology images, mitosis cells detection can be done by implementing the Relative Entropy Maximized Scale Space using area morphological opening and closing approaches. Nateghi et al. proposed a mitosis cells detection framework that implemented a novel Maximized Inter-class Weighted Mean method to reduce the false positive in mitosis cells candidates selection stage [25]. Color, textural, and shapes features are then extracted from the mitosis cells candidates and are used as input for the classifier.

    In deep learning approaches, Li et al. proposed a novel multi-stage deep learning framework [8]. The deep segmentation framework is able to detect the mitosis cells regions by using a weak label such as the centroid pixel of the mitosis cells. Also, the deep learning framework is facilitated using contextual region information and a deep verification network. The proposed method is termed as DeepMitosis. Cai et al. proposed an effective mitosis cells detection framework using a modified regional convolutional neural network (RCNN) [26]. The proposed RCNN implemented the Resnet-101 network as the backbone feature extractor and is pre-trained on the ImageNet dataset. The output channels are then reduced using a large separable convolution. This proposed method is found promising in reducing the computation time in both training and inference stages. Alom et al. proposed an end-to-end mitosis cells detection framework, namely MitosisNet [27]. The proposed method constitutes of segmentation, detection, and classification stages. The output from segmentation and detection stages are used as mitosis cells reference regions while further confirmation is performed using the classification stage. Sebai et al. proposed a multi-task deep learning framework for mitosis cells detection and instance segmentation using Mask RCNN which is termed as MaskMitosis [10]. The MaskMitosis is meant to perform multiple tasks such as mitosis mask estimation, bounding box labelling on weakly annotated and unannotated datasets, mitosis cells localization, and mitosis cells classification using the estimated pixel-level annotations.

    In fusion approaches, Wang et al. proposed a mitosis detection framework that combines the convolution neural network (CNN) and handcrafted features such as morphological, textural, and color features [28]. The proposed fusion approach was found promising in reducing the computation load while achieving high accuracy. Beevi et al. proposed a multi-classifier system for mitosis cells detection using the Deep Belief Network [29]. The proposed system combined methods such as the Localized Active Contour Model and Krill Herd Algorithm for segmentation purposes and a multi classifier system to improve the prediction accuracy. Das and Dutta proposed a deep CNN with wavelet decomposed images for mitosis detection [30]. The input images are decomposed into smaller images using Haar wavelet. These images are then used to develop a deep CNN for automatic mitosis cells detection. Mahmood et al. proposed a multi-stage mitosis cells detection framework based on a faster region convolutional neural network (Faster R-CNN) and deep CNNs [31]. The proposed framework included a post-processing stage that aims to reduce the false positive using handcrafted features.

    Overall, the recent works aforementioned benefit from applying similar image processing methodologies for mitosis detection. These methodologies included image pre-processing, mitosis cells candidate estimation and segmentation, feature extraction, feature selection, and classification. There are two known challenges in mitosis detection: the number of mitosis cells is very low as compared to non-mitosis cells and the mitosis can present in various shapes and morphologies (refer to the top row in Figure 1). A plethora of non-mitosis cells are usually found after the segmentation stage. This contributes to a high number of false positives and impinges the overall detection accuracy. In addition, a large number of non-mitosis cells can significantly increase the computation time of the proposed framework. Several recent works highlight and propose solutions to this issue. Khan et al. proposed a gamma-gaussian mixture model to reduce the number of non-mitosis objects after the segmentation stage [36]. A more recent work by Nateghi et al. proposed a false positive mitosis candidate reduction using a novel Maximized Inter-Class Weighted Mean method [25].

    Recognition of complex and inconsistent objects specifically in the medical histopathology images could be achieved by incorporating domain knowledge in the field of interest [24,47,48]. There are studies obtained encouraging results when incorporating domain knowledge, image processing, and classification methods [24,48,49,50,51,52,53], but the number is very few. During manual vision inspection, histopathologists do not check the presence of mitosis on every cell in each frame under a light microscope. Histopathologists scan through the frames using the ball rolling method and scrutinize cells with characteristics (e.g., hyperchromatic properties) that are specifically demonstrated by mitosis cells. In this study, a more promising handcrafted feature-based approach mitosis detection framework is proposed. The proposed framework considered both the challenges highlighted herein. The proposed mitosis detection framework is driven by domain knowledge in mitosis such that the domain knowledge and strategies of the histopathologist (human expert) were utilized to guide the development of the image processing framework for mitosis detection. Additionally, the proposed framework clusters the segmented cells into different groups based on the morphological structures. This aims to triage and eliminate false positives (i.e., non-mitosis cells) in different phases (i.e., interphase, prophase, metaphase, anaphase, and telophase). The main novelty of the proposed detection framework lies within the simple yet powerful detection capability of the framework with knowledge transferred from human experts, producing promising detection results, reflected with high percentages in F1-score. To the best of our knowledge, mitosis cells detection using a knowledge-assisted detection framework by incorporating domain knowledge as well as knowledge transfer from human experts is yet to be available in the literature.

    The objectives of this study are: 1) to remove and minimize the segmented non-mitosis cells (false positive); 2) to provide a mitosis cells detection framework that can serve as the second opinion for histopathologists to facilitate the mitosis cells assessment procedure; 3) to validate the applicability of the proposed detection framework by benchmarking the output of the proposed detection framework with ground truth and other recent works.

    This paper is organized as follows: Section 2 details the methodology of the proposed mitosis detection framework; Section 3 describes the datasets used in this study; Section 4 presents the experimental results and analysis obtained from the proposed framework. In this section, the proposed framework is benchmarked and compared with ground truth annotated by the histopathologist (using the custom dataset) and recent works (using the MITOS dataset [17]) for validation purposes. In Section 5, the conclusion of this study is presented.

    2.   Methods

    The proposed mitosis cells detection framework consists of five main stages: 1) color normalization using the optimal stain vector approach [54]; 2) hyperchromatic nucleus segmentation performed using the knowledge-assisted K-Mean; 3) the proposed knowledge-assisted false positive reduction stage based on the domain knowledge in mitosis cells; 4) features extraction on morphological and textural features; 5) classification using the Support Vector Machine (SVM) with radial-based function (RBF) kernel. Figure 2 shows the block diagram of the proposed knowledge- assisted mitosis cells detection framework. The following subsections detail each stage in the proposed detection framework.

    Figure 2.  Block diagram of the proposed knowledge-assisted mitosis cells detection framework.
    DownLoad: Full-Size Img PowerPoint

    2.1.   Color normalization

    Color variation in H & E-stained breast histopathology images is a common issue that arises from the use of different stains/dyes reactivity and coloring from a different manufacturer, use of different WSI scanner, and use of different equipment. To ensure excellent stability in the performance of the proposed detection framework, color normalization shall be performed to minimize color variation amongst the input images in any datasets. In this study, the optimal stain vector approach [54] was implemented for color normalization purposes. Briefly, the input images were first transformed to the optical density domain. The singular value decomposition was then calculated from the optical density tuples. The optical density was then transformed and projected onto the plane and was normalized to the unit length. Next, calculate the angle of each point with respect to the first singular value decomposition direction. Mapping of direction was then performed. All the extreme values were then converted back to the optical density domain. The optimal stain vector approach formalized three outputs: a color normalized image, a Hematoxylin-stained image, and an Eosin-stained image. In this study, only the Hematoxylin-stained image is used as the mitosis cells captured only the Hematoxylin stain in the standard staining procedure. Figure 3 shows a sample of outputs obtained from the color normalization stage.

    Figure 3.  A sample of outputs obtained from the color normalization stage.
    DownLoad: Full-Size Img PowerPoint

    2.2.   Hyperchromatic nucleus segmentation

    By nature, the chromosomal condensation that occurs during mitosis resulted in a higher degree of Hematoxylin stain absorption during the staining procedure. This causes the mitosis cells to become remarkable (i.e., darker in color) than the normal cells and is prominent in the Hematoxylin-stained image (Figure 3(c)). Thus, it is reasonable to conclude that the nucleus with low intensity is closely related to the mitosis cells. In this study, the nucleus with remarkable intensity values (i.e., darker in color) as compared to the normal nucleus in the Hematoxylin-stained image is termed as the hyperchromatic nucleus.

    To perform hyperchromatic nucleus segmentation, the Hematoxylin-stained image was first converted to the cyan channel (determined heuristically). The number of clusters for K-Mean was set as three, such that cluster 1 for hyperchromatic nucleus (white objects), cluster 2 for tumor regions (grey regions), and cluster 3 for background (black regions). In hyperchromatic nucleus segmentation, only information from cluster 1 (i.e., hyperchromatic nucleus) is relevant to this study. Nevertheless, it is important to assign two different clusters for tumor regions and background. If only two clusters were assigned in this stage (i.e., cluster 1 for hyperchromatic nucleus and cluster 2 for tumor regions and back-ground), the background (i.e., unstained areas that appeared in black color in the cyan channel) would act as an outlier and impinge the clustering accuracy for both the clusters. For knowledge-assisted K-Mean, the initial centroids are not generated randomly but based on domain knowledge. Since the hyperchromatic nucleus, tumor regions, and background respectively appeared in white, grey, and black pixel ranges, the initial centroids for each cluster can be selected by simply partitioning the gray level (i.e., 255) into three regions, such that region 1 ranges from 171 to 255, region 2 ranges from 86 to 170, and region 3 ranges from 0 to 85. Then, the median intensity of each region is selected as the initial centroid.

    2.3.   Knowledge-assisted false positive reduction method

    The segmented hyperchromatic nucleus images contain mitosis and non-mitosis cells. Mitosis cells in different phases demonstrate different morphological features specifically in shape (see Figure 1, top row). The knowledge-assisted false positive reduction method was mainly based on domain knowledge in mitosis cells and is formalized through empirical analysis. To support the empirical analysis, Monte Carlo simulation [55,56] was implemented to determine the number of mitosis cells to be used in this stage. Parameters such as area, mean intensity, aspect ratio (AR), form factor (FF), and filled area (FA) are observed as random variables. The Cochran formula (refer to Eq (1)) is implemented to calculate the sampling data at a confidence level of 95.0%, provided the z value is at 1.96 (obtained from the z table).

    n0=z2pqe2 (1)

    where z denotes the z value, p (set as 0.5 in this study) denotes the estimated probability a mitosis cell is found; q denotes the probability of 1-p; and e denotes the margin of error. From Eq (1), sampling data of 385 is required in order to obtain a confidence level of 95.0%. Thus, in this stage, the empirical analysis was performed using 385 mitosis cells. It is important to remark that all the 385 mitosis cells used for empirical analysis are first undergone color normalization as described in Section 2.1. As the mitosis cells demonstrate different morphological features, typically in shape, the mitosis cells can be grouped into four shapes (Figure 4): Large Circular-Shaped (LCS) nucleus, Ellipse-Shaped (ES) nucleus, Hollow-Shaped (HS) nucleus, and Curl-Shaped (CS) nucleus. Domain knowledge of mitosis cells and strategies used by the histopathologist were incorporated in this stage (i.e., Checking procedures 1 and 2, as in Figure 5) to reduce the number of false positive. The steps in the knowledge-assisted false positive reduction stage are shown in Figure 5.

    Figure 4.  Four different shapes of nucleus. In images (a) to (d), the left image shows the RGB image of nucleus and the right image shows the segmentation results obtained using the knowledge-assisted K-Mean.
    DownLoad: Full-Size Img PowerPoint
    Figure 5.  Block diagram showing processes in the knowledge-assisted false positive reduction stage.
    DownLoad: Full-Size Img PowerPoint
    2.3.1.   Post-processing

    Post-processing was first implemented to eliminate the small objects, typically noise and/ or non-mitosis cells. Manual segmentation was performed on the 385 mitosis cells to obtain the area in pixel. The mean area (± standard deviation (SD)) for the mitosis cells was 312 (± 76) pixels. The smallest area of the mitosis cell is 118 pixels. To avoid over-elimination, in this step, only objects with areas less than half of this smallest size (i.e., 59) were eliminated from the images and considered as artefacts. Table 5 shows the results of the empirical analysis for the area.

    Table 5.  Results of empirical analysis for the area in 385 mitosis cells.
    Feature Parameters
    Mean (± SD) Maximum Minimum
    Area 312 (± 76) 564 118
     | Show Table
    DownLoad: CSV
    2.3.2.   Grouping

    As aforementioned, the knowledge-assisted false positive reduction stage is meant to triage and eliminate false positive in different phases (i.e., interphase, prophase, metaphase, anaphase, and telophase). Thus, the outputs from the previous step (i.e., post-processing step) are characterized into four different groups to facilitate the morphological features screening step in Section 2.3.3. The four groups were LCS, ES, HS, and CS groups. These groups are derived from the morphological features of the mitosis cells from different phases (i.e., interphase, prophase, metaphase, anaphase, and telophase) as in Figure 1, top row, such that the LCS group aims to remove false positive in interphase and prophase, ES group in metaphase; HS group in anaphase, and CS group in telophase.

    Grouping was performed based on the following shape parameters: AR, FF, and FA. AR is a ratio of the major axis length to the minor axis length (Eq (2)). A high AR value (e.g., > 2.0) relates to an ellipse shape. When AR approaching 1.0, it indicates a circular shape. FF is a shape descriptor (Eq (3)). It is used to indicate the degree of similarity to a perfect circle [57]. FF value of 1.0 refers to a perfect circle and the object becomes less circular when approaching 0.0 [57]. FA is used to detect a hollow shape (Eq (4)). A high value of FA indicates an object with a cavity inside whereas a small value of FA refers to a solid object. Figure 6 illustrates how FA can be obtained. It is important to remark that in any dataset, the number of mitosis cells is very limited. Hence, it is very challenging to perform empirical analysis using 385 mitosis cells (at a confidence level of 95.0%) for each group. Therefore, the 385 mitosis cells were first manually examined and characterized into LCS, ES, HS, and CS groups. Based on the manual examination, from the 385 mitosis cells, there are 134, 91, 82, and 78 mitosis cells in LCS, ES, HS, and CS groups, respectively. The shape parameters (i.e., AR, FF, and FA) were then extracted from the mitosis cells in each group. Table 6 shows the parameter values of the mitosis cells in each group.

    AR=MajoraxislengthinpixelMinoraxislengthinpixel (2)
    FF=4π∗areaperimeter2 (3)
    FA=x−y (4)
    Figure 6.  Illustration on how to obtain FA.
    DownLoad: Full-Size Img PowerPoint
    Table 6.  Results of empirical analysis for shape parameters on the mitosis cells in each group.
    Groups Shape parameters
    AR FF FA
    Mean Max Min Mean Max Min Mean Max Min
    LCS 1.5 1.9 1.2 0.6 0.9 0.4 - - -
    ES 3.1 4.5 2.6 - - - - - -
    HS - - - - - - 41 57 28
    CS 1.6 1.8 1.4 0.2 0.3 0.1 - - -
     | Show Table
    DownLoad: CSV

    where x denotes the enclosed area of the nucleus and y denotes exact area of the nucleus.

    2.3.3.   Morphological features screening

    Based on the parameter values in Table 6, conditions for each group were set as shown in Table 7. The segmented hyperchromatic nuclei were put into any of these groups based on the conditions. The shape parameters of the segmented hyperchromatic nuclei were first checked to see if they fit the condition for the HS group. This is then followed by checking the conditions for the ES group, LCS group, and CS group. The mitosis candidates in each group then underwent two checking procedures: Checking Procedures 1 and 2. These procedures were used to eliminate the false positive in each group. Checking Procedure 1 was implemented on the LCS and ES groups, whereas Checking Procedure 2 was implemented on the HS and CS groups. In Checking Procedure 1, mitosis candidates with an area smaller than 118 pixels (obtained in Table 5) were recognized as false positive and eliminated from the image. In Checking Procedure 2 (Figure 7), the minor axis length (in pixel) (minoraxis) of each nucleus was obtained and was used to develop a square window, such that window = [minoraxis minoraxis]. The window was then superimposed and placed on the centroid of the respective nucleus. The mean intensity within the window patch was calculated. Checking Procedure 2 eliminates candidates with mean intensity lower than 0.23 (obtained in Table 8). This value is obtained from the empirical analysis of 385 mitosis cells. Table 8 shows the results of empirical analysis in terms of mean intensity calculated using the aforementioned square window.

    Table 7.  Setting conditions of shape parameters for four different shape groups.
    Groups Shape parameters
    AR FF FA
    LCS ≤1.9 ≥0.4 -
    ES > 1.9 - -
    HS - - > 28 pixels
    CS ≤1.9 < 0.4 -
     | Show Table
    DownLoad: CSV
    Figure 7.  Illustration on Checking Procedure 2 for HS and CS groups. The top row images show the nucleus from HS group and the bottom row images show the nucleus from CS group. (a) The sample nucleus, (b) the centroid of the corresponding nucleus is shown by the green dot, (c) a window patch with the size of minor axis length (pixel) of the corresponding nucleus was developed and (d) the mean intensity in the corresponding window is calculated.
    DownLoad: Full-Size Img PowerPoint
    Table 8.  Results of empirical analysis for mean intensity in 385 mitosis cells.
    Feature Parameters
    Mean (± SD) Maximum Minimum
    Mean intensity (calculated using square window, where window=[minoraxisminoraxis]) 0.53 (± 0.113) 0.76 0.23
     | Show Table
    DownLoad: CSV

    2.4.   Features extraction

    Morphological feature describes the form, shape and the structure of the mitosis cell. Morphological features such as eccentricity, extent, perimeter, convex area, area, equivalent diameter, AR, FF, FA, solidity, and ratio of perimeter to area were extracted from the mitosis candidates. Texture features hold a significant characteristic in describing objects or region of interest in an image. The condensation of chromosomes during mitosis causes the cell to show random but consistent texture properties throughout the five phases. Gray-Level Co-occurrence Matrix (GLCM) was applied to examine properties of the mitosis candidates. GLCM with radius, δ = 1 pixel and orientations (θ) of 0°, 45°, 90°, 135° were used. δ of 1 pixel was used as it could provide the best result in examining the texture properties [58]. As the mitosis cell is not an isotropic object (i.e., the directional information of mitosis cell is important), thus, it is essential to obtain GLCM at four different orientations [59]. 11 texture features were computed from the GLCM at four distinct orientations in different channels: entropy, energy, inertia, inverse different moment, correlation, sum average, sum variance, difference average, difference variance, skewness and kurtosis. Seven color channels were used to generate the co-occurrence matrix: red, green, blue, hue, Cr, Cb, and C channels. A total of 319 features (11 morphological features and 308 texture features) were extracted from the mitosis candidates and used as inputs to the SVM classifier.

    2.5.   Classification

    SVM was selected to classify the extracted features [60]. SVM is optimized based on risk minimization [60]. The optimization process is defined by a function in order to find the parameters of a hyperplane that minimizes the risk. For a non-linear case, a kernel-based decision function can be applied [61]. By introducing kernel, SVM will gain its flexibility during the choice in the form of threshold separating the classes. As the kernel implicitly contains a non-linear transformation, no assumption about the functional form of transformation, which makes the data linearly separable, is needed. This study used RBF as the kernel-based decision function [61]. In this study, the parameters used in the SVM classifier were the default settings. Some of the default settings for the SVM classifier can be found in Appendix A (Table A).

    3.   Datasets

    To evaluate the proposed knowledge-assisted detection framework, two datasets were used: a custom dataset and a publicly available dataset (i.e., MITOS dataset [17]). The custom dataset is a self-collected dataset locally in Malaysia. This dataset consists of 48 breast histopathology slides and was obtained from the Pathology Department, Hospital Tuanku Fauziah, Kangar, Perlis, Malaysia. These slides were prepared under a standard staining procedure known as H & E stains. An Aperio CS2 WSI scanner was used to convert the slides into digital form. 20 images were captured at 40x magnification from different dominant areas in each digital slide. The captured image is in 8-bit RGB color with dimensions of 614 × 1264 pixels (size of pixel: 0.2521 µm per pixel) and is in tiff file format. The custom dataset consists of 960 images with a total of 832 mitosis cells. Another dataset, namely MITOS dataset [17] was used in this study. MITOS dataset [17] is a publicly available dataset that is widely used to evaluate the superiority of a mitosis detection framework. Images in MITOS dataset [17] were prepared using the standard H & E stains and were captured at 40 x magnification using Aperio XT WSI scanner, Hamamatsu NanoZoomer WSI scanner, and multi-spectral microscope. In this study, the dataset from the Aperio XT WSI scanner was used for evaluation purposes. The captured image is in 8-bit RGB color with dimensions of 2084 × 2084 pixels (size of pixel: 0.2456 μm per pixel). This dataset consists of 50 images with a total of 327 mitosis cells. Table 9 summarizes the dataset used in this study.

    Table 9.  The summary of dataset used in this study.
    Datasets Number of images Number of mitosis cells Magnification Resolution
    Custom 960 832 40 x 614 × 1264
    MITOS [17] 50 327 40 x 2084 × 2084
     | Show Table
    DownLoad: CSV

    For the custom dataset, the ground truth (i.e., mitosis cells) annotation was performed by an experienced histopathologist from the Hospital Tuanku Fauziah, Kangar, Perlis, Malaysia. Each image was visually examined and the mitosis cells were annotated care-fully in accordance with the NHG system. The main purpose of using the custom dataset is to evaluate the applicability of the proposed knowledge-assisted detection framework and to determine the degree of agreement with respect to the ground truth. The main purpose of using the publicly available dataset (i.e., MITOS dataset [17]) is to perform benchmarking with recent works in order to justify the superiority of the proposed knowledge-assisted detection framework.

    4.   Results and analysis

    To evaluate the applicability of the proposed knowledge-assisted detection framework, performance evaluation was done in four main aspects: initialization method, false positive reduction method, capability in mitosis cells segmentation (after implementation of false positive reduction method), and overall performance in mitosis cells detection. The subsequent sub-sections show the performance evaluation in detail.

    4.1.   Initialization method performance evaluation

    To evaluate the applicability of the improved initialization method, the knowledge-assisted K-Mean method is compared to the baseline initialization method (i.e., random method [62]) and some other popular methods (i.e., Forgy method [63], K-Mean++ [64], and Greedy K-Mean++ [65]). The main purposes of this comparison are to: 1) justify if the knowledge-assisted initialization method is effective to avoid limitations in conventional K-Mean (using random initialization method) by minimizing the possibility of center redundancy, dead center, and the possibility of initial centroid to trap in local minima; 2) justify if the knowledge-assisted initialization method could reduce the iteration numbers (i.e., by reducing the search space) of the K-Mean algorithm. The boxplots in Figure 8 shows the comparison in terms of iteration numbers for different initialization methods using both the dataset. Based on the boxplot, it is found that the knowledge-assisted K-Mean has a smaller search space reflected by lower iteration numbers to compute the final centroids. As compared to other initialization methods, the boxplot of the knowledge-assisted K-Mean has a compact distribution and outperformed the Random method [62], Forgy method [63], and K-Mean++ [64]. When compare to Greedy K-Mean++ [65], the iteration numbers obtained from the knowledge-assisted K-Mean are highly comparative and slightly better by demonstrating lower iteration numbers with lower median value (Greedy K-Mean++ [65] (median): 9 and knowledge-assisted K-Mean (median): 7).

    Figure 8.  Comparison in terms of iteration numbers for different methods using both the datasets.
    DownLoad: Full-Size Img PowerPoint

    4.2.   Mitosis cells segmentation performance evaluation

    The main purpose of the mitosis cells segmentation performance evaluation is to determine the capability of retaining the mitosis cells after implementation of segmentation and false positive reduction stages in a detection framework. For ideal performance, in the segmentation stage, the detection framework aims to segment all the mitosis cells as foreground, whereas, in the false positive reduction stage, the detection framework aims to remove most of the false positive (i.e., non-mitosis cells) and retain all the true positive (i.e., mitosis cells). Therefore, in this study, a high total number of segmented mitosis cells is preferable. Tables 10 and 11 show the comparison in terms of the total number of segmented mitosis cells with different methods after implementation of the respective segmentation and false positive reduction stages using the custom and MITOS [17] datasets, respectively.

    Table 10.  Comparison in terms of the total number of segmented mitosis cells for different methods using custom dataset.
    Methods Total number of segmented mitosis cells Total number of mitosis cells
    Gamma-gaussian mixture model [36] 808 832
    Maximized Inter-Class Weighted Mean [25] 817
    Proposed 825
     | Show Table
    DownLoad: CSV
    Table 11.  Comparison in terms of the total number of segmented mitosis cells for different methods using MITOS dataset [17].
    Methods Total number of segmented mitosis cells Total number of mitosis cells
    Blue ratio [35] 314 327
    Anisotropic Diffusion [66] 314
    Gradient minimization [67] 315
    Gamma-gaussian mixture model [36] 317
    Pixel features [32] 318
    Maximized Inter-Class Weighted Mean [25] 318
    Proposed 321
     | Show Table
    DownLoad: CSV

    Based on the table, it is found that the proposed knowledge-assisted detection framework demonstrates a better performance as the total number of segmented mitosis cells for both the datasets are the highest (i.e., custom dataset: 825/ 832 and MITOIS dataset [17]: 321/ 327). In the proposed knowledge-assisted detection framework, all the mitosis cells were segmented as foreground in the segmentation stage using the knowledge-assisted K-Mean method with a trade-off of a high number of false positive. Nevertheless, it is important to remark that most of the false positive were then removed via the knowledge-assisted false positive reduction stage as in Section 2.3 with minimal loss in mitosis cells.

    4.3.   False positive reduction method performance evaluation

    Based on Tables 10 and 11 in the previous sub-section, generally, the majority of the established works can retain most of the mitosis cells after the segmentation and false positive reduction stages. Nevertheless, as aforementioned, the number of false positive (i.e., non-mitosis cells) versus the number of true positive (i.e., mitosis cells) are commonly in great unbalance, where a high number of false positive would impinge the overall performance of the detection framework (low F1-score). Thus, removing the false positive is essential to improve the performance of mitosis cells detection. To establish a fair com-parison, in this section, the performance evaluation is done by benchmarking with two established works that incorporate different false positive reduction methods (i.e., Gamma-gaussian mixture model [36] and Maximized Inter-Class Weighted Mean [25]) after the respective segmentation stage. The percentage of false positive reduction can be calculated using Eq (5).

    Percentageoffalsepositivecandidatesreduction=i−oix100% (5)

    where i denotes the number of mitosis candidates before the false positive reduction stage and o denotes the number of mitosis candidates after the false positive reduction stage. Figure 9 shows the comparison in terms of percentages in false positive reduction for different methods, where Figure 9 (a) and (b) show the outputs evaluated using the custom and the MITOS [17] datasets, respectively. A high percentage in false positive reduction reflects that the false positive reduction method is capable to remove more false positive (i.e., non-mitosis cells) and is preferable in this study.

    Figure 9.  Comparison in terms of percentage in false positive reduction using (a) custom and (b) MITOS [17] datasets.
    DownLoad: Full-Size Img PowerPoint

    Based on Figure 9, it is evident that the proposed knowledge-assisted false positive reduction method can remove most of the false positive (i.e., 90.0 and 87.1% of the false positive, respectively, for custom and MITOS [17] datasets). For the custom dataset, the percentage obtained by the proposed knowledge-assisted false positive reduction method is 14.1 and 6.6% higher than that of the Gamma-gaussian mixture model [36] and Maximized Inter-Class Weighted Mean [25], respectively, whereas, for the MITOS dataset [17], 19.6 and 7.7% higher than that of the Gamma-gaussian mixture model [36] and Maximized Inter-Class Weighted Mean [25], respectively.

    4.4.   Mitosis cells detection performance evaluation

    As aforementioned, in mitosis cells detection, the number of mitosis cells (true positive) versus the number of non-mitosis cells (true negative) is usually in great unbalance, where mitosis cells are very few as compared to non-mitosis cells in any dataset. Therefore, evaluation parameter such as F1-score is useful where the detection of non-mitosis cells (true negative) is excluded in the calculation. To calculate the F1-score, one must first compute the precision and recall. Equations 6 to 8 show the formula for F1-score, precision, and recall, respectively.

    F1−score=2xPrecisionxRecallPrecision+Recallx100% (6)
    Precision=TPTP+FPx100% (7)
    Recall=TPTP+FNx100% (8)

    where true positive (TP) denotes the mitosis cells that are correctly labeled as mitosis cells, false positive (FP) denotes the non-mitosis cells that are wrongly labeled as mitosis cells; and false negative (FN) denotes the mitosis cells that are wrongly labeled as non-mitosis cells. In this study, F1-score would be used as the primary evaluation parameter. A high percentage in F1-score reflects an accurate detection of mitosis cells and is preferable in this study.

    The mitosis cells classification was done using a 10-fold cross-validation SVM classifier with an RBF kernel. The data were randomly divided into 10 equal pieces. Each selected piece was chosen as a test set with training done on the remaining data. The cross-validation was then repeated 10 times (folds) with each subsample used exactly once as the validation data. All the observations were used for both training and validation, and each observation was used for validation exactly. To demonstrate the capability of the proposed detection framework, the outputs of the proposed detection framework using both datasets are benchmarked to the recent works. For the custom dataset, the comparison of the outputs was done on recent works which incorporate different false positive reduction methods: Gamma-gaussian mixture model [36] and Maximized Inter-Class Weighted Mean methods [25]. The proposed knowledge-assisted detection framework outperformed these recent works by achieving 89.1% in F1-score. For the MITOS dataset [17], the performance comparison is mainly focuses on recent works using handcrafted features-based approach. Nevertheless, considering the growing body of literature in deep learning and fusion (i.e., handcrafted features-based and deep learning) approaches, several recent works in respective approaches are included for benchmarking purposes to justify the applicability of the proposed detection framework. Based on the outputs, the proposed knowledge-assisted detection framework outperformed all the recent works using handcrafted feature-based approach by achieving 88.9% in F1-score. The F1-score of the proposed knowledge-assisted detection method is 0.5% higher than that of the best performing method amongst the handcrafted feature-based method (i.e., Nateghi et al. [25]). When comparing to recent works using deep learning and fusion approaches, the F1-score of the proposed knowledge-assisted detection framework is found promising and outperformed the best-performing methods using deep learning and fusion approaches by 1.1% (i.e., Alom et al. [27]) and 3.1% (i.e., Mahmood et al. [31]), respectively. Figure 10 shows the comparison of the overall outputs in terms of F1-score for different detection frameworks using custom (Figure 10 (a)) and MITOS [17] datasets (Figure 10 (b)).

    Figure 10.  Outputs comparison in terms of F1-score for different detection frameworks using (a) custom and (b) MITOS [17] datasets.
    DownLoad: Full-Size Img PowerPoint

    For both datasets, the classification obtained high TP values. The high TP is reflected in the high FI-score percentages. The high TP perhaps caused by the good performance in the hyperchromatic nucleus segmentation using the improved K-Mean and the knowledge-assisted false positive reduction methods. In the hyperchromatic nucleus segmentation, all the mitosis cells were retained on the images. This demonstrates the ability of the knowledge-assisted K-Mean method in providing a promising segmentation without removing the region of interest or ground truth (i.e., mitosis cells). In the knowledge-assisted false positive reduction stage, for custom and MITOS [17] datasets, only 0.8 (7 mitosis cells) and 2.0% (6 mitosis cells), respectively, of the ground truth were lost. In terms of false positives, this stage successfully removed 90.0 and 87.1% false positives in the custom and MITOS [17] datasets, respectively. The overall classification result demonstrates robustness with an optimal F1-score of 89.1 and 88.9% for custom and MITOS [17] datasets, respectively. Overall, the obtained F1-score was found promising and demonstrated superiority when compared to recent works using handcrafted feature-based, deep learning, and fusion approaches for both datasets. This shows integration between domain knowledge (from human experts) and image processing procedure can help to improve the performance of the mitosis cells detection producing complementary results.

    The low FP and high TP have enhanced the overall F1-score of the proposed knowledge-assisted detection framework. The combination of morphological and texture features has provided better feature discrimination between the mitosis and non-mitosis cells. The computation complexity of the proposed detection framework remains low as the proposed detection framework did not involve complex mathematic equations (as in the deep learning and fusion methods). Domain knowledge was incorporated in the detection framework and was used to guide the image analysis algorithms. The domain knowledge and strategies of the histopathologist were effectively incorporated in the mitosis cells detection framework and were used to formalize the basic building blocks in mitosis cells detection. Based on the results obtained from each stage and the overall performance, the domain knowledge is found prominent and potentially holds a significant role in medical image analysis, especially for mitosis cells detection.

    5.   Conclusions

    In this study, an automated knowledge-assisted mitosis cells detection framework is proposed. The proposed detection framework is developed by incorporating the strategies of the histopathologist and domain knowledge approach which aims to perform recognition of complex and inconsistent mitosis cells in breast histopathology images. The main novelty of the detection framework lies within the simple yet powerful detection capability with knowledge transferred from human experts, producing promising detection results. The contributions of the detection framework are: 1) a knowledge-assisted K-Mean that is able to segment all the mitosis cells from the background and reduce the search space in computing the final centroids, which is reflected with low iteration numbers; 2) a knowledge-assisted false positive reduction method which capable to remove most of the false positive (i.e., non-mitosis cells). For both the datasets, the knowledge-assisted K-Mean is found able to segment all the mitosis cells with a trade-off of high false positive numbers. The knowledge-assisted false positive reduction method is then applied to remove most of the false positive (i.e., 90.0 and 87.1% in the custom and MITOS [17] datasets, respectively); 3) the proposed framework achieves a high percentage in F1-score (i.e., 89.1%) when tested using the custom dataset which reflects a good agreement with respect to ground truth demarcated by the histopathologist, 4) the proposed framework achieves a high percentage in F1-score (i.e., 88.9%) when tested using the MITOS dataset [17] and outperformed the recent works with handcrafted feature-based, deep learning, and fusion approaches; 5) to the best of our knowledge, mitosis cells detection using a knowledge-assisted detection framework by incorporating domain knowledge as well as knowledge transfer from human experts is yet to be available in the literature. The proposed knowledge-assisted K-Mean and false positive reduction method are found capable to improve the performance of the mitosis cells detection producing complementary results. This is justifiable as the proposed knowledge-assisted detection framework obtains high F1-scores when the proposed detection framework is evaluated using two different datasets. Additionally, when benchmarking with recent works using handcrafted feature-based, deep learning, and fusion approaches, the F1-scores of the proposed detection framework are found promising and outperformed the recent works. For future work, the proposed framework is intended to test using other publicly available datasets (e.g., AMIDA [18], MITOS-ATYPIA [19], and TUPAC [20]) to further justify the applicability and capability of the proposed framework.

    Acknowledgments

    The authors gratefully acknowledge the financial support from the Fundamental Research Grant Scheme (FRGS) under a grant number of FRGS/1/2016/SKK06/UNIMAP/02/3 from the Ministry of Higher Education Malaysia. The protocol of this study had been approved by Medical Research and Committee of National Medical Research Register (NMRR) Malaysia referring to the protocol number: NMRR-17-281-34236.

    Conflict of interest

    The authors declare no conflict of interest.

    Appendix A

    Table A shows some of the default settings for the SVM classifier used in this study.

    Table A.  Settings of the SVM classifier (default settings).
    Parameters Settings
    Box constraint 1
    Kernel function RBF kernel
    Kernel scale 1
    Flag to standardize predictor data False
    Solver (optimization routine) Iterative Single Data Algorithm (ISDA)
    Flag to clip alpha coefficients True
     | Show Table
    DownLoad: CSV


    [1] L. Tong, Introduction to railways, China Railway Publishing House, (2016), 37–45.
    [2] A. Singhal, Introducing the knowledge graph: Things, not strings, 2012. Available from: https://blog.google/products/search/introducing-knowledge-graph-things-not/
    [3] H. Sun, X. Li, Named entity recognition for power distribution network data, Comput. Syst. Appl., 32 (2023), 387–393. https://doi.org/10.1145/1390156.1390177 doi: 10.1145/1390156.1390177
    [4] Y. Chen, Q. Dai, J. Liu, Named entity recognition of railway accident texts with character position features, Comput. Syst. Appl., 31 (2022), 211–219. https://doi.org/10.15888/j.cnki.csa.008860 doi: 10.15888/j.cnki.csa.008860
    [5] J. Yang, Y. Zhang, L. Li, X. Li, F. Liu, T. Solorio, YEDDA: A lightweight collaborative text span annotation tool, In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, (2018), 31–36.
    [6] H. Wang, G. Qi, H. Chen, Knowledge graph methods, practices and applications, Publish. House Electron. Industry, (2019), 137–142.
    [7] H. Yu, H. Zhang, Q. Liu, X. Lv, S. Shi, Chinese named entity identification using cascaded hidden Markov model, J. Commun., 27 (2006), 87–94. https://doi.org/10.3321/j.issn:1000-436X.2006.02.013 doi: 10.3321/j.issn:1000-436X.2006.02.013
    [8] Y. Zhang, Z. Xu, X. Xue, Fusion of multiple features for Chinese named entity recognition based on Maximum Entropy Model, J. Comput. Res. Dev., 45 (2008), 1004–1010. https://doi.org/CNKI:SUN:JFYZ.0.2008-06-013
    [9] L. Wu, L. Liu, H. Li, Y. Gao, A Chinese toponym recognition method based on Conditional Random Field, Geomat. Info. Sci. Wuhan. Univ., 42 (2017), 150–156. https://doi.org/10.13203/j.whugis20141009 doi: 10.13203/j.whugis20141009
    [10] Y. He, F. Du, Y. Shi, L. Song, Survey of named entity recognition based on deep learning, Comput. Eng. Appl., 57 (2021), 21–36. https://doi.org/10.3778/j.issn.1002-8331.2012-0170 doi: 10.3778/j.issn.1002-8331.2012-0170
    [11] Z. Huang, K. Xu, K. Yu, Bidirectional LSTM-CRF models for sequence tagging, 2015. Available from: https://doi.org/10.48550/arXiv.1508.01991
    [12] L. Yang, Research of railway fault accident text big data mining key technologies and application, Ph.D thesis, China Academy of Railway Sciences in Beijing, 2018. https://doi.org/CNKI: CDMD: 1.1018.130739
    [13] T. Mikolov, K. Chen, G. Corrado, J. Dean, Efficient estimation of word representations in vector space, In Proceedings of the 1th International Conference on Learning Representations, (2013).
    [14] J. Kong, L. Zhang, M. Jiang, T. Liu, Incorporating multi-level CNN and attention mechanism for Chinese clinical named entity recognition, J. Biomed. Inform., 116 (2021), 103737. https://doi.org/10.1016/j.jbi.2021.103737 doi: 10.1016/j.jbi.2021.103737
    [15] C, Li, K. M, Entity recognition of Chinese medical text based on multi-head self-attention combined with BILSTM-CRF, Math. Biosci. Eng., 19 (2022), 2206–2218. https://doi.org/10.3934/mbe.2022103 doi: 10.3934/mbe.2022103
    [16] J. Devlin, M. W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1 (2019), 4171–4186. https://doi.org/10.18653/v1/N19-1423
    [17] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, Language models are unsupervised multitask learners, 2019. Available from: https://insightcivic.s3.us-east-1.amazonaws.com/language-models.pdf
    [18] T. Yuan, X. Qin, C. Wei, A Chinese named entity recognition method based on ERNIE-BiLSTM-CRF for food safety domain, Appl. Sci., 13 (2023), 2849. https://doi.org/10.3390/app13052849 doi: 10.3390/app13052849
    [19] Q. An, B. Pan, Z. Liu, S. Du, Y. Cui, Chinese named entity recognition in football based on ALBERT-BiLSTM Model, Appl. Sci., 13 (2023), 10814. https://doi.org/10.3390/app131910814 doi: 10.3390/app131910814
    [20] Z. Guo, X. Deng, Intelligent identification method of legal case entity based on BERT-BiLSTM-CRF, J. Beijing. Univ. Posts. Telecom., 44 (2021), 129–134. https://doi.org/10.13190/j.jbupt.2020-241 doi: 10.13190/j.jbupt.2020-241
    [21] L. Li, H. Zhou, X. Guo, C. Liu, J. Su, Z. Tang, Named entity recognition of diseases and insect pests based on multi source information fusion, Trans. Chin. Soc. Agric. Mach., 52 (2021), 253–263. https://doi.org/10.6041/j.issn.1000-1298.2021.12.027 doi: 10.6041/j.issn.1000-1298.2021.12.027
    [22] H. Lin, W. Bai, R. Lu, R. Lu, Z. Zhao, X. Li, Named entity recognition of fault information of high-speed railway turnout from BMBC model, J. Railw. Sci. Eng., 20 (2023), 1149–1159. https://doi.org/10.19713/j.cnki.43-1423/u.t20220637 doi: 10.19713/j.cnki.43-1423/u.t20220637
    [23] X. Ma, E. Hovy, End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF, In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 1 (2016), 1064–1074. https://doi.org/10.18653/v1/P16-1101
    [24] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, et al., RoBERTa: A robustly optimized BERT pretraining approach, In Proceedings of the 8th International Conference on Learning Representations, (2020), 26–30.
    [25] Y. Zhang, Y. Wang, B. Li, Identifying named entities of Chinese electronic medical records based on RoBERTa-wwm dynamic fusion model, Data. Anal. Knowl. Discov., 6 (2022), 242–250. https://doi.org/10.11925/infotech.2096-3467.2021.0951 doi: 10.11925/infotech.2096-3467.2021.0951
    [26] X. Li, T. Shi, P. Li, M. Dai, X. Zhang, Research on knowledge extraction method for High-speed railway signal equipment fault based on text, J. Chin. Rail. Soc., 43 (2021), 92–100. https://doi.org/10. 3969/j.issn.1001-8360.2021.03.012 doi: 10.3969/j.issn.1001-8360.2021.03.012
    [27] J. Tian, H. Song, L. Chen, G. Sheng, X. Jiang, Entity recognition approach of equipment failure text for knowledge graph construction, Power. Syst. Technol., 46 (2022), 3913–3922. https://doi.org/10.13335/j.1000-3673.pst.2021.1886 doi: 10.13335/j.1000-3673.pst.2021.1886
    [28] J. Liu, H. Yang, Z. Sun, H. Yang, L. Shao, H. Yu, et al., Named entity recognition for the diagnosis and treatment of aquatic animal diseases using knowledge graph construction, Trans. Chin. Soc. Agric. Eng., 38 (2022), 210–217. https://doi.org/10.11975/j.issn.1002-6819.2022.07.023 doi: 10.11975/j.issn.1002-6819.2022.07.023
    [29] J. Yu, W. Zhu, L. Liao, Entity recognition of support policy text based on RoBERTa-wwm-BiLSTM-CRF, Comput. Eng. Sci., 45 (2023), 1498–1507. https://doi.org/10.3969/j.issn.1007-130X.2023.08.019 doi: 10.3969/j.issn.1007-130X.2023.08.019
    [30] J. Lin, E. Liu, Research on named entity recognition method of metro on-board equipment based on multiheaded self-attention mechanism and CNN-BiLSTM-CRF, Comput. Intell. Neurosci., 2022 (2022), 1687–5273. https://doi.org/10.1155/2022/6374988 doi: 10.1155/2022/6374988
    [31] S. Cheng, I. C. Prentice, Y. Huang, Y. Jin, Y. Guo, R. Arcucci, Data-driven surrogate model with latent data assimilation: Application to wildfire forecasting, J. Comput. Phys., 464 (2022). https://doi.org/10.1016/j.jcp.2022.111302 doi: 10.1016/j.jcp.2022.111302
    [32] Y. Zhang, S. Cheng, N. Kovalchuk, M. Simmons, O. K. Matar, Y. Guo, et al., Ensemble latent assimilation with deep learning surrogate model: Application to drop interaction in a microfluidics device, Lab Chip, 22 (2022), 3187–3202. https://doi.org/10.1039/D2LC00303A doi: 10.1039/D2LC00303A
    [33] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, et al., Attention is all you need, In Proceedings of the 31st International Conference on Neural Information Processing Systems, (2017), 6000–6010.

  • This article has been cited by:

    1. Teoh Leong Hoe, Tan Xiao Jian, Khairul Shakir Ab Rahman, Lu Juei Min, Quah Yi Hang, Wong Chung Yee, Thien Yee Von, Lim Chee Chin, Teoh Chai Ling, 2022, Nuclei Segmentation in Breast Histopathology Images using FCM, 978-1-6654-9581-3, 142, 10.1109/I2CACIS54679.2022.9815480
    2. Wong Chung Yee, Tan Xiao Jian, Khairul Shakir Ab Rahman, Teoh Leong Hoe, Lu Juei Min, Quah Yi Hang, Teoh Chai Ling, 2022, Performance Analysis of Color Normalization Methods in Histopathology Images, 978-1-6654-9581-3, 147, 10.1109/I2CACIS54679.2022.9815475
    3. Jiquan Li, Laixiang Qiu, Yujun Yang, Wang Zhou, 2022, Deep Learning Techniques for Breast Cancer Mitotic Cell Detection, 978-1-6654-9387-1, 1, 10.1109/ICCWAMTIP56608.2022.10016492
    4. Lu Juei Min, Tan Xiao Jian, Khairul Shakir Ab Rahman, Teoh Leong Hoe, Quah Yi Hang, Wong Chung Yee, Yip Sook Yee, Wan Zuki Azman Wan Muhamad, Teoh Chai Ling, 2022, A Non-Mitosis Reduction Method using Semantic Descriptors for Breast Cancer Mitosis Detection Application, 978-1-6654-9581-3, 131, 10.1109/I2CACIS54679.2022.9815478
    5. Rayed AlGhamdi, Mitotic Nuclei Segmentation and Classification Using Chaotic Butterfly Optimization Algorithm with Deep Learning on Histopathology Images, 2023, 8, 2313-7673, 474, 10.3390/biomimetics8060474
    6. Chai Ling Teoh, Xiao Jian Tan, Khairul Shakir Ab Rahman, Ikmal Hisyam Bakrin, Kam Meng Goh, Joseph Jiun Wen Siet, Wan Zuki Azman Wan Muhamad, A Quantitative Measurement Method for Nuclear-Pleomorphism Scoring in Breast Cancer, 2024, 14, 2075-4418, 2045, 10.3390/diagnostics14182045
    7. Evgin Goceri, Nuclei segmentation using attention aware and adversarial networks, 2024, 579, 09252312, 127445, 10.1016/j.neucom.2024.127445
    8. Joseph Jiun Wen Siet, Xiao Jian Tan, Wai Loon Cheor, Khairul Shakir Ab Rahman, Ee Meng Cheng, Wan Zuki Azman Wan Muhamad, Sook Yee Yip, A comprehensive review of tubule formation in histopathology images: advancement in tubule and tumor detection techniques, 2024, 57, 1573-7462, 10.1007/s10462-024-10887-z
    9. Xiao Jian Tan, Wai Loon Cheor, Ee Meng Cheng, Khairul Shakir Ab Rahman, Wan Zuki Azman Wan Muhamad, Wai Zhe Leow, Breast cancer status, grading system, etiology, and challenges in Asia: an updated review, 2023, 25, 1765-2839, 99, 10.1515/oncologie-2022-1011
    10. Amal Alshardan, Nazir Ahmad, Achraf Ben Miled, Asma Alshuhail, Yazeed Alzahrani, Ahmed Mahmud, Transferable deep learning with coati optimization algorithm based mitotic nuclei segmentation and classification model, 2024, 14, 2045-2322, 10.1038/s41598-024-80002-3
    11. Joseph Jiun Wen Siet, Xiao Jian Tan, Wai Loon Cheor, Khairul Shakir Ab Rahman, Yee Von Thien, 2024, Clustering-Based Image Registration of Tubule Candidates in Breast Histopathology Images, 979-8-3315-3047-1, 231, 10.1109/MCSoC64144.2024.00046
    12. Evgin Goceri, A convolution and transformer-based method with effective stain normalization for breast cancer detection from whole slide images, 2025, 110, 17468094, 108138, 10.1016/j.bspc.2025.108138
  • Reader Comments
  • © 2024 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)
通讯作者: 陈斌, bchen63@163.com
  • 1. 

    沈阳化工大学材料科学与工程学院 沈阳 110142

  1. 本站搜索
  2. 百度学术搜索
  3. 万方数据库搜索
  4. CNKI搜索
Mathematical Biosciences and Engineering
4.4

Metrics

Article views(1777) PDF downloads(83) Cited by(1)

Preview PDF
Download XML
Export Citation
Article outline
Abstract
   Introduction
   Methods
   Datasets
   Results and analysis
   Conclusions
Acknowledgments
Conflict of interest
Appendix A
References

Figures and Tables

Figures(7)  /  Tables(12)

Other Articles By Authors

Related pages

Tools

Copyright © AIMS Press

/

DownLoad:  Full-Size Img  PowerPoint
Return
Return

Catalog