Lactic acid bacteria (LAB) are members of an heterogenous group of bacteria which plays a significant role in a variety of fermentation processes. The general description of the bacteria included in the group is gram-positive, non-sporing, non-respiring cocci or rods. An overview of the genetics of lactococci, Streptococcus thermophilus, lactobacilli, pediococci, leuconostocs, enterococci and oenococciis presented with special reference to their metabolic traits. The three main pathways in which LAB are involved in the manufacture of fermented foods and the development of their flavour, are (a) glycolysis (fermentation of sugars), (b) lipolysis (degradation of fat) and (c) proteolysis (degradation of proteins). Although the major metabolic action is the production of lactic acid from the fermentation of carbohydrates, that is, the acidification of the food, LAB are involved in the production of many beneficial compounds such as organic acids, polyols, exopolysaccharides and antimicrobial compounds, and thus have a great number of applications in the food industry (i.e. starter cultures). With the advances in the genetics, molecular biology, physiology, and biochemistry and the reveal and publication of the complete genome sequence of a great number of LAB, new insights and applications for these bacteria have appeared and a variety of commercial starter, functional, bio-protective and probiotic cultures with desirable properties have marketed.
1.
Introduction
Mastitis is the most common infectious disease in dairy cows, causing huge economic losses to dairy farmers and milk processing companies [1,2]. The cells found in healthy milk comprise 75%–85% of white blood cells (macrophages, neutrophils, and lymphocytes) and 15%–25% of epithelial cells. During mastitis, the count of white blood cells can reach 99%, a useful biomarker to indicate infection [3,4]. Yet, the severity of cow mastitis infection and the counts of various cells may sometimes differ [3,4]. Somatic cell count (SCC) is a key method used in the dairy industry to determine the quality of cattle and bovine milk [5]. SCC measurement can be divided into direct and indirect methods. A direct approach, which relies on manual counting under the microscope or calculating and analyzing the number of somatic cells using computerized image processing techniques from the microscopic images of breast samples, is considered the most accurate method. However, the manual counting method is susceptible to the subjective factors of the experimenter, is time-consuming and less automated, and may lead to counting errors [6,7]. Indirect methods include the California cell assay and the Wisconsin mastitis test [8]. The most popular indirect approaches are cell analyzers (e.g., MoFlo XDP ultra-fast flow cytometer from Beckman Coulter (USA), Nucleo Counter NC-3000 cell analyzer from ChemoMetecA/S (Denmark), and Fossomatic™ 7 DC somatic cell analyzer from FOSS (Denmark)); yet, these methods are expensive and subject to slightly less correlated results and have poor accuracy.
Image processing technology has been increasingly used for classifying and identifying different cell types, mainly by collecting the color images of stained cells using a microscope [9]. Conventional recognition methods are mainly based on the following steps: obtaining the color images of cells through machine vision technology, extracting the cell features, and building models for cell classification and recognition using related machine learning methods [9]. Khan et al. [10] classified malignant and benign cells using a support vector machine (SVM) by extracting the texture information from breast cells. Okmen et al. [11] used the k-Nearest Neighbor (KNN) method to classify renal tumor cells, showing an accuracy of 83.8%. Moreover, Mishra et al. [12] extracted multiple features and classified lymphocytes using the random forest classifier. Gao et al. [13] proposed a rapid and accurate method for detecting mastitis in cattle by using the two-way two-dimensional principal component analysis. In addition, Gao et al. [14,15] proposed a Relief-F algorithm to extract the features of milk somatic cells for classification. Besides, Zhang et al. [16] proposed a recognition algorithm based on random forest, with an accuracy of 96%. Machine learning is also used for microorganism identification and classification [17,18,19,20].
Furthermore, over the last few years, novel image processing-based cell counting methods have been proposed; they combine the diagnostic experience of pathologists with the advantages of rapid processing and accurate calculation of computers to achieve the counting and classification of somatic cells, reducing the influence of subjective factors, and providing a convenient and accurate detection method. Liang et al. [21] proposed a set containing convolutional neural network (CNN) (using Xception) and recurrent neural network (RNN) (using long and short-term memory (LSTM)) features, with a cell classification accuracy of 90.79%. Moreover, Bani-Hani et al. [22] proposed applying CNN and genetic algorithm (GA) for image classification of four types of blood cells, showing an accuracy of 91.00%. Habibzadeh et al. [23] used a pre-trained model, ResNet and InceptionNet, to identify white blood cells. Acevedo et al. [24] upgraded the structure of the Inception-v3 network model to classify and recognize eight types of blood cell images, reaching an accuracy of 90%. Malkawi et al. [25] extracted the features of leukocytes through the visual geometry group (VGG) network and then adopted the SVM model for classification and recognition, reaching an accuracy of 98.7%. Ghosh & Kundu [26] combined RNN and CNN to establish a multilayer network model and classify different blood cells, achieving an accuracy of 87.29%. AlexNet is the name of a convolutional neural network (CNN) architecture; AlexNet contains eight layers; the first five are convolutional layers, some of them followed by max-pooling layers, and the last three are fully connected layers [27]. MobileNetV2 is a convolutional neural network that is 53 layers deep and is a very effective feature extractor for object detection and segmentation [28].
Over recent years, an increasing number of studies started employing deep learning methods for microorganism detection [17,18,19]. The recognition model constructed by the deep learning algorithms is portable and has the advantages of automatic learning and extracting shallow and deep features [29]. Therefore, research and construction of a recognition model based on deep learning can reduce the difficulty of manual feature selection and present good mobility, suitable for milk somatic cell images with complex backgrounds [30]. In this study, we introduced PolyLoss and product of cross-attention matrices (PCAM)-ResNet50, a model based on the improved structure of ResNet50, and applied it to bovine milk somatic cell datasets to solve the problem of multiple classifications in which the dataset samples themselves are difficult to obtain, and the data are unevenly distributed. In consideration of the characteristics of milk cells, traditional data enhancement methods were used to expand the data set, increase its diversity, and reduce its overfitting. The PCAM module was introduced into the algorithm to make the network learn with the target, while PolyLoss was selected as the loss function to solve the difficult acquisition of samples and imbalance classification to improve the detection and classification accuracy. This new, improved method (i.e., PCAM-ResNet50 combined with PolyLoss) introduces a position attention module and a channel attention module into the classical ResNet50 network structure, respectively, which allows for more efficient and accurate extraction of cell features. Hence, the identification of the milk cells could be more accurate, providing more accurate identification of milk quality.
The workflow of the proposed method in this paper is shown in Figure 1.
The main contributions of this paper are as follows. 1) To our knowledge, this is the first study that uses deep learning techniques to classify images of bovine milk somatic cells. 2) This study proposes a new method (PCAM-ResNet50) that introduces a position attention module and a channel attention module into the classical ResNet50 network structure, respectively, to make it more effective in extracting the features of various types of cells. 3) The loss function of the original ResNet50 structure is improved, and PolyLoss was adopted to compensate for the poor classification effect caused by the lack of sample size of bovine milk cells and the unbalanced data set. 4) The performance of the proposed method is better than other methods, and the classification accuracy reaches 97%.
2.
Materials and methods
2.1. ResNet model
In Figure 2, Relu is input as the activation function, and the feature image is x. First, the feature image is reduced by checking the convolution of 1 × 1, doing a 3 × 3 convolution operation, and restoring the dimension by convolution 1 × 1. The solid line in Figure 2(a) and the dashed line in Figure 2(b) represent the input downsampling function. Figure 2(a), (b) illustrate the residual block without changing the size and that by adding scale, respectively. The output feature image is half of the original height and width. After such a jump connection, the shallow features are identically mapped to the deep layer. The deep gradient can be directly transmitted back to the shallow layer during the back propagation, which solves network degradation and improves performance.
2.2. Loss function
The loss function here refers to the difference between the predicted value and the true value of the output after the training data are input into the model. In the classical ResNet50 network, the cross-entropy of each sample is averagely weighted to obtain the final value of the loss function, shown in Eq (1):
where N and m present the number of samples and categories, respectively, and $ {y}_{i} $ and $ {\stackrel{\wedge }{y}}_{i} $ stand for the true value and the predicted value of class i, respectively.
2.3. Model refinement
2.3.1. Attention mechanism
Milk has large amounts of fat, protein, and cellular debris, regarded as image backgrounds [32]. When cells are stained, the contrast between cells and the background is affected due to uneven illumination. The attention mechanism [33] was introduced in the residual model to reduce the influences of the above factors and make the model more targeted, improving recognition accuracy. The attention mechanism makes the model focus on the important information in the image [34]. The position attention mechanism (PAM) refers to the spatial dependence on different positions on the feature map [35]. The channel attention mechanism (CAM) and soft attention were used here. At a particular position, the feature is updated by a weighted sum at all positions, where the weights are determined by the feature similarity between the two positions. PAM encodes a wide range of contextual information into local features to improve its representation power. Figure 3 illustrates the structure of PAM.
The calculation process of Figure 3 was as follows:
Ⅰ. Three feature maps $ \left\{B, C, D\right\}\in {R}^{C\times H\times W} $ were obtained after passing the input feature map $ A\in {R}^{C\times H\times W} $ through three convolutional layers.
Ⅱ. The transposes of the matrices C and B were multiplied to obtain the similarity matrix of the pixels within A, and the spatial attention map $ S\in {R}^{N\times N} $ was obtained by Softmax with the following Eq (2).
where $ N = H\times W $ represents the number of pixels in the spatial range, and Sji is the relationship of the i th and j th positions. The more similar the characteristic representations of the two positions are, the more relevant they are.
Ⅲ. Matrix multiplication was performed by transposing matrices S and D, and the result was restored to the original input feature size $ {R}^{C\times H\times W} $. The final output result $ E\in {R}^{C\times H\times W} $ was obtained by summing it with the original features element by element through the scale coefficient α, as shown in Eq (3).
where α is initialized to 0 and gradually learns to assign more weights, Di and Aj are the element of D and A, respectively. The above equations suggest that the feature E obtained at each position was the weighted sum of the features with the original features at all positions.
For the CAM [35], advanced features of each channel could be regarded as category-specific information extraction of the dependency in the channel dimension. All channel features were weighted, and each channel feature was updated to improve classification accuracy (Figure 4). In this paper, the correlation of channels was modeled by using the spatial information of all relevant positions.
As demonstrated in Figure 4, the channel attention map $ A\in {R}^{C\times H\times W} $ was calculated directly from the input feature map $ X\in {R}^{C\times C} $, as follows:
1. The input feature graph $ A\in {R}^{C\times H\times W} $ was performed the reshape operation, which becomes$ {R}^{C\times N} $, where $ N = H\times W $.
2. The transition matrixes of $ {R}^{C\times N} $ and $ {R}^{C\times N} $ were multiplied to obtain the channel attention matrix $ X\in {R}^{C\times C} $ by Softmax, as follows:
where, xji represents the effect of the i th channel on the j th channel.
3. The matrix $ {R}^{C\times N} $ was obtained by multiplying the matrix X with A. After it was multiplied by the scale factor β, the element-level summing operation with A was performed to obtain the final output result $ E\in {R}^{C\times H\times N} $. The specific calculation is shown in Eq (5):
2.3.2. The PolyLoss function
Lin et al. [36] proposed to adopt the focus loss function to avoid the overfitting caused by the cross-entropy loss function, but the loss function is not the most effective on some unbalanced data sets. The PolyLoss function [37] was selected for the improved ResNet50 model in this paper to solve the poor classification effect caused by insufficient cells and class imbalance. PolyLoss was used to decompose the commonly used classification loss functions (such as cross-entropy loss function and focal loss function) into a series of weighted polynomial bases through Taylor expansion, as shown in Eq (6):
where $ {\epsilon }_{j} > 0 $ is the polynomial coefficient, $ \gamma \in [-1/\gamma , \infty ] $ $ j\in [-1/j, \infty ] $ is the perturbation term, and Pt represents the probability of the target-label prediction.
2.3.3. Resedual block model based on PCAM-ResNet50
The deep residual network model used in this paper was based on the classical ResNet50 model (Figure 5(a)) realized by integrating the PAM and CAM, forming a PCAM-ResNet50 model, as shown in Figure 5(b). The figure reveals that the input image sizes of the two model structures were the same, which was a color milk somatic cell image in 224 × 224 × 3. Moreover, the model structure included the conv layer, bn layer, maxpool layer, avgpool layer, fc layer, and Softmax classification layer. Where Conv1 was the convolution layer, Conv2_x (x = 1, 2, 3, 4) was the residual block with scale added, and there was only one residual block. Besides, Ident1_x (x = 1, 2, 3, 4) presented the residual block without changing the size, with 2, 3, 5, and 2 residual blocks, respectively. The improved part is marked in green in Figure 5(b). The PAM and CAM were added after the Max pooling layer and the last residual block, respectively. It aimed to ensure that the effective milk somatic cell image information could be transmitted to the deep layer, and the feature weight of the cell area was increased. The interference information, such as contrast and brightness, was reduced, and the impurity debris in the image background was filtered out to improve the accuracy of automatic recognition.
The network structure contained five convolutional blocks. The first was a 7 × 7 convolutional layer; the remaining four were residual blocks. Each residual block comprised three convolutional layers, with 1 + 3 × (3 + 4 + 6 + 3) = 49 convolutional layers. The specific parameter settings for the PCAM-ResNet50 are listed in Table 1.
The flow of the image classification and recognition method under the improved ResNet 50 model is shown in Figure 6.
3.
Results and discussion
3.1. Model training environment
This experiment was carried out using the Windows 10 operating system (central processing unit: Intel (R) Core (TM) i5-12400F CPU 2.50 GHz, running memory of 32 G; graphics card: NVIDIA GeForce RTX 2060). The open-source framework for Pytorch deep learning was used to build the deep neural network. CUDA11.6 was used to accelerate the network training. The programming was realized with Pycharm and Python3.8.5.
3.2. Experimental data set and preprocessing
The dataset used in this experiment was obtained from the Basic Veterinary Laboratory of Inner Mongolia Agricultural University. It was magnified 400× using an OLYMPUS BX51 optical microscope, with a resolution of 2048 pixels × 1536 pixels, and it contained 57 original large-field images (as shown in Figure 7), which were segmented using the Kmeans algorithm [38]. After extraction, a pathologist screened and manually classified the cells into four types. There were 168 macrophages (MΦ), 68 epithelial cells (EPI), 1209 neutrophils (NG), and 153 lymphocytes (LYP), corresponding to the numbers 1, 2, 3, and 4 in Figure 7, respectively. A total of 1598 images were initially analyzed. In order to enhance the generalization ability of the model and reduce overfitting, the sample size was expanded by random rotation, horizontal flip, vertical flip, and random grayscale. Hence, 7990 images were finally obtained and randomly divided into a training set and a test set using a ratio of 8:2. The data samples are shown in Figure 8.
3.3. Experimental parameter setting
In order to ensure the model was in the optimal state, the parameters were set after several experimental verifications as follows: the iteration times of the network training epoch were set to 200, the batch size was set to 16, and the Adam optimizer [39] was adopted (its initial learning rate was 0.001). However, the Warm-up learning rate was adopted to optimize the Adam algorithm. The Warm-up stage of network training was performed by gradually increasing the low learning rate. The initial learning rate was used for training when the model was relatively stable. At this time, the learning rate gradually decreased, accelerating the convergence and improving the effect.
3.4. Model evaluation method
The classification results were shown through the confusion matrix to evaluate the effectiveness of the identification method in this paper. The accuracy (A), accuracy (P), recall rate (R), harmonic average (F1) of recall rate, and macro average F1 (F1-macro) were adopted for a comprehensive evaluation. They could be calculated with the following equations:
In the above equations, TP represents the true type of cell sample, i.e., the number identified by the model as the correct type; FN indicates that the cell sample was the true type, i.e., the number of types identified as wrong by the model; FP represents the wrong type of cell sample, i.e., the number of the correct type in the recognition model; TN is the cell sample is the error type, i.e., the recognition model is also the number of error types. In addition, n refers to the number of cell types.
A relational diagram of evaluation indicators was constructed (Figure 9) to elucidate the relationship among TP, FN, FP, and TN, where P and N represent the correct and wrong samples, respectively.
In the experiment, the receiver operating characteristic (ROC) curve was used to evaluate the model performance. The abscissa is the false positive rate (FPR), i.e., the proportion of all the predicted samples of the correct type but the wrong type in all the wrong samples, as shown in Eq (12). The ordinate marks the true positive rate (TPR), i.e., the proportion of all the predicted samples of the correct type in all the samples of the correct type, as shown in Eq (13). The effect of the model was evaluated by calculating the area under the ROC curve (AUC) and the coordinate axis. The larger the value, the better the classification effect of the model.
3.5. Result analysis
3.5.1. Loss function validity analysis
In order to verify the effectiveness of the Polyloss function for the classification of unbalanced milk somatic cell samples, the model proposed in this paper was compared with the classic ResNet50. The experimental results are shown in Table 2. The macro average F1 value of the proposed algorithm was 8% higher than that of the classic ResNet50 model, indicating that the model pays more attention to the sample imbalance after introducing the Polyloss function, thus improving the generalization ability of the model.
Meanwhile, the loss functions of the new model, ResNet18, ResNet34, and ResNet50 model training, were drawn, as shown in Figure 10. With the increase in training times, the loss value of the model in this paper greatly decreased, so the model was relatively stable. During training, the loss value of the model in this paper was 1/7 that of other methods. Besides, the curve tends to converge as the number of iterations increases.
3.5.2. Analysis of ablation experiment results
The ResNet50 model, the ResNet50 model using the PolyLoss function, and the new model were applied to the test set to verify the overall impacts of each module. Table 3 shows that the accuracy of the ResNet50 network was 93%. It decreased by 1% after the loss function was replaced with the PolyLoss, while it increased by 4% after applying the proposed algorithm. Such data suggest that the new method can direct the network to focus more on the useful feature information in various cell images. Thus, the network can enhance the expression ability of milk somatic cell features and more accurately classify all types of milk cells.
3.5.3. Comparison of overall recognition effect of models
The overall recognition effect of the proposed algorithm was compared to ResNet18, ResNet34, ResNet50, AlexNet [27], and MobileNetV2 [28] models to verify the superiority of the proposed algorithm. As shown in Table 4, the accuracy, precision rate, recall rate, and F1 value of the new proposed algorithm were 97%, 94.5%, 90.75%, and 92.25%, respectively, and were higher than those of the other five methods, indicating that the proposed algorithm is more suitable for milk somatic cell image recognition. Meanwhile, the ROC curves of the six recognition models generated are shown in Figure 11.
3.5.4. The recognition effect of different models on different types of cells.
In order to verify the effectiveness of the proposed algorithm for milk somatic cell classification, it was compared and evaluated with the other three models in terms of precision rate, recall rate, and F1 comprehensive index. The comparison results are shown in Tables 5–7. In addition, after 1598 samples are tested, the confusion matrix of the four types of cells is shown in Figure 12.
4.
Conclusions
The new method (i.e., PCAM-ResNet50) introduces a position attention module and a channel attention module into the classical ResNet50 network structure, respectively, making the older tool more effective in extracting the features of various types of cells. In addition, the loss function of the original ResNet50 structure was improved, and the polynomial-based combination loss function (PolyLoss) was adopted to make up for the poor classification effect caused by the lack of sample size of bovine milk cells and the unbalanced data set. The performance of the new method was superior to other methods, and the classification accuracy reached 97%. Hence, the identification of the milk cells could be more accurate, providing more accurate identification of milk quality. The proposed method in this paper also has the potential to be used in some other research fields about AI, such as image segmentation [40], medical image analysis [41,42], feature extraction [43], and video analysis [44].
Acknowledgments
This study was supported by the Inner Mongolia Autonomous Region Higher Education Scientific Research Project (#NJZY21486), the National Natural Science Foundation of China (#31960494), the Inner Mongolia Autonomous Region Science and Technology Project (#2020GG0169), and the Inner Mongolia Agricultural University Basic Subject Scientific Research Funding Project (#JC2018001).
Conflict of interest
The authors declare there is no conflict of interest.
DownLoad:
