Infusion port level detection for intravenous infusion based on Yolo v3 neural network

1.   Introduction

Intravenous infusion is a common treatment method in the medical process ^[1,2,3]. It is widely used because of the fast infusion speed and good treatment effect. At present, medical technology continues to improve, and intravenous infusion is used to deliver nutrient solutions to the human body to improve resistance. In medical infusion, the problem of not being able to check the liquid level of the infusion bottle in time has always been a headache for patients and nursing staff ^[4]. If it is not handled in time, the infusion bottle will be empty, and the blood may return. If the situation is light, the patient will have swelling and pain at the injection site. Worse, it may also cause the patient to shock ^[5].

The application of intravenous infusion in the medical field has a long history, and unintelligent ways cause frequent problems. Therefore, the problem of infusion monitoring has caused many scholars to study. Mainly divided into the following categories: capacitive infusion monitoring ^[6,7], photoelectric infusion monitoring ^[8,9], laser sensor monitoring ^[10,11,12] and gravity infusion monitoring ^[13,14,15].

This paper uses image processing and deep learning methods to detect the liquid level of the infusion bottle. Our facility installs a high-definition camera that can perform image processing in the ward to take images of the infusion in real time. First, the target detection is performed on the position of the infusion bottle, then the level line of the infusion bottle is detected, and finally the position of the bit line is judged. When the liquid level line in the infusion bottle is lower than the pre-set warning line threshold, it indicates that the infusion is about to end, and an alarm is issued so that the nurse can go to the ward to change the liquid or draw the needle to the patient. This can improve work efficiency and allocate time reasonably.

2.   Methodology

2.1.   Liquid level automatic detection methods

2.1.1.   Framework based on deep learning

Figure 1 shows a block diagram of the liquid level detection process of the entire infusion bottle. We put the image to be detected into the trained Yolo v3 target detection model ^[16,17] for target detection, then save the result image of the detection and perform area cropping. The crop size is the position size of the target frame, because the target frame is cropped, it is equivalent to the extraction of the infusion bottle.

Next, we send the cropped image into the trained model, perform classification judgment, and judge whether there is a liquid level. If the liquid level is over, a warning will be issued, otherwise, it will not be used.

2.1.2.   Detection process based on traditional image processing

As shown in Figure 2, our technique is depicted by a block diagram of the liquid level detection flow based on image processing. First, we preprocess the image, including grayscale processing, image filtering and pooling operations; then we use Canny edge ^[18,19,20] to detect the overall contour; and finally we use projection method to coarsely locate the liquid level. When the liquid level is below the set threshold, an alarm will be sent out, and if the liquid level is greater than the threshold, it means that the infusion has not ended.

The Canny algorithm calculates the gradient by calculating the first derivative of the Gaussian function to find the local maximum. For weak edge and strong edge detection, thresholds will be set separately for output. Therefore, the Canny algorithm has good anti-noise ability and can distinguish noise from edge detection and detect weak edges.

Integral projection method ^[21,22] is a mathematical statistical method that needs to accumulate the pixel values of each row or column and then present it. The method is simple to operate, fast in detection speed, and high in accuracy.

2.2.   Deep learning based methods

2.2.1.   Yolo v3

The Yolo series of target detection algorithms improve the accuracy of target detection. This paper uses the Yolo v3 algorithm ^[23]. The Yolo v3 algorithm is formed on the basis of v1 ^[24] and v2 ^[25,26]. The structure of Yolo v3 is shown in Figure 3.

Darknetconv2d-BN-Leaky (DBL) is the basic component of Yolo v3 ^[27]. Concat is to splice the upsampling of the darknet middle layer and a later layer.

2.2.2.   RCNN

Regions with CNN features (RCNN) is a milestone in the application of the CNN method to the target detection problem. It was proposed by Girshick ^[28]. With the help of CNN's good feature extraction and classification performance, the Region Estimation method is used to achieve the transformation of target detection problems. Region Estimation is a type of traditional region extraction method, which can be regarded as sliding windows of different widths and heights, and potential target images are obtained through window sliding. The network structure of RCNN is shown in Figure 4.

2.2.3.   Fast-RCNN

The main network of Fast-RCNN algorithm ^[29,30] is still VGG16 ^[31,32]. After inputting the image, after 5 convolutional layers and 2 downsampling layers, it enters the ROI Pooling layer. The input is the output of the conv5 layer and the region Estimation. Then go through two fully connected layers whose output is 4096. Finally, it passes through two fully connected layers with 21 and 84 outputs respectively. The network structure of Fast-RCNN is shown in Figure 5.

2.2.4.   SURF

Speeded Up Robust Features (SURF) is a method based on traditional image processing ^[33,34]. In the field of computer vision, the scale space is the result of repeated convolution and sub-sampling. The more vivid image is the image pyramid. The SURF algorithm uses the method of changing the size of the filter to process multi-level spatial images, so the performance of the algorithm is improved.

As shown in Figure 6, the traditional pyramid and the SURF pyramid are compared. Image (A) is a Gaussian pyramid, changing the image size without changing the size of the filter; Image (B) is a pyramid constructed by the SURF algorithm, changing the size of the filter without changing the size of the original image.

2.3.   Establishment of our experimental database

2.3.1.   Data production

Since there is no relevant standard data set available on the Internet, all the data in this paper are obtained by hand-photographing on-site using Apple mobile phones in our hospital. We collected photos of patients who came to our hospital for intravenous infusion treatment from September 2019 to September 2020. We obtained 280 original images.

2.3.2.   Data enhancement

In order to solve the problem of insufficient sample set, we carried out data enhancement processing on each image, as much as possible to simulate the noise generated by the sensor material property or circuit structure or image decoding when the image is collected in the real use scene, and the exposure caused by the photo. Perform the following 6 operations on each original image: adding salt and pepper noise, Gaussian noise, blurring, perspective transformation, contrast enhancement, and contrast reduction in sequence.

Finally, a total of 1680 data sets were obtained, and we randomly divided them into 80% training set and 20% test set. The specific numbers of training set and test set are shown in Table 1.

2.3.3.   Instruments

We use the Bard three valve type intravenous infusion port produced by Bard Access Systems, Inc. of the United States, product number: 0603654, placed under the skin as a catheter and an injection seat. The catheter is made of radiographic silicone, with an inner diameter of 1.3 mm and a flow rate greater than 500 mL/hr. Other auxiliary equipment mainly include guide wire, puncture needle, curved non-damaged steel needle, vascular dilator and intubation sheath.

2.4.   Evaluation indicators

2.4.1.   Precision

Precision refers to the proportion of true positive classes among all the positive classes judged to be positive. The computation is as shown in Eq (1).

2.4.2.   Recall

Recall refers to the proportion of all real positive classes that are judged to be positive. As shown in Eq (2).

2.4.3.   IoU indicator

The IoU indicator refers to the intersection and ratio, and has been used as a standard metric in semantic segmentation. Cross-combination is not only used in semantic segmentation, but is also one of the commonly used indicators in target detection and other directions. As shown in Eq (3).

2.4.4.   Test time

We count and compare the test time based on traditional image processing methods and deep learning-based detection methods, and then determine which method is more advantageous.

3.   Experimental results

3.1.   Inspection images

Figure 7 shows the detection images of Yolo v3, RCNN, Fast-RCNN and SURF. It can be seen from the results that our proposed method has the highest accuracy.

We note that the liquid level detection based on Yolo v3 is the clearest, while the liquid level lines of the other three methods are fuzzy and intermittent. Yolo v3 is the closest to the liquid level in the infusion bottle, while the liquid level of RCNN, Fast-RCNN and SURF fluctuates up and down.

3.2.   Evaluation results

Table 2 shows the performance comparison of Yolo v3, RCNN, Fast-RCNN and SURF. We use Precision, Recall, IOU and test time to evaluate the four methods.

4.   Discussion

Intravenous infusion is the most common treatment in hospitals. However, because of the speed of infusion, the leakage of needles, and the untimely removal of needles, they may have unpredictable consequences for patients. Therefore, doctors and nurses need to pay close attention to the infusion process. This will undoubtedly increase the burden on medical staff, so it is very necessary to introduce computer technology into the process of intravenous infusion and realize the intelligentization of infusion.

Yolo regards target detection as a regression problem. Its basic idea is to first extract the input features through a feature extraction network to obtain a certain size of output, and then divide the image into several grids. There will be a fixed number of bounding boxes for the target, and the coordinates of the bounding box, the confidence of the object contained in the bounding box, and the probability of classification can be directly obtained by the target through the Yolo network. The most notable feature of Yolo is its relatively fast speed. Secondly, when Yolo processes the input, it is aimed at the entire image rather than a local area. In this case, the context information of the input image can be used well, so that the interference of the background is significantly reduced. As a target detection network, Yolo has a relatively strong generalization ability, and it can often achieve better results in target detection in natural environments.

In the process of liquid level detection of the infusion bottle, the solution proposed in this paper has achieved good detection results. It can be used for reference in the real hospital environment, which is helpful for the development of intelligent medical management and has broad application prospects. However, there are still some shortcomings in this paper that need further improvement.

First, we need to collect a large number of infusion bottle samples, and it takes more time to manually create a data set. The number of sample sets collected in this paper is limited. In order to achieve better detection results, the data set should be diversified; the object of this paper is to detect static images, and the detection of real-time video can be improved later.

Second, in order to improve the accuracy and real-time performance of target detection, one can try to introduce the latest target detection network framework proposed by authoritative scholars in the field of deep learning.

Third, when the liquid level line of the infusion bottle is severely covered and obscured by the medical trademark, it will affect the accurate detection of the liquid level line. The integral projection method needs to be improved, and the projection characteristics of the vertical edge and the horizontal edge are comprehensively used to set the line segment length reasonably. The threshold value can be used for further study.

5.   Conclusions

Based on the problem of detecting the liquid level of the infusion bottle, we adopted four methods to compare the detection results. The Yolo v3 liguid level detection that is proposed in this paper has the advantages of simple operation and fast calculation speed. It can project the characteristics of the pixels in the image in the horizontal and vertical directions, so this method can accurately and quickly detect the liquid in the infusion bottle. Next, this paper divides the liquid level into two categories: anhydrous and with liquid, as a dual-classification problem, and then conducts training and learning, and finally can more accurately classify whether the liquid level category is liquid or anhydrous. After detecting the liquid level state, it can be judged whether the liquid level of the infusion bottle is in the no-liquid alarm state, so as to realize the function of intravenous infusion no-liquid detection.

The liquid level detection method based on Yolo v3 has certain practical value. This method can play a certain auxiliary role in the hospital environment and improve work efficiency. In addition, this method can also be applied to other liquid level detection environments, such as lake level detection, milk bottle level detection, etc.

Acknowledgments

There is no ethics approval required for this paper.

Conflicts of interest

The authors declare no conflicts of interest.