Weakly supervised salient object detection via bounding-box annotation and SAM model

1.   Introduction

Salient object detection (SOD) aims to detect regions of an image that can attract human attention. It has important applications in various vision tasks, such as object detection ^[1], image retrieval ^[2], and object tracking ^[3]. Traditional methods detect salient object through handcrafted features ^[4,5,6,7,8], which usually achieve low performance.

The emergence of deep convolutional neural networks in recent years has greatly improved the performance of SOD. Fully supervised methods ^{[9,10,11,12,13]} have shown satisfactory results while needing expensive pixel-level annotation.

Due to the fully supervised deep learning algorithm relying on pixel-wise annotation, researchers have gradually focused on the SOD task using weakly supervised annotation. Most previous weakly supervised SOD methods focus on using image-level category ^{[14,15,16,17]} and scribble ^[18,19] as weak supervision, which makes it difficult to accurately locate objects and sometimes miss some salient parts. To address the above problems, Zhang et al. ^[18] used bounding-box ^[20,21] annotation, adopted the traditional unsupervised SOD methods and iterative refinement to obtain pseudo labels, and trained a SOD neural network. Wang et al. ^[22] considered the GrabCut algorithm ^[23] on the bounding-box annotation to generate pseudo labels, which is regarded as supervision to train the SOD model. Both these bounding-box annotation based methods ^[18,22] achieve limited performance and demonstrate a huge performance gap with fully supervised methods. For example, compared with the fully supervised method multi-guidance SOD model (Mguid-Net) ^[13], the recent bounding-box annotation based weakly supervised method GC ^[22] shows higher mean absolute error (MAE) by 67.57% and lower maximum f-measure ($F_\beta$) by 9.64% on the DUTS-TE ^[15] benchmark.

In this paper, we propose one weakly supervised SOD method based on the bounding-box annotation and the recent large visual model Segment Anything (SAM) ^[24]. Given one training dataset and its bounding-box annotations, we first regard the bounding-box annotations as box prompt for the SAM model to generate segmentation as SOD initial pseudo labels. Although SAM is one powerful segmentation model, it still may segment incomplete object or incorrectly segment the background region as object. To conquer the influence of undesired results, object completeness and object inversion check is proposed to select reliable pseudo labels, then these reliable pseudo labels are regarded as supervision to train one initial SOD network. Excluding the images with unreliable pseudo labels from the initial model training is helpful to achieve higher performance, but also reduces the diversity of training data. To address the impact of dataset reduction and enrich the training dataset, we adopt the initial SOD network and SAM with the everything mode to generate more reliable pseudo labels, then train one refined SOD model.

The experimental results show that we achieve significantly better performance than other weakly supervised methods and also achieve comparable performance with fully supervised methods.

The main contributions of this work include: 1) We propose one method to generate initial pseudo labels, then carry object incomplete and object inversion check to select reliable pseudo labels for the training initial SOD model. 2) We propose one method combining the initial SOD model and SAM with the everything mode to predict more reliable pseudo labels and enrich the training dataset. 3) We propose one simple yet effective SOD network, which can capture rich global context information. 4) The proposed method significantly outperforms other weakly supervised methods and also achieves comparable performance with fully supervised methods.

2.   Related work

In this section, we first review the previous SOD methods with weak supervision including image-level category, scribble, and bounding box annotation, then introduce the recent large visual model SAM.

2.1.   Weakly supervised SOD

To achieve a trade-off between labeling efficiency and performance, several weakly supervised methods have been proposed to detect salient object.

Due to existing large-scale classification datasets, image-level category based methods have received more attention. Wang et al. ^[15] first proposed to perform SOD with image-level category labels and design a foreground inference network to predict saliency. Li et al. ^[14] proposed to combine a coarse salient object activation map from the classification network and saliency maps generated from unsupervised methods as pixel-level annotation to train fully convolutional networks. Piao et al. ^[16] presented a multi-filter directive network including a saliency network as well as multiple directive filters, which is designed to extract and filter more accurate saliency cues from the noisy pseudo labels. Piao et al. ^[17] designed a noise-robust adversarial learning framework and a noise-sensitive training strategy to conquer noise pseudo labels generated from category.

The scribble annotation on foreground and background also can be regarded as weak supervision for SOD. Zhang et al. ^[18] designed one model based on scribble annotation and edge detection, then proposed one edge-structure-aware module as a supplement of scribble annotation. Yu et al. ^[19] proposed a local coherence loss to propagate the scribble labels to unlabeled regions based on image features and pixel distance.

Weakly supervised SOD with bounding-box annotation has received wide attention in the past two years. Liu et al. ^[25] used bounding-box annotation and the traditional methods to generate initial pseudo labels, then iteratively refined the initial pseudo labels by learning a multitask map refinement network with bounding-box annotation finally, they trained one salient object detector supervised by refined pseudo labels. Wang et al. ^[22] considered the GrabCut algorithm ^[23] on the bounding-box annotation and some adjustment steps to generate pseudo labels, which is regarded as supervision to train the SOD model.

Some researchers consider combining different forms of weak supervision. Zeng et al. ^[26] designed a multisource weak supervision framework to integrate category and caption labels.

Although the above weakly SOD methods were proposed in recent years, these existing weakly supervised methods achieve limited performance and demonstrate a huge performance gap with fully supervised methods. In this work, we propose one new weakly method and aim to achieve comparable performance with fully supervised methods.

2.2.   SAM

With the development of deep learning ^[27,28,29], many deep learning-based image segmentation methods ^[30,31,32] have achieved significant results, but they usually rely on predefined conditions. Recently, SAM ^[24] was introduced to address challenges such as diverse object shapes, sizes, and complex background conditions. The technique aims to achieve efficient and accurate segmentation of arbitrary targets, regardless of target shape, size, and background conditions.

SAM can segment the corresponding mask based on prompts such as the given foreground or background points, bounding-box, mask, and text. The implementation of the above function consists of three steps: 1) extracting the image embedding, which is a time-consuming process but only needs to be calculated once for each image, 2) encoding the user input prompt by the prompt encoder, and 3) decoding the image embedding and prompt embedding by the mask decoder, extracting the interested segmentation, and outputting the corresponding predicted masks and the intersection over union (IoU) score of each mask.

SAM is widely used in computer vision tasks. Chen et al. ^[33] combined image-level annotation and SAM for semantic segmentation. Yamagiwa et al. ^[34] proposed a novel zero-shot edge detection model with SAM. Zhang et al. ^[35] introduced a training-free object segmentation approach for SAM by one-shot tuning on a pair of an image and a mask. Chen et al. ^[36] incorporated domain-specific information or visual prompts into the segmentation network and significantly elevated the performance of SAM. Zhao et al. ^[37] proposed a speed-up alternative method for SAM with comparable performance.

In weakly supervised SOD with bounding-box annotation, generation pixel-level segmentation with bounding-box annotation is the most important step. We can regard the bounding-box annotation as a box prompt input for the SAM model to generate pixel-level segmentation.

3.   Proposed method

The main flowchart of our method includes two stages, which is shown in Figure 1. In the first stage, given one training dataset and its bounding-box annotations, we regard the bounding-box annotations as box prompt for the SAM model to generate initial pseudo labels, then carry two label selection strategies to select reliable pseudo labels as supervision to train one initial SOD network. Excluding the images with low quality pseudo labels from the initial model training is a double-edged sword, which helps the model achieve higher performance but also reduces the diversity of training data. The second stage of our method aims to generate reliable pseudo labels for these excluded images and enrich the training dataset to train a refined SOD model. Specifically, we use the initial SOD network to predict the saliency of excluded images and adopt SAM with the everything mode to generate segmentation candidates, then fuse the saliency and segmentation candidates to predict pseudo labels. Finally we train one refined SOD model using all reliable labels generated in the two stages.

3.1.   Bounding-box annotation

In this work, we use bounding-box annotation as weak supervision with two principles: 1) Each bounding-box is a minimum rectangular box that contains at least one salient object, and 2) the overlapped objects would be merged into one larger region and annotated by one bounding-box. Some examples of bounding-box annotation are shown in Figure 2.

3.2.   Initial pseudo labels generation with SAM

According to the definition of bounding-box annotation, the outside of the bounding-box is the background and the inside of the bounding-box contains salient object while lacking pixel level location. Generation pixel level segmentation with bounding-box annotation is the key step of this task. The existing method ^[25] considered traditional SOD methods and the constraint of the bounding box to compute saliency, followed by saliency adjustment and refinement. However, these traditional SOD methods have poor generalization ability. In ^[22], authors proposed one bounding-box based segmentation via the GrabCut algorithm ^[23], which mainly considers low level image feature and leads to limited performance.

Recently, one new large visual segmentation model SAM ^[24] was proposed. SAM supports bounding-boxes as segmentation prompts. In this weakly supervised SOD task, the bounding-box annotation in each training image can be used as a box prompt input for the SAM model, and the SAM segmentation result can be regarded as an initial pseudo label for SOD. Figure 3 shows four examples of the initial pseudo label generated by the SAM model, where (a) is the input image and bounding-box annotation, (b) is manual annotated ground-truth, and (c) is the segmentation result of SAM with box prompt. The left three columns show satisfactory results, while the right three columns show undesired results.

3.3.   Initial pseudo label selection

Although the SAM model achieves satisfactory results in many training images, it still shows obvious deficiencies in some cases. Some object segmentation results are incomplete, and some background regions are segmented as object. Simply regarding all initial pseudo labels to train salient object detector will lead to limited performance. To conquer these problems, we propose object completeness and object inversion check to exclude undesired pseudo labels while reserving high quality labels.

Object completeness check. Given the input image and bounding-box annotation, which is shown in Figure 4(a), the ground truth is shown in Figure 4(b), and the SAM with box mode may generate an incomplete segmentation result. Two segmentation examples are shown in Figure 4(c), where the top and bottom demonstrate incomplete and complete segmentation results, respectively. To remove the incomplete result from training, we compute the mean IoU (mIoU) to measure the object completeness. Specifically, given one image and its bounding-box annotations $B_i$ ($i$ = 1..$n$), the segmentation result of SAM contains $m$ regions and their minimum external rectangle can be denoted as $C_j$ (j = 1..$m$). For each bounding-box $B_i$, we calculate its IoU with each segmentation rectangle $C_j$ and select the maximum value, then calculate the mean IoU of all bounding-boxes:

where | | means the area function. The $mIoU$ indicates the completeness of an initial pseudo label. In Figure 4(d), the top result shows $mIoU$ with value 0.596 and the bottom result shows $mIoU$ with value 0.971. For each image, its initial pseudo label with $mIoU$ is less than one threshold $th_{mIoU}$, which can be regarded as an incomplete label.

Object inversion check. When salient object occupies most of the image, SAM sometimes ignores the foreground region and incorrectly segments the background region as object. One example is shown in the top row of Figure 5. To conquer this problem, we first perform morphological dilatation filtering on the initial pseudo label, then compute the ratio of object pixels touching the boundary of the image as follows:

where $H$ and $W$ are the height and width of the image, respectively, and $E_t$, $E_b$, $E_l$, and $E_r$ are the numbers of pixels touched with top, bottom, left, and right boundary of the image. The $Bnd$ measures the object inversion of the initial label. In Figure 5(d), the top result shows $Bnd$ with value 0.923 and the bottom result shows $Bnd$ with value 0.066. For each image, its initial pseudo label with $Bnd$ larger than one threshold $th_{Bnd}$ can be regarded as an inverse label.

3.4.   Training initial SOD network

In the aforementioned section, given one training dataset, we adopt SAM with box mode to generate initial pseudo labels, which is followed by label selection to exclude undesired pseudo labels while reserving high quality pseudo labels. The reliable pseudo labels can be regarded as supervision to train one initial SOD network.

In this paper, inspired by ^[38], we propose one simple yet effective SOD network, which is depicted in Figure 6. The proposed SOD model contains three components: an encoder, an atrous spatial pyramid pooling (ASPP) module, and an attention mechanism based decoder. The encoder is based on the ResNet101 ^[39] to extract low-level to high-level visual features and reduce the resolution of feature maps. The encoder consists of a head-convolution and four residual layers denoted as $Layer_i$ (i $\in$ 1, 2, 3, 4). In these four residual layers, the number of residual learning bottlenecks are 3, 4, 23, 3, the strides are set as 1, 2, 2, 1, and the dilations are set as 1, 1, 1, 2, respectively. The resolution of each residual layer feature is 1/4, 1/8, 1/16, 1/16 of the image resolution, respectively.

The ASPP module performs multiscale feature extraction and fusion via dilated convolutions, which allows the extracted features to have a larger receptive field while preserving the resolution of the feature maps. In the ASPP module, the input feature pass through five parallel layers including a $1 \times 1$ convolution, three $3 \times 3$ convolutions with dilation rate of 12, 24, and 36, respectively, and a global average pooling layer, then the outputs of five parallel layers are concatenated to one high-level feature.

The decoder aims to predict a high-resolution saliency map with object details by fusion high-level features and low-level features. Specifically, the high-level output of the ASPP module is followed by $1 \times 1$ convolution ($conv1$) to yield feature $F_1$. Simultaneously, the low-level feature from $Layer_1$ is dimension reduced by $1\times1$ convolution ($conv2$) to yield 64-channels feature $F_2$. The high-level feature $F_1$ is concatenated with low-level feature $F_2$ and followed by a $3 \times 3$ convolution ($conv3$), which outputs 256-channels feature $F_3$.

To assign higher weights on potential salient regions and eliminate redundant noise, the feature $F_3$ is fed into one attention mechanism module, which contains a sequential channel attention operation and a spatial attention operation, inspired by ^[40]. Simultaneously, the $Layer_1$ feature is dimension reduced to 64-channels feature by $1\times1$ convolution ($conv4$) and concatenated with the outputs of the attention mechanism module to recover more object details:

where $concat$ is the concatenation operation among channel axis, $Attn$ denotes the attention mechanism, and $conv4$ denotes one convolution operation.

Finally, the concatenated feature $F_4$ passes through two $3\times3$ convolutional layers ($conv5$, $conv6$), one $1\times1$ convolutional layer ($conv7$), and one upsampling and sigmoid operations to predict saliency. The cross-entropy loss is selected as the loss function:

where $N$ means batch size, $PG$ means the pseudo label, and $S$ means the network predicted saliency.

3.5.   Excluded pseudo labels prediction

In the training of the initial SOD model, some images with incomplete or inverse pseudo labels are excluded from the training dataset for better performance. This operation also reduces the diversity of training data and degrades the generalization of the model. In this section, we propose one method to generate reliable pseudo labels for these excluded images.

Given one excluded image $I$ with bounding-box annotations $B_i$ ($i$ = 1..$n$), the desired pseudo labels need two requirements: salient and with accurate boundary. For the former condition, we use the trained initial SOD network to predict the image saliency $S$, which indicates salient region but lacks accurate object boundary. For the latter condition, we adopt SAM with the everything mode to generate all possible segmentation candidates $R_i$ ($i$ = 1..$k$), which can provide accurate object boundary. For each segmentation candidate $R_i$, in order to determine whether it is salient object or not, two factors are considered:

$(1)$ The position relative to bounding-box annotations. We denote all pixels inside bounding-box annotations as $M$, then calculate the overlapped proportion between segmentation candidate $R_i$ and $M$:

where | | means the area function.

$(2)$ The mean saliency of segmentation candidate. We binarize the predicted saliency $S$ as $S_{bin}$, then calculate pixel-level intersection ratio between segmentation candidate $R_i$ and binaried saliency $S_{bin}$:

If both the above two metrics exceed threshold $th_{Rpos}$ and $th_{Rsal}$, respectively, the segmentation $R_i$ can be regarded as one salient region. All salient regions in the image $I$ together form the predicted pseudo label.

One example of excluded pseudo labels prediction is shown in Figure 7. Given the input image and annotation, which is shown in Figure 7(a), its ground truth is shown in Figure 7(b), its initial pseudo label generated by SAM with box mode is incomplete object and is shown in Figure 7(c), its saliency map predicted by the initial SOD network and binarization result is shown in Figure 7(d) and (e), and SAM with the everything mode generates all possible segmentation candidates with accurate object boundary, which is shown in Figure 7(f). The segmentation candidates inside bounding-box annotation and having high saliency are selected to form predicted pseudo label, which is shown in Figure 7(g).

The excluded pseudo label prediction can generate satisfactory results in most cases, but it inevitably failed in some cases. The object completeness check is applied on these excluded pseudo label prediction results once again to select reliable pseudo labels.

The described procedure can be represented using pseudo-code and is shown in Algorithm 1. All reliable pseudo labels generated in this stage and initial stage are regarded as supervision to train the refined SOD model.

4.   Experiments

In this section, to prove the effectiveness of the proposed method, we present experimental results including implementation details, evaluation metrics, performance comparison, and ablation study.

4.1.   Implementation details

This work uses bounding-box annotation as the prompt of SAM to generate an initial label, then carries object completeness and object inversion check to select reliable labels with threshold parameter $th_{mIoU}$ and $th_{Bnd}$, which is selected as 0.9 and 0.3, respectively. In the excluded pseudo label prediction stage, we use the initial trained SOD model to predict saliency followed by binarization with threshold 0.25 and we use SAM with the everything mode to generate all possible segmentation candidates, then select salient region with threshold parameter $th_{Rpos}$ and $th_{Rsal}$, which is selected as 0.95 and 0.3, respectively. The choice of important parameters will be discussed in the ablation studies.

Our proposed SOD network uses ResNet101, which was pretrained on ImageNet ^[41] as encoder backbone. For the decoders, the convolutional layers weights are initialized by normal distribution with 0.01 standard deviation and zero mean value. Each $3 \times 3$ convolutional layer is followed by a relu and a batch norm layer and the $1 \times 1$ convolutional layer $conv2$ and $conv4$ are followed by a batch norm layer only. We use the stochastic gradient descent to train the SOD network 40 epochs with an initial learning rate of $2 \times 10^{-8}$, a weight decay of $5\times 10^{-4}$, and a momentum of 0.9. The learning rate is decreased by 10% at 10 epochs and batch size is set as 10. To predict the pseudo label of excluded images in the first stage, we use the initial trained SOD network to predict their saliency and fuse SAM segmentation with the everything mode. For predicting better saliency with the initial SOD network, the image feed to training initial SOD network is resized to 512 × 512. In training the final SOD network, following fully supervised method stacked cross refinement network (SCRN) ^[9], we resize the input image to $352 \times 352$ and scaled resolution with [0.75, 1, 1.25] times to augment data.

4.2.   Datasets

Train data: Following the weakly supervised method based on the bounding-box annotation ^[25], we take DUTS-TR ^[15] as the training set, which contains 10553 images. Although the training set already has pixel-level annotation, we do not use pixel-level annotation.

Test data: To verify the performance of the proposed method, tests were conducted on four public datasets: DUTS-TE ^[15], extended complex scene saliency dataset (ECSSD) ^[42], DUT-OMRON ^[8], and HKU-IS ^[43]. The DUTS-TE dataset contains 5019 test images, which contain important scenes for saliency detection. The ECSSD dataset consists of 1000 images with different sizes and multiple targets. The DUT-OMRON dataset contains 5168 images, and the dataset scenes are complex and contain one or multiple objects. The HKU-IS dataset contains 4447 images with high-quality pixel-wise annotations.

4.3.   Evaluation metrics

Maximum F-measure($F_\beta$): The performance measurement computed by the weighted harmonic of the precision and recall is below:

where $\beta^2$ is set to 0.3 to raise more importance on precision.

MAE: The difference between the saliency map $S$ and the ground-truth $G$ can be calculated as below:

4.4.   Performance comparison

We compare our method against some recent state of the art methods, including fully supervised models progressive attention guided recurrent network (PAGR) ^[44], gated bi-directional message passing module (BMPM) ^[45], detect globally and refine locally (DGRL) ^[46], pixel-wise contextual attention network (PiCANet) ^[47], SCRN ^[9], visual saliency transformer (VST) ^[10], PoolNet-R+ ^[11], integrity cognition network (ICON-R) ^[12], Mguid-Net ^[13], weakly supervised models {alternate saliency map optimization (AMSO) ^[14], weakly supervised saliency (WSS) ^[15], NWS ^[26], saliency bounding boxes (SBB) ^[25], GC ^[22], weakly-supervised SOD via scribble annotations (WSSA) ^[18], multi-filter directive network (MFNet) ^[16], structure-consistent weakly supervised SOD (SCWSSOD) ^[19], noise-sensitive adversarial learning (NSAL) ^[17], and unsupervised models texture-guided saliency distilling (TSD) ^[48], uncertainty mining network (UMNet) ^[49], unsupervised domain adaptive SOD (UDASOD) ^[50], and encoder-bigbigan (E-BigBiGAN) ^[51]. For fair comparison, we use either the implementations or the saliency maps provided by the authors.

Quantitative Comparison: Table 1 shows quantitative comparison between our work and nine fully supervised, nine weakly supervised, and four unsupervised methods. The up (down) arrow indicates that the larger (smaller) value is better. Compared with unsupervised and weakly supervised models, we significantly show better performance on all datasets and every evaluation metric. Compared with fully supervised models PAGR ^[44], BMPM ^[45], DGRL ^[46], and PiCANet ^[47], the proposed method also achieves better performance. Compared with fully supervised models SCRN ^[9], VST ^[10], PoolNet-R+ ^[11], ICON-R ^[12], and Mguid-Net ^[13], our work shows only a slightly worse, even comparable performance, while these methods need human labeling pixel-wise annotation.

We also trained our proposed SOD network with manual ground-truth of the training dataset DUTS-TR^[15]. The parameters and training setting are the same as training the final SOD model in weakly mode. In manual ground-truth, although the object and background are usually presented as 1 and 0, we find that it is not binarized on the object boundary. Each training image is annotated by several people, and their annotations show little difference in object boundary. The final ground-truth is the mean of all annotations. Since our pseudo-label is binarized, for fair comparison, we trained our proposed SOD network with manual ground-truth and their binarization, which are denoted as Ours_GT and Ours_GT* in Table 1. Compared with Ours_GT*, our weakly supervised method shows comparable performance except for a slightly worse performance on challenging the DUT-OMRON dataset, and Ours_GT shows larger MAE, mainly due to lacking binarization on object boundary of ground-truth.

Qualitative Comparison: Figure 8 shows a few saliency maps of the evaluated methods. The visual comparison shows that our method performs well and is robust, and can adapt to complex or small object scenes.

4.5.   Ablation study

4.5.1.   Approach effectiveness analysis

In this work, we first adopt the bounding-box annotations to generate initial pseudo labels and select reliable pseudo labels to train one initial SOD model, then predict more reliable pseudo labels to enrich the training dataset and learn the refined SOD model. For fair comparison, we follow SCRN ^[9] and use multiscale augmented training data to train the final SOD model. To verify the effectiveness of each step, we also train one baseline SOD model with all initial pseudo labels. The performance of these SOD models is shown in Table 2. Compared with the baseline model, the initial SOD model achieves significant performance improvement, especially the MAE metric reduced by 13.04%. This result proved that the removal of low-quality labels for model training is effective. The refined SOD model further improves performance and the MAE metric is reduced by 5%, which mainly benefited from the excluded pseudo label prediction. The final SOD model demonstrates the best performance.

4.5.2.   Initial reliable labels selection

To generate initial reliable labels from SAM with box mode, we carry object completeness and object inversion check to select labels. For each image, its initial pseudo label with object completeness measure $mIoU$ less than threshold $th_{mIoU}$ or object inversion measure $Bnd$ larger than one threshold $th_{Bnd}$ will be removed from the initial SOD model training. The label selection is one key step of our method. In order to analyze the parameter influence of the label selection, we use different threshold parameter settings to select reliable labels and train the initial SOD model, then evaluate the model performance. The experimental results are shown in Table 3. The parameter value $\times$ means the selection parameter is invalid and turns off the selection operation.

We first turn off the two label selection operations as baseline, which is shown in the top row of Table 3. We set object inversion parameter $th_{Bnd}$ with 0.9, 0.6, 0.5, 0.3, and 0.15 and set object completeness parameter $th_{mIoU}$ with 0.3, 0.6, 0.7, 0.9, and 0.95. The smaller parameter $th_{Bnd}$ and the larger parameter $th_{mIoU}$ means more strict label selection condition, which can generate more reliable pseudo labels and achieve performance improvement. On the other hand, the excessively strict label selection condition also decreases the number and diversity of training data, which leads to performance drop. The parameter $th_{Bnd}$ with 0.3 and the parameter $th_{mIoU}$ with 0.9 shows the best performance, which is selected as the best parameter.

We also explored the effect of only using object inversion parameter $th_{Bnd}$ or object completeness parameter $th_{mIoU}$, which is shown in Tables 4 and 5, respectively. Parameters of $th_{Bnd}$ and $th_{mIoU}$ settings remain the same as in Table 3. The experimental results conclusion is similar to Table 3. The smaller parameter $th_{Bnd}$ and the larger parameter $th_{mIoU}$ means more strict label selection condition, which can generate more reliable pseudo labels and achieve performance improvement. On the other hand, the excessively strict label selection condition also decreases the number and diversity of training data, which leads to a performance drop. When only using object inversion parameter $th_{Bnd}$ to select reliable labels, $th_{Bnd}$ with 0.30 still shows the best performance in Table 4, but demonstrates a slightly worse performance than the performance in Table 3 when $th_{Bnd}$ with 0.30 and $th_{mIoU}$ with 0.90. This phenomenon proves that object inversion check and object completeness check are complementation operations to select reliable labels. One similar conclusion is also presented in Table 5.

4.5.3.   Excluded label prediction

To predict the pseudo label of excluded images in the first stage, we use the initial SOD network to predict their saliency and adopt SAM with the everything mode to generate segmentation candidates. For each segmentation candidate, we calculate its position measure $Rpos$ and saliency measure $Rsal$. If both $Rsal$ and $Rpos$ exceed threshold $th_{Rpos}$ and $th_{Rsal}$, the segmentation candidate will be considered as a salient region. We set parameter $th_{Rpos}$ with 0.60, 0.90, 0.95, 0.97, 0.99 and set $th_{Rsal}$ with 0.1, 0.2, 0.3, 0.5, 0.7, then train the refined SOD model and evaluate the performance, which is shown in Table 6. The model performs the best performance when the parameter $th_{Rpos}$ is set to 0.95 and $th_{Rsal}$ is set to 0.3.

In addition, we further discuss the result of only using position measure parameter $th_{Rpos}$ and saliency measure parameter $th_{Rsal}$, which is shown in Tables 7 and 8, respectively. Parameters of $th_{Rpos}$ and $th_{Rsal}$ settings remain the same as in Table 6. When only using position measure parameter $th_{Rpos}$ to predict reliable labels of excluded images, higher parameter $th_{Rpos}$ means more strict condition, which can generate more reliable pseudo labels and achieve performance improvement. For the parameter $th_{Rpos}$, even when selected as the very large value 0.99, there are still 10214 images for training the refined SOD model and maintaining the diversity of training data. When only using saliency measure parameter $th_{Rsal}$ to predict reliable labels of excluded images, higher parameter $th_{Rsal}$ means more strict condition, which can generate more reliable pseudo labels and achieve performance improvement. On the other hand, the excessively strict condition also decreases the number and diversity of training data, which leads to performance drop. The parameter $th_{Rsal}$ with 0.3 still shows the best performance in Table 8, but demonstrates a slightly worse performance than the performance in Table 6 when $th_{Rsal}$ with 0.3 and $th_{Rpos}$ with 0.95. This phenomenon proves that position measure and saliency measure are complementation measures to select reliable labels.

5.   Conclusions

Fully supervised SOD based on deep learning usually needs a large amount of data with human annotation. The weakly supervised method based on category, scribble, and bounding-box consumes low annotation cost while achieving limited performance and demonstrating a huge performance gap with fully supervised methods. Research on the weakly supervised method with low annotation cost but achieving comparable performance with fully supervised methods is still challenging for the SOD task.

In this paper, we propose one weakly supervised SOD using bounding-box annotation with two stages. In the first stage, we use the bounding-box annotation as the box prompt of SAM to generate initial pseudo labels, then carry object completeness and object inversion check to select reliable pseudo labels for training the initial SOD model. In the second stage, we use the initial SOD model and SAM with the everything mode to predict more reliable pseudo labels, then use the reliable pseudo labels generated in the two stages as supervision to train one refined SOD model. Experiments show that we achieve significantly better performance than other weakly supervised methods, and also achieve comparable performance with other fully supervised methods.

However, the proposed method contains two stages and is difficult to achieve global optimization. In the future, we will consider one end-to-end method to solve this problem. In addition, we will consider further reducing the annotation cost with unsupervised methods such as k-means clustering ^[52] and contrastive learning. We also will consider applying this method to related fields such as SOD on optical remote sensing images^[53,54,55].

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

This work was supported by R & D Program of Beijing Municipal Education Commission (KM202011232014).

Conflict of interest

The authors declare there is no conflicts of interest.