DPUSegDiff: A Dual-Path U-Net Segmentation Diffusion model for medical image segmentation

1.   Introduction

In the domain of computer vision (CV), one of the main challenges is medical image segmentation, including tasks such as skin lesion segmentation in dermoscopic images, polyp segmentation in colonoscopic images, and brain tumor segmentation in MRI images ^[1]. Segmentation results offer critical insights into anatomical structures, facilitating detailed analysis and significantly aiding physicians in describing lesions, monitoring disease progression, and evaluating the necessity for appropriate treatment ^[2]. Given the escalating demand for intelligent medical image analysis, the development of accurate and robust segmentation techniques has become increasingly imperative.

Traditional medical image segmentation methods encompass threshold-based approaches ^[3], region-based techniques ^[4], edge-based algorithms ^[5], and clustering-based methodologies ^[6], among others ^[7]. While these traditional methods remain effective in simple scenarios, deep learning methods have progressively emerged as the predominant choice due to the inherent challenges of medical images, such as high noise levels, low contrast, and anatomical structures. In the initial stages of deep learning research, convolutional neural networks (CNNs) dominated the semantic segmentation tasks, exemplified by fully convolutional networks (FCNs) ^[8] and U-Net ^[9]. CNNs use convolutional kernels that traverse the input image to extract local features, which limits their ability to capture global dependencies due to the restricted receptive field defined by kernel size ^[10].

To address these limitations, Transformer architectures ^[11] have been introduced into medical image segmentation. The self-attention mechanism within Transformer establishes connections between tokens, thereby demonstrating superior capability in capturing global contextual information. However, the computational complexity of the self-attention mechanism increases significantly with the length of the input sequence, as it requires calculating correlations between each position and all other positions. This results in a substantial increase in computational time and resource requirements when processing high-resolution feature maps.

Denoising diffusion probabilistic models (DDPM) ^[12] are a class of deep learning methods based on probabilistic generation. Their core idea lies in two processes—the forward diffusion process and the reverse denoising process—to achieve data generation from noise. Owing to the superior performance of DDPM in image generation, recent studies have extended its application to tasks such as super-resolution ^[13], deblurring ^[14], and image restoration ^[15]. Previously, these areas were typically dominated by generative adversarial networks (GAN) ^[16]. For example, DAGAN ^[17] performs image segmentation by training a generator and two discriminators. However, there has been limited exploration of DDPM in semantic segmentation. This is primarily due to the fact that most diffusion methods are applied in domains where absolute ground truth is absent, and the quality and practical value of generated results often depend on subjective evaluation. Medical images frequently exhibit uncertainty, boundary ambiguity, and noise issues ^[18]. DDPM can generate diverse segmentation predictions through multiple iterations, offering various potential medical interpretations ^[19]. Moreover, the stepwise denoising process inherent in DDPM facilitates the generation of natural and smooth segmentation outcomes. Building on this foundation, researchers have proposed novel image segmentation methods that leverage diffusion models to address this challenging task ^[20]. Although these methods have achieved impressive performance, medical images often contain irregular target contours and blurred boundaries. Most existing diffusion models adopt a single feature extraction approach (either CNN or Transformer), leading to a trade-off between modeling local edge features and capturing global spatial relationships ^[21]. Moreover, current methods typically rely on image features as guidance but lack an effective feature fusion mechanism during the denoising process ^[22]. As a result, the guidance information has limited influence on segmentation, especially when it comes to leveraging high-level semantic information.

We have developed a novel hybrid network-based segmentation diffusion model, termed DPUSegDiff, to address these challenges. The proposed model employs two parallel U-Nets during the diffusion process, utilizing CNN and Transformer as the underlying encoder-decoder units, respectively. DPUSegDiff promotes the reuse and reconstruction of contextual information through multi-scale information interaction between the two pathways. To facilitate this rich information exchange, we have designed distinct cross-layer information interaction modules for each pathway. In the CNN-based pathway, we integrate the Sobel operator into the convolutional blocks to create a novel local information fusion and enhancement module. This module effectively fuses and enhances high-frequency information such as boundaries and textures. In the Transformer-based pathway, to incorporate low-level detailed features extracted by the local pathway, we construct a global information fusion and enhancement module guided by cross-attention following the self-attention mechanism. At the bottleneck layer of the network, feature maps typically contain more abstract semantic information. We introduce the bilateral gated transformer module (BGTM) to fuse information from the bottleneck layers of both pathways. Additionally, we design a dual-path fusion module (DPFM) that learns trainable parameters to allocate the final output weights of the two pathways.

In this work, we investigate the efficient and interactive integration of the strengths of convolutional neural networks (CNNs), Transformers, traditional edge detection algorithms, and diffusion models. The contributions of this paper are as follows:

1) We introduce a novel denoising probabilistic diffusion segmentation model that integrates CNNs, Transformers, edge detection techniques, feature fusion mechanisms, and diffusion-based denoising methods. By employing various information interaction mechanisms, this model effectively leverages both local details and global context information.

2) We propose two specialized blocks, EALE and MTGE, to construct dual U-Net networks. The EALE block, built upon convolutional layers and augmented with a Sobel edge detection operator, is designed to extract and enhance local features. The MTGE block, based on Transformer architecture, incorporates an efficient cross-attention mechanism to capture global information. Importantly, these two pathways facilitate continuous information exchange at all stages, rather than operating in isolation.

3) We present the BGTM module, which exploits the global and dynamic properties of Transformers to integrate abstract, coarse-grained, high-level semantic features from deeper network layers.

4) We leverage the stochastic sampling capabilities of the diffusion model to generate multiple segmentation predictions, thereby quantifying the model's uncertainty in specific regions.

Extensive experiments on challenging tasks such as skin lesion, polyp, and brain tumor segmentation consistently demonstrate that our proposed network achieves superior segmentation performance and generalization ability.

2.   Related works

2.1.   CNN-based method

Image segmentation involves assigning a label to each pixel to determine its category membership. Fully convolutional networks (FCNs) ^[8] were among the first to introduce a decoder-like architecture for generating segmentation masks. DeepLabv3+ ^[23] incorporated multi-scale information through atrous (dilated) convolutions and a spatial pyramid pooling module. U-Net ^[9] subsequently introduced skip connections and proposed a symmetric encoder-decoder structure, enhancing the model's ability to preserve spatial information. Building on this foundation, U-Net++ ^[24] introduced dense skip connection paths, while U-Net3+ ^[25] further advanced the concept with comprehensive skip connections. Other networks have integrated additional modules with U-Net, such as Res-UNet ^[26] and Attention-UNet ^[27]. CE-Net ^[28] proposed a context encoder to capture more high-level semantic information to improve performance. Considering that 3D images contain richer contextual information and exhibit more complex spatial structures compared to 2D images, specialized convolutional methods have been developed for 3D segmentation, including 3D-UNet ^[29] and V-Net ^[30]. However, since convolution operations can only focus on local features, the model has certain limitations in capturing long-range dependencies and global contextual information. Although researchers have continuously proposed new methods to expand the receptive field of convolutions, the issue of local perception has not been fundamentally resolved.

2.2.   Transformer-based method

Although CNN-based methods have achieved remarkable success in various medical image segmentation tasks, they are known to have limitations in capturing global contextual information, which is crucial for semantic segmentation. Transformer-based methods, on the other hand, excel at addressing this limitation by effectively capturing long-range dependencies. TransUNet ^[31] incorporates Vision Transformer blocks ^[32] into the encoder structure and has demonstrated superior segmentation performance. Swin-UNet ^[33], inspired by the Swin Transformer ^[34] and U-Net ^[9], showcases the potential of pure Transformer networks in medical segmentation. MissFormer ^[35] introduced the self-attention mechanism in skip connections. Although the self-attention mechanism can attend to all features globally to varying degrees, its lack of explicit modeling of local neighborhood structures may lead to insufficient perception of fine-grained edge features and local texture information.

2.3.   Boundary-based method

Boundary information also plays a crucial role in medical image segmentation ^[36]. In medical images, multiple tissues or organs are frequently in close proximity. Preserving distinct boundaries is essential to prevent the blurring of target areas ^[37], particularly for segmenting small structures such as skin lesions and tumor boundaries ^[38]. Traditional edge detection techniques ^[39], especially operator-based methods ^[40], have been central to this task, with the Sobel operator being notably prominent. The Sobel operator, a first-order derivative edge detection method, enhances boundary information by calculating gradient changes in both horizontal and vertical directions, thereby sharpening object contours. However, most diffusion-based medical image segmentation methods tend to overlook boundary information, treating all regions uniformly, which can compromise the accuracy of target boundary segmentation. This paper advocates for incorporating the Sobel operator into the model to enhance boundary perception.

2.4.   Diffusion-based method

DDPM ^[12] represents a class of generative models based on Markov chains that transform simple distributions, such as Gaussian distributions, into data sampled from complex distributions. Diffusion models exhibit significant potential in medical image segmentation. The operational principle of DDPM in medical image segmentation ^[18] is shown in Figure 1. In fact, DDPM leverages stochastic sampling processes to generate implicit segmentation sets, thereby enhancing segmentation performance.

The training process consists of two stages: the forward and the reverse processes. In the forward process, Gaussian noise is progressively added to the original segmentation labels to simulate the degradation from structured labels to random noise. The reverse process learns how to reconstruct clear segmentation labels from any noisy state, guided by the input image. During inference, the process starts from pure Gaussian noise as the initial segmentation map, and under the guidance of the input image, the model progressively denoises and reconstructs the segmentation result until the final prediction is obtained.

SegDiff ^[20] was pioneering in applying diffusion models to iteratively refine image segmentation. Ensdiff ^[21] introduced diffusion methods to generate implicit ensembles. MegSegDiff ^[22] integrated dynamic conditional encoding to adaptively manage varying sampling steps. MedSegDiff-V2 ^[41] used Transformer architectures to integrate noise and semantic features. Traditional diffusion models typically use either CNN or Transformer to build the network or directly adopt classic image segmentation architectures (such as U-Net or ViT) as the backbone for the denoising process. However, such single-network structures often suffer from biased capabilities. A single feature extraction approach struggles to simultaneously ensure global semantic consistency and local boundary accuracy, thereby limiting the segmentation performance of the model. In addition, diffusion models need to handle three types of data simultaneously during the denoising process: the noisy image, the time-step encoding, and the original image. This imposes higher modeling demands on the network, requiring it to effectively integrate temporal dynamics, noise distribution, and image content. Existing diffusion model architectures often simply concatenate these inputs or feed them directly into the backbone network, lacking targeted mechanisms for information decoupling and fusion. As a result, the model's understanding and utilization of different information sources are insufficient, which negatively impacts the final reconstruction quality and segmentation performance.

3.   Methods

3.1.   Diffusion method

DDPM is a class of generative models based on Markov chains, typically used to generate high-quality data samples. The core idea of DDPM involves defining a forward process that gradually adds noise to the data until it reaches a Gaussian distribution and then employing a reverse process to gradually denoise and recover the original data. The process of DDPM in medical image segmentation can be explained using mathematical formulas. In the diffusion process, given a segmentation mask ${x}_{0}\in {\mathbb{R}}^{H\times W\times 1}$, Gaussian noise is gradually added to it, generating a series of noisy samples until a distribution close to Gaussian noise is obtained, as shown in Eq (1).

where ${x}_{t}$ is the noisy sample at time step t, ${\beta }_{t}$ is a hyperparameter that changes with t, $\mathcal{N}$ represents a Gaussian distribution with mean $\mu$ and covariance matrix $\sum$. ${x}_{t}$ can be sampled from ${x}_{0}$ by using the reparameterization trick, as shown in Eq (2).

where ${\alpha }_{t} = 1-{\beta }_{t}$, ${\stackrel{̄}{\alpha }}_{t} = {\prod }_{s = 1}^{t}{\alpha }_{s}$, ${x}_{t}$ can be represented by Eq (3). After multiple diffusion steps, the segmentation mask ${x}_{0}$ is gradually destroyed, eventually becoming pure noise.

In the denoising process, a neural network is trained to progressively denoise through ${p}_{\theta }\left({x}_{t-1}|{x}_{t}, g\right)$, ultimately restoring the segmentation mask ${x}_{0}$. The original image $(I\in {\mathbb{R}}^{H\times W\times C})$ is used as a guide to reconstruct the original structure of the segmentation mask disturbed during the diffusion process, as shown in Eq (4).

where the mean ${\mu }_{\theta }\left({x}_{t}, I, t\right)$ is determined by the noise ${ϵ}_{\theta }\left({x}_{t}, I, t\right)$ predicted by the neural network, as shown in Eq (5).

Starting from pure noise ${x}_{T}\sim \mathcal{N}\left(0, I\right)$, each step iterates as shown in Eq (6).

where ${\sigma }_{t}$ is determined by the noise intensity. When t = 0, the denoised prediction is obtained.

During actual training, the model learns to predict the noise $ϵ$ added to the mask rather than directly predicting the segmentation mask. The noise prediction loss is shown in Eq (7).

Algorithm 1 represents the inference process, while Algorithm 2 describes the training process. The primary difference between the two is that during training, the noisy images are generated by progressively adding noise to the ground truth. However, during inference, the noisy images are randomly generated through Gaussian sampling.

3.2.   Architecture

Our proposed DPUSegDiff model is illustrated in Figure 2. Unlike traditional medical image segmentation methods that directly input raw image data to predict the corresponding segmentation label map, our diffusion model utilizes a noisy segmentation mask, the original image, and a time-guided embedding t as inputs to learn the denoising process.

The structure of DPUSegDiff comprises a dual-path U-Net. One path employs the edge augmented local encoder (EALE) as its fundamental unit to extract image edges and fine details, whereas the other path utilizes the mixed transformer global encoder (MTGE) to capture global and contour-aware features. EALE and MTGE internally integrate a Sobel-guided local information fusion module and a cross-self-attention-guided global information fusion module, respectively. At each stage of upsampling or downsampling, the dual paths share information flow via the embedded local and global fusion modules. In the deeper layers of the network, more abstract and coarse-grained features are fused through the bilateral gated transformer module (BGTM), which leverages the global and dynamic properties of the Transformer to integrate features rich in high-level semantic information. At the end of the model, we designed a dual-path fusion module (DPFM) to selectively integrate the output features of the two U-Net paths, followed by a convolution layer to generate the final prediction. Specifically, DPUSegDiff models each time step of the denoising process through the designed neural network, learning to predict the noise in order to achieve progressive reconstruction. Let the input image be denoted as $g\in {\mathbb{R}}^{H\times W\times C}$, the segmentation label as ${x}_{t}\in {\mathbb{R}}^{H\times W\times 1}$, and the noise variable as $ϵ\in \mathcal{N}\left(0, I\right)$. The operation mechanism of the diffusion model is described in Section 4.1, and the modeling approach of the diffusion model is shown in Figure 2.

3.2.1.   Edge augmented local encoder (EALE)

In the U-Net path focused on local feature extraction, the encoder and decoder share a symmetric structure with a total of 8 stages. Each stage is composed of a ResNet ^[42] and a local feature fusion module, as shown in Figure 3. This fundamental encoding unit has two advantages: (ⅰ) ResNet, with its residual units, enables the convolutional blocks to generate more discriminative feature representations; (ⅱ) the local feature fusion module utilizes the Sobel edge detection operator to extract richer edge information while integrating global semantic information from the other path. We use ${g}^{i}$, ${x}_{t}^{i}\in {\mathbb{R}}^{H\times W\times C}$ to represent the feature map from the previous stage and the feature map from the other path, respectively. The formulation of this fundamental unit is shown in Eqs (8) and (9).

where GN represents group norm. The scale and shift matrices are obtained by linearly projecting the timestep t, which serves to dynamically suppress or enhance the input features. The adjusted features are then passed through a convolutional block for further feature extraction, as shown in Eq (10).

where $\otimes$ represents element-wise multiplication, and ${g}_{t}^{i}$ is first compressed into a single-channel feature using a 1 × 1 convolution before applying a sigmoid activation function, as shown in Eq (11). Subsequently, the Sobel operator is used to extract gradient information (edge features).

${x}_{t}^{i}$ undergoes a similar process. Then ${\widehat{x}}_{t}^{i}$ and ${g}_{t}^{edge}$ are multiplied element-wise and added together to obtain the output, as shown in Eq (12).

The proposed method employs convolution, the Sobel operator, and the sigmoid activation function. The main branch is dedicated to extracting local details of the input image ${g}^{i-1}$, such as edges and textures. Meanwhile, the Transformer extracts global features ${x}_{t}^{i}$ to fill in the long-range dependencies that might be missed by the Sobel or convolution operations.

In this encoder, the input ${{g}_{t}}^{i-1}$ represents the original image features, which are first passed through two convolutional layers to extract shallow local features. These features are then enhanced using a combination of the sigmoid activation function and the Sobel edge detection operator, aiming to emphasize local edges and texture information in the image. Meanwhile, the other input path, ${x}_{t}^{i}$, represents the noisy image features. This branch also undergoes normalization via a sigmoid function and is then element-wise multiplied with the enhanced ${{g}_{t}}^{i-1}$ features, enabling effective fusion of the original and noisy image information. To prevent information loss and strengthen feature propagation, residual connections are incorporated into all operations within the module.

3.2.2.   Mixed transformer global encoder (MTGE)

The global path encoder is constructed based on a Transformer architecture. Figure 4(b) is the main structure of MTGE. In each stage, efficient Transformer blocks are concatenated with self-attention and cross-attention mechanisms. Similar to the local path, the global path also employs Scale and Shift to enhance the feature representation. According to ^[43], we project a linear matrix, Gate, to adjust the output features of the multilayer perceptron to align the diffusion steps, as shown in Figure 4(a). In Figure 4(a), the time step t is linearly transformed into shift, scale, and gate by a multilayer perceptron. They are categorized as additive $\alpha \left(\times \right)$ and multiplicative $\beta \left(+\right)$ adjustments based on their respective mechanisms. Figure 4(c) is the efficient self/cross-attention structure. If the input features are used as Q, K, V, it forms self-attention; if the input features are used as Q, and features from the skip connection are used as K, V, it forms cross-attention.

In each stage, self-attention allows each position in the input features to interact with other positions for capturing long-range dependencies. On the other hand, cross-attention derives its Query, Key, and Value from different input features. This mechanism allows the model to align local features with global features, resulting in more precise segmentation boundaries. The computation process of cross-attention is analogous to that of self-attention, as shown in Figure 4(c). Specifically, the input feature $z = \left[{z}_{1}, {z}_{2}, \cdots, {z}_{h}\right]$ is divided into multiple heads along the channel dimension. Each head independently performs a simplified self-attention operator SA, and the results are then concatenated back together. The processing procedure is shown in Eqs (13)–(15).

where $Attention$ refers to the self-attention operation. Q, K, and V represent the query, key, and value matrices, respectively, while ${d}_{k}$ is the dimension of the key. Generally, shallow feature maps have large spatial resolutions, and attention operations result in computational and memory costs that grow quadratically.

To reduce memory usage and computational costs, we introduce a pooling reduction factor ${P}_{s}$ at the shallow stages. ${P}_{s}$ applies pooling operations only to the K and V matrices, which significantly reduces these overheads while preserving important global context information. Meanwhile, we introduce depthwise separable convolutions ^[44] (depthwise convolution and pointwise convolution) to replace the traditional multilayer perceptron (MLP). This is because depthwise separable convolution offer the following two advantages: (ⅰ) a lower parameter count and computational complexity, which helps improve the model's computational efficiency and training speed; and (ⅱ) compared to MLP, which only models features along the channel dimension, depthwise separable convolution can model local features while preserving spatial structure information. This is particularly crucial for tasks such as medical image analysis, where boundary and structural sensitivity are essential. Depthwise convolution applies $k\times k\left(k > 1\right)$ convolution kernels to each channel. The features within the $k\times k$ kernels are aggregated to compute new features. The computation can be expressed as Eqs (16) and (17).

where $X$ represents input feature map, ${W}_{depth}$ and ${W}_{point}$ represent the depthwise convolution kernel and pointwise convolution kernel, respectively. $i, j, p, q$ are spatial position indices, and $c, k$ are channel position indices.

3.2.3.   Bilateral gated transformer module (BGTM)

The deep features in the bottleneck layer encapsulate more abstract and higher-dimensional information. Relying solely on traditional convolution operations makes it challenging to effectively integrate all potential information. Thus, we developed a bilateral gated Transformer structure, as shown in Figure 5, to fully merge the features from both paths. Specifically, Q and K are obtained from the linear projection of feature maps from the same pathway, whereas V originates from the alternate pathway. A gating mechanism is incorporated, where features from the opposite path are processed through a sigmoid activation function and then multiplied with the original path features. This facilitates dynamic modulation of the weight distribution for the original features. The precise calculation is detailed as Eq (18).

where $V$ represents the original features, and $\widehat{V}$ represents the features from the contrasting path. The gate captures the importance relationship between the features from the contrasting path and the current features, assigning different attention weights to the original features, thereby enhancing the expression of key features while suppressing redundant or irrelevant ones.

3.2.4.   Dual-path fusion module (DPFM)

The design of dual-path networks is not commonly seen in existing research, as the information transfer between paths and the fusion of features at the end of the paths present certain challenges. We propose the dual-path fusion module (DPFM), which can merge and reallocate the output features from the two branching paths. It uses learnable parameters $\alpha \left(\theta \right)$ to optimize the entire architecture via backpropagation. The output features from one path are multiplied by parameter $\frac{\alpha \left(\theta \right)}{2}$, while the output features from the other path are multiplied by $\frac{\left[1-\alpha \left(\theta \right)\right]}{2}$. The formula can be expressed as Eq (19).

where ${I}_{x}$ and ${I}_{g}$ are the output feature maps from the EALE and MTGE, respectively. The parameter $\theta$ is constrained within the range of 0 to 1 to ensure numerical stability. ${I}_{F}$ is the output after fusion, which is then passed through a convolution layer to produce the final prediction results.

4.   Experiments

4.1.   Implementation details

The proposed DPUSegDiff is fully implemented in PyTorch, and experiments are conducted in a single GPU setup with an NVIDIA 3090 GPU (24 GB). During the training of the diffusion model, the initial learning rate is 0.0001, the batch size is 16, the Adam optimizer is used for backpropagation, and it adopts a hybrid loss function combining Dice loss and cross-entropy loss. To enhance the model's robustness, the total number of diffusion steps, T, is set to 250. During the inference stage, to improve the segmentation accuracy, 10 samples are taken for each input image, and the results are averaged to obtain the final segmentation mask. During the experiments, we resized the input resolution of all datasets to 128 × 128 and applied various combinations of data preprocessing using the Albumentations library, including but not limited to scaling, translation, rotation, brightness adjustment, RGB channel enhancement, Gaussian noise, and Gaussian blur. To ensure a unified and comprehensive evaluation of performance across all datasets, we adopted four metrics as our evaluation criteria: Dice similarity coefficient (DSC), intersection over union (IoU), recall, and specificity. The calculation methods of the four evaluation metrics are as follows:

1) DSC: The segmentation capability of a model is typically measured using the Dice similarity coefficient (also known as F1 score), as shown in Eq (20). This coefficient represents the similarity between two samples, with a range of [0, 1], where a higher value indicates better model performance.

where TP represents true positives, FP represents false positives, and FN represents false negatives.

2) IoU: Intersection over union (IoU) is a standard performance evaluation metric for image segmentation problems. It is also used to measure the similarity between two samples. The IoU score is calculated as shown in Eq (21).

The range of IoU is also [0, 1]. It is important to note that Dice calculates the ratio of the intersection between the predicted and ground truth regions to the union of those regions, while IoU calculates the weighted proportion of the intersection between the predicted and ground truth regions relative to the total area. The value of DSC is usually higher than that of IoU because DSC gives more weight to the intersection, enhancing the contribution of true positives (TP).

3) Recall: Recall measures the ability of the model to correctly identify all positive samples. Its calculation formula is shown in Eq (22).

4) Specificity: Specificity measures the model's ability to correctly identify negative samples (background). The formula is shown in Eq (23).

4.2.   Task 1: Skin lesion segmentation

4.2.1.   Skin disease dataset

We first validated the proposed DPUSegDiff for skin lesion segmentation. Skin lesion segmentation is an important task in medical image analysis, often used to assist in the diagnosis of diseases such as malignant melanoma. This experiment involves the ISIC 2018 ^[45] and PH² datasets ^[46]. ISIC 2018 is a widely used skin lesion dataset that includes a rich variety of lesion areas, presenting significant challenges. It helps evaluate the model's performance in complex scenarios, such as blurred boundaries and noise interference. The ISIC 2018 dataset comprises 2594 RGB images, which are divided into training, validation, and test sets at 70%, 10%, and 20%, respectively. The PH² dataset, consisting of merely 200 images, is relatively small. Consequently, we refrain from splitting the PH² dataset for training purposes and instead employ it to evaluate the generalization capabilities of the ISIC 2018 dataset.

4.2.2.   Segmentation performance

Table 1 presents the performance of various methods on the skin lesion segmentation task. The column labeled ISIC 2018 lists the evaluation metrics for each model on the ISIC 2018 dataset. From the experimental results, it can be observed that the proposed DPUSegDiff method significantly outperforms existing mainstream approaches across all metrics.

Specifically, compared to the classic CNN-based U-Net, DPUSegDiff achieves a 4% improvement in DSC, 2% in IoU, 5% in recall (Rec), and 2% in specificity (SP). When compared to the popular Transformer-based architecture TransUNet, DPUSegDiff shows a 5% gain in DSC, 4% in IoU, 5% in Rec, and 2% in SP. Even against the advanced hybrid method UCTransNet, which integrates both CNN and Transformer structures, DPUSegDiff still achieves improvements of 2% in DSC, 3% in Rec, and 2% in SP. Furthermore, compared to the current state-of-the-art diffusion model MedSegDiff, DPUSegDiff achieves a 1% increase in DSC.

These quantitative results clearly demonstrate that the proposed DPUSegDiff possesses stronger feature representation capabilities and more precise target region recognition in the skin lesion segmentation task. Whether compared with traditional CNN-based methods, Transformer-based methods, or hybrid models that combine the strengths of both, DPUSegDiff consistently shows a clear advantage. At the same time, DPUSegDiff also outperforms diffusion models that adopt a single network structure, highlighting the higher effectiveness and robustness of our design in terms of information fusion and denoising mechanisms.

Figure 6 illustrates the visual segmentation results of different models on representative samples. As shown in the figure, for lesion regions with clear boundaries and regular shapes, all models achieve satisfactory segmentation performance. However, in more challenging cases—such as lesions with blurred edges or complex structures—DPUSegDiff is able to more accurately reconstruct the target regions, with segmentation boundaries that closely match the ground truth. Other methods commonly suffer from over-segmentation, under-segmentation, or mis-segmentation issues, indicating limitations in their generalization ability under complex conditions.

To further validate the generalization ability of each model, we conducted transfer testing on the PH² dataset using model weights trained on the ISIC 2018 dataset. The results show that DPUSegDiff consistently outperforms other methods across all evaluation metrics, further confirming its excellent generalization capability and robustness.

In summary, both quantitative analysis and qualitative evaluation demonstrate that DPUSegDiff achieves superior segmentation performance compared to existing methods, highlighting its broad applicability and practical value in the task of skin lesion segmentation.

4.3.   Task 2: Polyp segmentation

4.3.1.   Polyp dataset

The second evaluation dataset comprises the colonoscopy images from the CVC-ClinicDB ^[48] dataset, which are used for early detection and diagnosis of colorectal cancer. CVC-ClinicDB is a colorectal cancer dataset that has been validated by numerous studies and includes standardized segmentation annotations. It has been used by multiple research institutions, which can provide a reliable benchmark for evaluating our model. The experiment employs 612 images in total, with 428 images for training, 61 for validation, and 123 for testing.

4.3.2.   Segmentation performance

Table 2 quantitatively presents the segmentation performance of various models on the polyp dataset. Figure 7 qualitatively illustrates their segmentation results from a visual perspective. Compared to the classic method U-Net, our approach improves DSC and IoU by 2% and Rec by 5%. When compared to the mainstream method, TransUNet, it achieves a 1% gain in DSC and IoU, and a 4% increase in Rec. These quantitative and qualitative results demonstrate the effectiveness and superiority of our proposed method.

4.4.   Task 3: Brain tumor segmentation

4.4.1.   Brain tumor dataset

The third evaluation centers on the BraTS brain tumor segmentation challenge dataset. This dataset comprises MRI scans from 369 unique patients. Unlike other datasets, the BraTS dataset offers four different MRI modalities for each case: FLAIR, T1w, T1gd, and T2w. These multimodal datasets provide rich texture and structural information, which aids in enhancing segmentation performance. We organized the dataset following the previously used training, validation, and testing splits, and subsequently processed the 2D slices to align with the model's input format.

4.4.2.   Segmentation performance

Table 3 provides a quantitative analysis of the evaluation metrics for different methods on the BraTS dataset ^[50]. The results indicate that DPUSegDiff outperforms all compared models across the board. Specifically, on the brain tumor dataset, our method achieves a significant improvement of 11% in both DSC and IoU, and a 13% increase in Rec compared to the CNN-based U-Net. When compared with the Transformer-based Swin-UNet, DPUSegDiff also shows a 4% improvement in DSC.

Figure 8 presents the qualitative visual segmentation results of different models on test samples. As shown in the figure, although all methods can roughly segment the tumor regions, DPUSegDiff demonstrates superior performance in capturing tumor details and produces sharper and more accurate boundaries. These results strongly demonstrate that DPUSegDiff significantly improves both the accuracy and stability of segmentation in brain tumor segmentation tasks.

4.5.   Ablation study

We conducted a comprehensive ablation study to validate the effectiveness of the proposed modules. The results are presented in Table 4. The symbol √ indicates that the module is deployed in the network, whereas × signifies that the module is not used. Specifically, EALE represents the scenario where the Sobel edge detection operator is not used, and MTGE signifies the scenario without the cross-attention mechanism. As indicated in the table, both EALE and MTGE significantly improved fusion performance. Moreover, the bilateral gated Transformer and DPFM also proved to be effective in enhancing the segmentation performance.

5.   Discussion

According to ^[21], implicit integration is a key concept in diffusion model-based methods, primarily related to the sampling strategy and stability during the generation process. It typically refers to the process of implicitly optimizing the target distribution through multiple iterative sampling steps during the reverse process of the diffusion model (generating data from noise). The importance of multiple sampling runs in implicit integration is rooted in the following reasons: (ⅰ) Each sampling can be considered an implicit optimization step, and (ⅱ) multiple runs enable the exploration of various generation paths and assist in avoiding entrapment in local optima.

In Figure 9, we present the original image, Ground Truth, two different sampled segmentation masks, as well as the mean and variance maps. From the figure, it is evident that each sampling introduces some regions of uncertainty. This uncertainty typically arises from the model's noise prediction bias or the inherent randomness in the generation process. By conducting multiple samples on the same input and averaging the outcomes, we can significantly diminish the randomness of the predictions, reduce the model's uncertainty, and make the final segmentation results more stable and closer to the ground truth.

Figure 10 illustrates multiple batches and depicts the relationship between the sampling iterations in the ensemble and the Dice score. We can observe that as the sampling iterations increase, the Dice score gradually rises and then stabilizes.

We randomly selected several batches from the test dataset, each with a batch size of 8, and performed 100 independent samplings for each batch. The segmentation results from each sampling were integrated with the previously obtained results, and the average Dice score was calculated accordingly. This process was used to evaluate the model's stability and performance variation under different numbers of sampling iterations.

6.   Conclusions

This paper introduces a dual-path U-Net diffusion segmentation model with rich information interaction, called DPUSegDiff. The model leverages a Markov chain denoising process and generates implicit ensemble predictions with multiple outcomes. Specifically, the two paths of DPUSegDiff apply CNN and Transformer modules, focusing on the extraction of local and global features. To enhance the information flow between the two pathways, we have integrated information fusion modules within each pathway, tailored to their respective emphases. These fusion modules incorporate traditional edge detection algorithms and cross-attention mechanisms, thereby sensitively enhancing the information extracted by the dual-path encoder. The proposed BGTM selectively integrates deeper features from the bottleneck layer, effectively bridging the two pathways. Extensive experiments conducted on three challenging tasks—skin lesion, polyp, and brain tumor—consistently demonstrate that the proposed DPUSegDiff achieves superior segmentation performance and generalization capability.

Although DPUSegDiff demonstrates significant advantages in image segmentation tasks, our work has certain limitations. Inference speed remains one of the main limitations of current diffusion models. Specifically, diffusion models typically require hundreds or even thousands of iterative denoising steps to reconstruct high-quality predictions from pure noise, which leads to considerable computational cost and inference latency in practical applications. Although some recent methods have attempted to accelerate the inference process by reducing the number of sampling steps, these approaches often suffer from accuracy degradation or reduced generalization ability. As such, it remains challenging to fully overcome the speed bottleneck without sacrificing performance. Therefore, we believe that optimizing inference efficiency will continue to be an important research direction for diffusion models in the future.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

This work is supported by the Natural Science Foundation of Fujian Province (Grant Nos. 2024J01821, 2024J01820), Research Project on Undergraduate Higher Education Teaching of Fujian Province (Grant No. FBJY20230083), Model Courses of Ideological and Political Education in Graduate Programs at Minnan Normal University (Grant No. SZ202406), Research Project on Postgraduate Education and Teaching Reform of Minnan Normal University (Grant No. YJG202316) and the Principal Foundation of Minnan Normal University (Grant No. KJ18010)

Author contributions

Yazhuo Fan was responsible for proposing the research idea, designing the core structure of the medical image segmentation model, implementing the algorithm, conducting the experiments, and drafting the initial manuscript. Jianhua Song supervised the overall research, provided methodological support, contributed to the model design and optimization, and revised the manuscript extensively. Yizhe Lu assisted in the model development and parameter tuning, and contributed to the analysis and visualization of the experimental results. Xinrong Fu participated in data preprocessing, experimental pipeline setup, and model evaluation, and contributed to interpreting the results. Xinying Huang conducted literature review, supported model validation, and contributed to writing part of the manuscript. Lei Yuan provided clinical insights and domain expertise, supported medical image dataset preparation, and offered constructive suggestions for improving the final manuscript.

Conflict of interest

The authors declare there is no conflict of interest.