Forecasting crude oil price using LSTM neural networks

1.   Introduction

Deep learning techniques have proven successful in various computer vision fields, such as image classification ^[1], object detection ^[2], and semantic segmentation ^[3]. However, the effectiveness of deep learning relies heavily on large, labeled training datasets, which could be labor-intensive to annotate. When dealing with large unlabeled datasets, it is often impractical to label enough data for training a deep learning model. An alternative approach is transfer learning, where labeled data from related domains (source domain) are utilized to enhance the model's performance in the domain of interest (target domain). Transfer learning is the process of applying knowledge learned from a labeled source domain to a target domain, where labeled data may be limited or unavailable.

Pan ^[4] classified transfer learning into three categories according to labeled data in the two domains used during training. They are (1) inductive transfer learning: When the target domain data is labeled, irrespective of whether the source domain data is labeled or not; (2) transductive transfer learning: When only the source domain data is labeled, while the target domain data remains unlabeled; and (3) unsupervised transfer learning: When both domains lack labels. Transductive transfer learning can be further divided into two types: (1) domain adaptation: When both domains use the same attributes but have different marginal probability distributions; and (2) sample selection bias: When the sample spaces or data types of the two domains are different, such as images in the source domain and text in the target domain. This paper focuses on unsupervised domain adaptation (UDA), which aims to minimize the distribution discrepancy between data from two domains, enabling successful knowledge transfer from the source domain to the target domain.

Currently, numerous domain adaptation methods have been researched and developed ^{[4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23]}. These methods fall into three main categories ^[4]: Instance reweighting methods ^[5,6,7], feature extraction methods ^[8,9,10,11], and classifier adaptive approaches ^[12,13]. Feature extraction methods aim to learn domain-invariant feature representations and are broadly categorized into two types ^[14]: Adversarial learning-based approaches ^[15,16,17] and statistics-based approaches ^[18,19,20]. Adversarial learning-based methods seek to achieve domain-invariant features by generating images or feature representations from different domains. For instance, the deep reconstruction classification network (DRCN) ^[21] establishes a classifier for labeled source domain data and constructs a domain-invariant feature representation shared with unlabeled target domain data. Statistical methods involve defining a suitable measure of difference or distance between two distinct distributions ^{[18,24,25,26,27,28,29]}. Various distance metrics, such as quadratic ^[30], Kullback-Leibler ^[31] and Mahalanobis ^[32], have been proposed over the years. However, these methods are not easily adaptable to different domain adaptation (DA) models and may not effectively describe complex distributions like conditional and joint distributions due to theoretical limitations. In recent years, the MMD ^[28], initially used for two-sample testing, has been found to be effective in calculating the distance between sample distributions from two domains in feature space. It facilitates alignment between the distributions by minimizing the MMD between them. The method presented in this paper falls under this category. Long et al. ^[9] introduced the regularization of MMD, utilizing it to reduce the distribution difference between the feature distributions of two domains in hidden layers of deep adaptation networks.

The use of MMD focuses mainly on aligning the overall distribution of two domains, but often falls short in ensuring precise alignment of data within the same category across domains. In response, Long et al. ^[19] proposed the class-wise maximum mean discrepancy (CWMMD) to enhance robust domain adaptation. The two-domain samples are linearly mapped into a common feature space and the MMD for each category is calculated, then summed to obtain a CWMMD. Wang et al. ^[33] highlighted that minimizing the MMD is equivalent to minimizing the overall data variance while simultaneously maximizing the intra-class distances of the source and target domains, leading to a decrease in feature discriminativeness. They adjusted balance parameters to mitigate this issue but were limited to linear transformations in the feature space. However, they used the L2 norm as the MMD estimator in a linearly transformed feature space. It is worth noting that the L2 norm is not well suited for general estimation ^[34,35], and that linear transformations may not adequately capture complex data relationships, especially if nonlinear mappings are required.

In contrast, deep neural networks, particularly convolutional neural networks (CNNs), learn powerful and expressive nonlinear transformations. This paper proposes a method to improve upon this, which involves training a CNN architecture, so that the model automatically learns feature representations that are well-suited for the task at hand. Furthermore, the loss function used in the domain adaptation process can be efficiently evaluated in a reproduced kernel hilbert space (RKHS). This facilitates effective alignment of data belonging to the same class from both the source and target domains in the shared feature space.

2.   Materials and methods

This section presents the related research, including pseudo labels and different variants of MMD.

2.1.   Pseudo labels

Computing a class-level MMD during training requires the use of pseudo-labels for unlabeled target domain data. A simple way to generate pseudo labels is directly applying formula (1) to the source domain model ^[14]; that is, input the target sample ${x}_{t}^{}$ into the source domain model f = C $\bullet$ F, which comprises a feature extractor F and a classifier C, to obtain the softmax result of classification $\delta = ({\delta }_{1}, {\delta }_{2}, \dots, {\delta }_{C})$, and then set the index of the maximum of components in the output vector $\delta$ as the pseudo label for the target sample ${x}_{t}^{}$. However, due to domain shift, the pseudo-label generated by this method may have large bias. Instead, this study adopts another method, the self-supervised pseudo-label strategy proposed by Liang et al. ^[36]. The strategy first uses the current target domain model to calculate the centroid of each category for the target domain data, which is similar to weighted K-means clustering, as shown in formula (2). These centroids robustly represent the distribution of different classes in the target domain data. Next, the category of the nearest centroid for each target domain data is obtained as its pseudo label, as shown in formula (3), where ${\text{D}}_{\text{cos}}\text{(}a\text{, }\text{b}\text{)}$ means the cosine distance between a and b. The new pseudo-label is then utilized to recalculate the centroid, as shown in formula (4), and update the pseudo-label again, as shown in formula (5). Finally, with the new pseudo labels, the target model is self-supervised using the cross-entropy loss function, as shown in (6).

2.2.   MMD from two-sample tests

The MMD is a distance measure between feature means. Gretton et al. ^[28] introduced an MMD measure, which involves embedding distribution metrics in the RKHS and using it to conduct a two-sample test for detecting differences between two unknown distributions, p and q. The purpose of this test is to draw two sets of samples X and Y from these distributions and to determine whether p and q are different distributions. They applied a kernel-based MMD to two-sample tests on various problems and achieved excellent performance. Furthermore, these kernel-based MMDs have been shown to be consistent, asymptotically normal, robust to model misspecification, and have been successfully applied to various problems, including transfer learning ^[29], kernel Bayesian inference ^[37], approximate Bayesian computation ^[38], two-sample testing ^[28], optimal degree-of-fit testing ^[39], generating moment matching networks (GMMN) ^[40], and autoencoders ^[41].

Gaussian kernel-based MMDs are commonly used estimators, which have the key property of universality, allowing estimators to converge to the best approximation for generating distribution of the (unknown) data in the model, without making any assumptions about this distribution. In contrast, the L2 norm lacks the above properties, suffers from the curse of dimensionality, and is not suitable for universal estimation ^[34,35]. Furthermore, Gaussian kernel-based MMDs also serve as an effective measure for domain differences in UDA scenarios, and their computation is streamlined by applying a kernel function directly to the samples.

2.2.1.   Formulation of MMD

The squared MMD in an RKHS, denoted as ${‖{\mu }_{p}-{\mu }_{q}‖}_{\mathcal{H}}^{2}$, can be straightforwardly expressed using kernel functions. Additionally, it is possible to easily derive an unbiased estimate for finite samples. Considering independent random variables x and x' from distribution p, as well as independent random variables y and y' from distribution q, let $\mathcal{H}$ be a universal RKHS with unit ball denoted $\mathcal{F}$, and one can give the squared MMD, as shown in (7) ^[32], where $\phi (\cdot)$ is a function mapping the samples to $\mathcal{H},$ ${\mu }_{p} = {\mathbb{E}}_{x\sim p}\left[\phi \right(x\left)\right]$ and ${\mu }_{q} = {\mathbb{E}}_{y\sim q}\left[\phi \right(y\left)\right]$ representing the kernel mean embeddings, and $k$ is set to the commonly used Gaussian kernel, as shown in (8). An unbiased empirical estimate is given in (9), where $X = \{{x}_{1}, \dots, {x}_{m}\}$ and $Y = \{{y}_{1}, \dots, {y}_{n}\}$ are two sets randomly sampled from two probability distributions p and $q$, respectively. However, there is no definitive method for selecting the bandwidth $\sigma$ of the kernel $k$ in (9). Gretton et al. ^[28] suggested using the median distance between samples as the bandwidth, but did not verify that this choice is optimal.

2.2.2.   Class-wise MMD based on L2-norm

Although MMD has been commonly used in cross-domain problems, minimizing the MMD between the samples from two domains only narrows their marginal distributions. Long et al. ^[8] proposed joint distribution adaptation (JDA), which jointly adapts marginal and conditional distributions in a reduced-dimensional principal component space and constructs new feature representations. They adopted a principle component analysis (PCA) transformation and minimized the Euclidean distance between the sample means of the two domains in a reduced-dimensional principal component space. They referred to this method as MMD for marginal distribution, as shown in formula (10), where ${X}_{s}$ $\in {R}^{d\times {n}_{s}}$ and ${X}_{t}\in {R}^{d\times {n}_{t}}$ are the samples from the source and target domains, respectively. Here, ${n}_{s}$ and ${n}_{t}$ are the numbers of samples from the source and target domains, $d$ is the sample dimension, and $\phi$ represents a linear transformation function. Let $A$ be the $d\times K$ standard matrix of the linear transformation function $\phi$. Formula (10) can then be rewritten as formula (11), where ${X}_{st} = \left[{X}_{s}\right|{X}_{t}]$ and ${M}_{0}\in {R}^{{n}_{st}\times {n}_{st}}$ are calculated as shown in formula (12).

In addition to the MMD for marginal distribution, they proposed also the MMD for conditional distribution. However, during empirical estimation, to obtain samples for each category, labels for target domain samples that do not exist need to be provided. To address this, they suggested using pseudo labels for the target samples, which can be obtained either from the current classifier trained on the samples from the source domain or through other methods. They named the MMD for conditional distribution as class-wise MMD (CWMMD), as shown in Eq (13). Here, ${X}_{s}^{c}$ and ${X}_{t}^{c}$ represent the data samples of the c^th category from the source and target domains, respectively, while ${n}_{s}^{c}$ and ${n}_{s}^{c}$ are the respective sample sizes and the calculation of ${M}_{c}\in {R}^{{n}_{st}^{c}\times {n}_{st}^{c}}$ is detailed in (14).

In JDA, both marginal distribution discrepancy and conditional distribution discrepancy across domains are simultaneously minimized. Consequently, the optimization problem for JDA is resolved by combining Eqs (11) and (13), as shown in Eq (15). In (15), the constraint condition ${{A}^{T}X}_{st}{H}_{st}{X}_{st}^{T}A = {I}_{K\times K}$ limits the overall data variation to a fixed value, ensuring that data information on the subspace is statistically retained to some extent. ${‖A‖}_{F}^{2}$ controls the size of the matrix $A$, and $\alpha$ is a regularization parameter ensuring a well-defined optimization problem. It is important to note that ${H}_{st}{ = I}_{{n}_{st}\times {n}_{st}}-\frac{1}{{n}_{st}}{1}_{{n}_{st}\times {n}_{st}}$ is a centering matrix, where ${n}_{st}^{c} = {n}_{s}^{c}+{n}_{s}^{c}$ and ${1}_{{n}_{st}\times {n}_{st}}$ is a matrix of size ${n}_{st}\times {n}_{st}$ with all elements being one.

2.2.3.   Discriminative CWMMD based on L2-norm

Wang et al. ^[33] proposed an insight into the working principle of MMD and theoretically revealed its high degree of agreement with human transferable behavior. In Figure 1 ^[33], when considering a pair of classes labeled "desktop computers", respectively, from the source and target domains, the process of minimizing the MMD between these two distributions involves two key transformations. They are (1) two relatively small red circles (hollow and mesh circles) transformed into larger red ones, which are magnified; i.e., maximizing their specific intra-class distance and (2) two tiny red circles gradually moving closer along their specific arrows; i.e., minimizing their joint variance. This process is analogous to how humans abstract common features to encompass all possible appearances, but the detailed information is heavily decayed. Wang et al. also theoretically demonstrated this insight.

Let ${\left(S(A, X)\right)}_{inter}^{c}$ = $tr\left({{A}^{T}X}_{st}{M}_{c}{X}_{st}^{T}A\right)$ denote the inter-class distance (i.e., square of MMD) between the $c\mathrm{t}\mathrm{h}$ class data in the source domain and the target domain in the transformation space according to the transformation matrix $A$, and let ${S}_{inter} = {\sum }_{c = 1}^{C}{\left(S(A, X)\right)}_{inter}^{c},$ then (15) is written as (16). Wang et al. derived ${{S}_{inter} = S}_{var}-{S}_{intra}$, so (16) is written as (17), where ${S}_{intra}$ represents the intra-class distance, and ${S}_{var}$ is the variance of the entire data. Therefore, minimizing the inter-class distance ${S}_{inter}$ is equivalent to maximizing their variation ${S}_{var}$, and maximizing the intra-class distance ${S}_{intra}$ at the same time, which will reduce feature discriminativeness. To address this, a trade-off parameter is introduced to adjust the hidden intra-class distance in ${S}_{inter}$, as shown in Eq (18). They obtained an optimal linear transformation matrix $A$, thus minimizing the loss evaluated in this transformation space.

3.   The proposed method

The unsupervised domain adaptation training proposed in this paper focuses on using discriminative CWMMD (DCWMMD) to align data of the same class between the source and target domains. By alleviating the problem of MMD through reducing feature discriminativeness while minimizing the mean difference between the two domains, the proposed method effectively achieves the goal of unsupervised domain adaptation.

Unlike Wang et al. ^[33], who used the L2-norm as an MMD estimator in the linearly transformed feature space, this study employed a network to train a feature space. Samples in this space are then projected into an RKHS to efficiently evaluate and minimize the loss function. The Gaussian kernel is commonly used because the RKHS with the Gaussian kernel is guaranteed to be universal ^[30]. Wang et al. proposed that the inter-class distance is equal to the variation minus the intraclass distance under the MMD they defined. This section reformulates the interclass distance, intra-class distance, and variation as defined by Wang et al. and adopts the MMD with the Gaussian kernel. Moreover, it provides a proof that when using this MMD to measure the distance of distribution of samples from two domains, the interclass distance is indeed equal to the variation minus the intra-class distance.

3.1.   Symbols and notations

This study considers only two domains, one source domain and one target domain. ${X}_{s}^{c}$ and ${X}_{t}^{c}$ represent the sample sets of class c from the source domain and the target domain, respectively, and ${X}^{c}$ (or ${X}_{st}^{c}$) represents the union of all sample sets of class c in both domains, i.e., ${X}^{c} = {X}_{st}^{c} = {X}_{s}^{c}\cup {X}_{t}^{c}$. More symbols and notations are presented in a nomenclature table provided in Table 1.

3.2.   Relations between interclass distance, intra-class distance, and variance

This subsection uses the RKHS-based MMD and leverages the kernel trick to efficiently compute the interclass distance, intra-class distance, and variation between samples from the two domains. Moreover, it demonstrates that when using the Gaussian kernel-based MMD, the inter-class distance can be decomposed into their respective intraclass distances and variations.

Definition 3.1. Interclass distance.

The square of the interclass distance between the samples from the source domain and the target domain is defined as ${S}_{inter} = {\sum }_{c = 1}^{C}{\left(S\right)}_{inter}^{c}$, where ${{\left(S\right)}_{inter}^{c} = \left({S}_{st}\right)}_{inter}^{c}$ is the square of the interclass distance (or MMD) between the samples of class c from the source domain and the target domain, as shown in (19), which is derived as the forms in (20) and (21).

Definition 3.2. Intraclass distance.

The square of the intra-class distance between the samples from the source domain and the target domain is defined as ${S}_{intra} = {\sum }_{c = 1}^{C}{\left(S\right)}_{intra}^{c}$, where ${\left(S\right)}_{intra}^{c} = {\left({S}_{st}\right)}_{intra}^{c}$ is the square of the intraclass distance of the samples of class c from the two domains, as defined in (22).

Definition 3.3. Variance.

The joint variance of the samples from the source domain and the target domain is defined as ${S}_{var} = {\sum }_{c = 1}^{C}{\left(S\right)}_{var}^{c}$, where ${{\left(S\right)}_{var}^{c} = \left({S}_{st}\right)}_{var}^{c}$ is the joint variance of the samples of class c from the source domain and the target domain, as shown in (23).

The square of the MMD between samples ${X}_{s}$ and ${X}_{t}$ from the source domain and the target domain, respectively is defined using (24) or derived as (25). Conceptually, this is equivalent to treating the samples from both domains as belonging to the same class and computing the interclass distance. This can be expressed as ${\left(S\right)}_{inter}^{o} = {{\left({S}_{st}\right)}_{inter}^{o} = \left(MMD\left({X}_{s}, {X}_{t}\right)\right)}^{2}$ or ${\left({MMD}_{u}\left({X}_{s}, {X}_{t}\right)\right)}^{2}$ when unbiased estimation is employed, as demonstrated in formulas (26) and (27), respectively.

Theorem 3.1. The square of the interclass distance equals the data variance minus the square of the intra-class distance; that is, ${S}_{inter} = {S}_{var}-{S}_{intra}$.

Proof. For ${\left(S\right)}_{inter}^{c} = \frac{{n}_{s}^{c}+{n}_{t}^{c}}{{n}_{s}^{c}{n}_{t}^{c}}\left({n}_{s}^{c}k\left({m}_{s}^{c}-{m}_{st}^{c}, {m}_{s}^{c}-{m}_{st}^{c}\right)+{n}_{t}^{c}k({m}_{t}^{c}-{m}_{st}^{c}, {m}_{t}^{c}-{m}_{st}^{c})\right)$, ${\left(S\right)}_{var}^{c} = \frac{{n}_{s}^{c}+{n}_{t}^{c}}{{n}_{s}^{c}{n}_{t}^{c}}{\sum }_{{x}_{i}\in {X}_{st}^{c}}^{}k\left({x}_{i}-{m}_{st}^{c}, {x}_{i}-{m}_{st}^{c}\right)$, and ${\left(S\right)}_{intra}^{c} = \frac{{n}_{s}^{c}+{n}_{t}^{c}}{{n}_{s}^{c}{n}_{t}^{c}}\left({\sum }_{{x}_{i}\in {X}_{s}^{c}}^{}k\left({x}_{i}-{m}_{s}^{c}, {x}_{i}-{m}_{s}^{c}\right)+{\sum }_{{x}_{j}\in {X}_{t}^{c}}^{}k\left({x}_{j}-{m}_{t}^{c}, {x}_{j}-{m}_{t}^{c}\right)\right)$, it is sufficient to prove that ${\left(S\right)}_{inter}^{c}+{\left(S\right)}_{intra}^{c} = {\left(S\right)}_{var}^{c}$ or that ${n}_{{s}_{d}}^{c}k\left({m}_{{s}_{d}}^{c}-{m}^{c}, {m}_{{s}_{d}}^{c}-{m}^{c}\right)+{\sum }_{{x}_{i}\in {X}_{{s}_{d}}^{c}}^{}k\left({x}_{i}-{m}_{{s}_{d}}^{c}, {x}_{i}-{m}_{{s}_{d}}^{c}\right) = {\sum }_{{x}_{i}\in {X}_{{s}_{d}}^{c}}^{}k\left({x}_{i}-{m}^{c}, {x}_{i}-{m}^{c}\right)$ for $1\le c\le C$ and ${s}_{d}\in \left\{s, t\right\}$. Since ${n}_{{s}_{d}}^{c}k\left({m}_{{s}_{d}}^{c}-{m}^{c}, {m}_{{s}_{d}}^{c}-{m}^{c}\right) = {\sum }_{{x}_{i}\in {X}_{{s}_{d}}^{c}}^{}k\left({m}_{{s}_{d}}^{c}-{m}^{c}, {m}_{{s}_{d}}^{c}-{m}^{c}\right) = {\sum }_{{x}_{i}\in {X}_{{s}_{d}}^{c}}^{}\left(k\left({m}_{{s}_{d}}^{c}, {m}_{{s}_{d}}^{c}\right)+k\left({m}^{c}, {m}^{c}\right)-2k\left({m}_{{s}_{d}}^{c}, {m}^{c}\right)\right) = {\sum }_{{x}_{i}\in {X}_{{s}_{d}}^{c}}^{}\left(k\left({m}_{{s}_{d}}^{c}, {m}_{{s}_{d}}^{c}\right)+k\left({m}^{c}, {m}^{c}\right)-2k\left({x}_{i}, {m}^{c}\right)\right)$ and ${\sum }_{{x}_{i}\in {X}_{{s}_{d}}^{c}}^{}k\left({x}_{i}-{m}_{{s}_{d}}^{c}, {x}_{i}-{m}_{{s}_{d}}^{c}\right) = {\sum }_{{x}_{i}\in {X}_{{s}_{d}}^{c}}^{}\left(k\left({x}_{i}, {x}_{i}\right)+k\left({m}_{{s}_{d}}^{c}, {m}_{{s}_{d}}^{c}\right)-2k\left({x}_{i}, {m}_{{s}_{d}}^{c}\right)\right) = {\sum }_{{x}_{i}\in {X}_{{s}_{d}}^{c}}^{}\left(k\left({x}_{i}, {x}_{i}\right)+k\left({m}_{{s}_{d}}^{c}, {m}_{{s}_{d}}^{c}\right)-2k\left({m}_{{s}_{d}}^{c}, {m}_{{s}_{d}}^{c}\right)\right) = {\sum }_{{x}_{i}\in {X}_{{s}_{d}}^{c}}^{}\left(k\left({x}_{i}, {x}_{i}\right)-k\left({m}_{{s}_{d}}^{c}, {m}_{{s}_{d}}^{c}\right)\right)$, we have ${n}_{{s}_{d}}^{c}k\left({m}_{{s}_{d}}^{c}-{m}^{c}, {m}_{{s}_{d}}^{c}-{m}^{c}\right)+{\sum }_{{x}_{i}\in {X}_{{s}_{d}}^{c}}^{}k\left({x}_{i}-{m}_{{s}_{d}}^{c}, {x}_{i}-{m}_{{s}_{d}}^{c}\right) =$ ${\sum }_{{x}_{i}\in {X}_{{s}_{d}}^{c}}^{}\left(k\left({m}_{{s}_{d}}^{c}, {m}_{{s}_{d}}^{c}\right)+k\left({m}^{c}, {m}^{c}\right)-2k\left({x}_{i}, {m}^{c}\right)+k\left({x}_{i}, {x}_{i}\right)-k\left({m}_{{s}_{d}}^{c}, {m}_{{s}_{d}}^{c}\right)\right) = {\sum }_{{x}_{i}\in {X}_{{s}_{d}}^{c}}^{}k\left({x}_{i}-{m}^{c}, {x}_{i}-{m}^{c}\right).$ This completes the proof.

3.3.   Discriminative class-wise loss

Theorem 3.1 indicates that minimizing the interclass distance is equivalent to minimizing their variation, while simultaneously maximizing the intra-class distance, thus reducing feature discriminativeness. To address this, the strategy proposed by Wang et al. ^[33] was adopted with a trade-off parameter $\beta$ $(-1\le \beta \le 1)$ introduced to adjust the hidden intra-class distance within ${S}_{inter}$, resulting in the formulation of the discriminative class-level loss function, denoted as ${L}_{dcwmmd}^{}$ in formula (28), and its expansion is given in formula (29).

The terms ${\left({S}_{st}\right)}_{inter}^{c}$, ${\left({S}_{st}\right)}_{intra}^{c}$, and ${\left({S}_{st}\right)}_{var}^{c}$, defined in Definitions 3.1 to 3.3, can be expressed in terms of individual sample representations ${x}_{j}\text{'}s$ using formulas (30) to (32). To ensure unbiased estimation and calculate deviations in the feature space, this study uses the loss function ${L}_{dcwmmd}^{u}$, represented by the feature representations ${z}_{j}\text{'}s$ of the samples ${x}_{j}\text{'}s$, as shown in (28). By setting ${\alpha }_{1} = \frac{\left(\beta +1\right)\left({n}_{s}^{c}+{n}_{t}^{c}\right)}{{n}_{s}^{c}{n}_{t}^{c}}$, ${\alpha }_{2} = -\left(\frac{{n}_{s}^{c}+\left({n}_{s}^{c}+{n}_{t}^{c}\right)\beta }{{\left({n}_{s}^{c}\right)}^{2}{n}_{t}^{c}}\right)$, ${\alpha }_{3} = -\left(\frac{{n}_{t}^{c}+\left({n}_{s}^{c}+{n}_{t}^{c}\right)\beta }{{n}_{s}^{c}{\left({n}_{t}^{c}\right)}^{2}}\right)$, ${\alpha }_{4} = -\left(\frac{2}{{n}_{s}^{c}{n}_{t}^{c}}\right)$, ${\gamma }_{1} = \frac{1}{{n}_{s}\left({n}_{s}-1\right)}$, ${\gamma }_{2} = \frac{1}{{n}_{t}\left({n}_{t}-1\right)}$, and ${\gamma }_{3} = -\frac{2}{{n}_{s}{n}_{t}}$, the simplified form of ${L}_{dcwmmd}^{u}$ is given in formula (34). During the training process, these scalar values can be precomputed and stored, eliminating the need for subsequent recalculation.

A categorical cross-entropy, ${L}_{cls}$, is commonly used as the error for the classifier's classification results on the source domain data, as shown in formula (35). Here, ${\widehat{\mathcal{l}}}_{sj}^{c}$ is the c-th element of ${\widehat{\mathcal{l}}}_{si}^{} = \mathbf{C}\left({z}_{i}^{s}\right)$ and ${y}_{si}^{c}$ is the c-th element of the ground truth one-hot label vector ${\mathit{y}}_{\mathit{s}\mathit{i}}^{}$, where ${y}_{si}^{c} = 1$ if the label of the original sample ${x}_{i}^{s}$ corresponding to ${z}_{i}^{s}$ is c, and ${y}_{si}^{c} = 0$ otherwise. To encourage the samples to form dense, uniform, and well-separated clusters, the label-smoothing (LS) technique ^[42] is applied to the cross-entropy loss. This involves substituting the smooth label $\left(1-\alpha \right){y}_{si}^{c}+\alpha /C$, a weighted average of ${y}_{si}^{c}$ and $1/C$, with ${y}_{si}^{c}$ in the categorical cross-entropy to form the smoothed categorical cross-entropy ${L}_{cls}^{ls}\left({Z}_{s}, {Y}_{s}\right)$, as shown in (36). Here, $\alpha$ is a smoothing factor generally set to 0.1 for better performance and $C$ represents the number of classes. The goal of LS is to prevent the model from becoming too confident in its predictions and to reduce overfitting. Rafael Müller et al. ^[42] have shown that LS encourages the representations of training examples from the same class to group in tight clusters.

During training with target samples, the entropy of predicted results for those samples is minimized, as illustrated in formula (37). This strategy is employed because it has been indicated that unlabeled examples are especially beneficial when class overlap is small ^[43]. Minimizing this entropy encourages the predicted results to be more inclined toward a specific category, making the feature distribution between categories in the target domain more distinct and explicit. The training loss function of the entire network is defined as ${\mathcal{L}}_{DCWDA}$, as shown in formula (38), where ${\omega }_{1}$ and ${\omega }_{2}$ are weight parameters.

The training architecture of the proposed discriminative class-wise domain adaptation (DCWDA) system, as shown in Figure 2, consists of a feature extractor ($\mathbf{F}$) used for extracting domain-invariant features and a classifier (C). The feature extractor $\mathbf{F}$ and the classifier C are duplicated to represent the data paths of the source domain and the target domain, and a dotted line is drawn in the middle to indicate shared weights. During training, the source domain samples ${x}_{s}$ and target domain samples ${x}_{t}$ are first separately input into the feature extractor $\mathbf{F}$, which outputs features ${z}_{s} = \mathbf{F}\left({x}_{s}\right)$ and ${z}_{t} = \mathbf{F}\left({x}_{t}\right)$. The discriminative class-wise loss function ${L}_{dcwmmd}^{u}$ is then computed for ${z}_{s}$ and ${z}_{t}$. Subsequently, ${z}_{s}$ and ${z}_{t}$ are separately input into the classifier C, producing classification results ${\widehat{\mathcal{l}}}_{s} = \mathbf{C}\left({z}_{s}\right)$ and ${\widehat{\mathcal{l}}}_{t} = \mathbf{C}\left({z}_{t}\right)$. This allows the calculation of the cross-entropy ${L}_{cls}$ for the predicted result for the source domain sample ${x}_{s}$ and the entropy ${L}_{ent}$ for the predicted result of the target domain sample ${x}_{t}$. The training algorithm is shown in Algorithm 1, where the batch sizes of both the source sample and the target sample are set to N.

4.   Experimental results

The proposed method was evaluated using digit datasets and office object data. The digit datasets used in the experiments include modified national institute of standards and technology database (MNIST) ^[44], U.S. postal service (USPS) ^[45], and street view house numbers (SVHN) ^[46]. MNIST and USPS are handwritten datasets. MNIST has 60, 000 training samples and 10, 000 testing samples, all grayscale images of size 28×28. USPS consists of 9, 298 grayscale images of size 16x16. SVHN contains 73, 257 training images and 26, 032 test images, which are color images of size 32×32 captured from street-view house number photo images. For each image, the digit to be recognized are a single digit in a house number located in the center of the image, surrounded by other digits or distracting objects. In the experiment, the images are scaled to a size of 32×32 pixels. Figure 2 shows some images from MNIST, USPS, and SVHN, and the image in each blue frame is used as a training sample. Figure 3 displays some images from MNIST, USPS, and SVHN, where the numbers within blue frames in SVHN images are the digits to be recognized. The Office-31 ^[47] dataset comprises three domains: Amazon (A), DSLR (Digital Single - Lens Reflex) (D), and Webcam (W). Each domain comprises 31 object categories in an office environment, totaling 4, 110 images, with varying numbers of images for each category. Figure 4 displays some images from Webcam, DSLR, and Amazon.

In the training process, the batch sizes for the digit dataset and Office-31 dataset are set to 128 and 64, respectively. Resnet-18 and Resnet-50 ^[1] are adopted as the network architectures for the feature extractors on the digit dataset and the Office-31 dataset, respectively. Both architectures undergo fine-tuning with pre-trained ImageNet network parameters. In addition, the pseudo-labels of all target domain training data are updated with the current classifier parameters at the beginning of each epoch.

The accuracies of various combinations of source domain and target domain were evaluated. The combinations for digital datasets include: MNIST to USPS (M → U), USPS to MNIST (U → M), and SVHN to MNIST (S → M). The combinations for Office-31 datasets include: Amazon to DSLR (A → D), Amazon to Webcam (A → W), DSLR to Amazon (D → A), DSLR to Webcam (D → W), Webcam to Amazon (W → A), and Webcam to DSLR (W → D). Table 1 compares the proposed method with various unsupervised domain adaptation methods on the digit datasets, including adversarial discriminative domain adaptation (ADDA) ^[17], adversarial dropout regularization (ADR) ^[48], conditional adversarial domain adaptation (CDAN) ^[49], cycle-consistent adversarial domain adaptation (CyCADA) ^[50], sliced wasserstein discrepancy (SWD) ^[51] and source hypothesis transfer (SHOT) ^[36]. Table 2 compares the proposed method with various unsupervised domain adaptation methods on the Office-31dataset, including: Wang et al. ^[33], deep adaptation networks (DAN) ^[18], domain-adversarial neural network (DANN) ^[16], ADDA ^[17], multi-adversarial domain adaptation (MADA) ^[52], SHOT ^[36], collaborative and adversarial network (CAN) ^[14] and mini-batch dynamic geometric embedding (MDGE) ^[23]. Each accuracy represents the average accuracy rate of three test results. The best-performing methods for each source-to-target combination are highlighted in bold. The "Source-only" category indicates that the classifier is directly trained using the source domain data without domain adaptation, and then tested using the target domain data. The "Target-supervised" category shows that the classifier is directly trained using the target domain data and tested using the target data. Typically, the accuracies of "source-only" and "target supervised" serve as the lower and upper bounds for domain adaptation accuracy, but there's no guarantee that the accuracy will fall within this range.

As can be seen from Table 2, the proposed method outperforms other methods in testing most digital dataset pairs except S → M, and achieves the highest average accuracy. It is worth noting that SVHN images have obvious color changes and noise. Compared with other digital imaging datasets, the USPS is a smaller digital dataset with smaller images. Hence, the test results for the combinations of USPS and SVHN are not very informative. In view of these results, the datasets M → S, S → U, and U → S are not used in the digital dataset experiment. As can be seen from Table 3, the proposed method outperforms other methods in testing two of the three digital dataset combinations and achieves the highest average accuracy.

5.   Discussion and conclusions

In this paper, we tackled the domain adaptation problem by using a deep network architecture with a DCWMMD as a loss function. The MMD used is based on embedding distribution metrics in the reproducing kernel Hilbert space. This not only leverages the kernel trick to enhance computational efficiency but also conforms to the original MMD definition. Marginal MMD helps align the data distributions regardless of class alignment. To alleviate this limitation, CWMMD was introduced to align data distributions of the same class from the two domains. However, this adjustment may lead to a reduction in feature discriminativeness. By deconstructing CWMMD into variance minus intra-class distance, an adjustable weight parameter for the intra-class distance term was introduced, providing flexibility to preserve feature discriminability. The experimental results show that our proposed method improves upon the approach proposed by Wang et al. ^[33]. In terms of the error function, we not only applied the LS technique to the cross entropy for the training of the source domain, but we also added the entropy of the predicted label for the target samples to enhance the overall training performance. The proposed architecture was evaluated using two datasets, the digital dataset and Office-31 dataset. The results demonstrate competitive accuracy rates for domain adaptation when compared to other methods.

In the future, we will continue to improve the performance of training process in the system, such as applying data augmentation to increase the diversity of data, using high-confidence data from the target domain to provide pseudo-labels for supervised post-processing training, etc. Last but not least, we also want to apply our work to other domain adaptation tasks, such as face recognition, object recognition, and image-to-image translation.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

This work was supported by the National Science and Technology Council, Taiwan, R.O.C. under the grant NSTC 112-2221-E-032-041.

Conflict of interest

The authors declare no conflict of interest.