Unsupervised domain adaptation through transferring both the source-knowledge and target-relatedness simultaneously

1.   Introduction

In the field of machine learning and pattern recognition, some tasks are posed with amount of labeled data, while other tasks with abundant unlabeled data. Generally, a discriminant estimator with good generalization is relatively easier to generate for the former tasks since abundant supervision prior is available to train the desired estimator. By contrast, it is very challenging to build a discriminative model with good generalization for the latter tasks since no or very limited discrimination is available. A natural way of handing this problem is to apply the knowledge from those supervised tasks to benefit the unsupervised tasks. However, the data distributions of these tasks are usually not consistent. To overcome such challenges, the learning paradigm of domain adaptation was proposed ^[1].

In domain adaptation, we refer to the dataset with labels as the source domain, and the dataset to be transferred with no labels or few labels as the target domain ^[2,3,4]. In general, the source domain and the target domain have different distributions ^[5,6]. From the perspective of whether target domain contains labeled samples, domain adaptation can be divided into semi-supervised domain adaptation ^[7] and unsupervised domain adaptation (UDA) ^[8]. Compared with semi-supervised domain adaptation, UDA is more challenging to distinguish the transfer relationship from source domain to target domain. In this paper, we mainly study UDA. In order to perform UDA from the source domain to the target domain, three levels of modeling strategy have been proposed, as follows.

Instance-Level Adaptation When the source domain and the target domain share the same distribution, the distribution of the target domain can usually be deduced through the probability distribution of the source domain, so as to complete the transferring on different domains. However, in UDA, the source domain and the target domain usually follow different probability distributions, or the feature weights of samples between the source and the target domain are different, which makes it difficult to directly deduce the data feature distribution matching of the target domain from the source domain. To this end, the instance reweighting strategy was proposed to perform UDA for such scenarios ^{[9,10,11,12,13,14,15]}. Such strategy typically resamples samples from the source domain, weight them to match the target domain, and then combine them with the target samples to train estimators on the target domains.

Feature-Level Adaptation In many application scenarios, the feature representation distributions between the source domain and the target domain are not the same, e.g., heterogeneous feature spaces. To achieve the goal of UDA, the strategy of feature adaptation was presented ^{[16,17,18,19,20]}. Such methods mainly align the distributions of the domains involved through minimizing the feature distribution inconsistency characterized by, for example, Maximum Mean Discrepancy (MMD) or its variants ^[16,17]. Although such UDA strategy has been widely used and achieved promising results, it tends to over-adaptive when the domain feature differences are too large.

Classifier-Level Adaptation When the target domain is not consistent with the source domain, the pattern estimators (e.g., classifier, regressor) trained on the source domain would not be applicable to the target domain. To make the estimators applicable from source domain to the target domains, a common classifier is usually employed to connect these domains ^[21,22], in which how to generate a common classifier from multiple domains is one of the critical issues. To this end, these methods usually gradually classify the samples of the target domain, add those with high confidences into the training set, and in turn update the classifier in iterative manner until convergence. However, one critical drawback is they usually cannot effectively learn all the samples, bringing about not so high estimation accuracy.

According to the numbers of source domain and target domain, we can group UDA methods into three scenarios: single source domain with single target domain (1S1T) ^[23,24,25], single source domain with multi-target domains (1SmT) ^[26] and multi-source domains with single target domain (mS1T), as shown in Figure 1. Although the 1S1T scenarios are widely researched, the target domain usually cannot be completely covered by the single source domain, resulting in the target domain cannot transfer enough knowledge during the process of UDA learning ^[27]. To overcome this shortcoming, the paradigm of mS1T was proposed by combining knowledge from multiple source domains to the single target task ^[28] and achieved better results than the 1S1T methods. All the methods in both 1S1T and mS1T concentrate on tasks with a single target domain. However, if multiple target domains are involved, they will definitely become unapplicable. To this end, the 1SmT modeling strategy has been proposed. One of the representative is the PA-1SmT ^[26], which achieved UDA by transferring knowledge from the source domain to each of the target domains. However, it does not consider the domain shift in distributions when aligning the target domains to the source. To this end, in this paper, we concentrate on 1SmT and proposes a UDA model through transferring both the source-knowledge and target-relatedness, coined as UDA-SKTR for short. In addition, we also present an alternating optimization algorithm to solve the proposed model with convergence guarantee. Overall, the contributions of this paper are three-fold as follows.

$1)$ A 1SmT UDA model, coined as UDA-SKTR for short, is constructed by transferring both the source-knowledge and the target-relatedness, especially through the target-shared transfer component matrix ${\textbf{Q}}$ and the target-individual transfer component matrix ${\textbf{V}}^m$ to reconstruct the each of the target models, which is more flexible to better cater for real UDA scenarios and is solved by a specially designed alternating algorithm with convergence guarantee.

$2)$ Different from existing methods that transfer directly from source domain data, we perform domain knowledge transfer from the source domain projection rather than the source domain data itself, which better protecting the privacy of the source data.

$3)$ Extensive evaluation experiments testify the effectiveness and superiority of the proposed method.

The rest of this paper is organised as follows. Section 2 reviews the works related to this paper. Section 3 elaborates the proposed model and the optimization algorithm, and gives theoretical convergence proof. Section 4 reports experimental evaluations and provides analysis. Finally, Section 5 concludes this paper and gives future research direction.

2.   Related work

In this section, we briefly review several methods mostly related to our work, i.e., SLMC, STC, TSC, TFSC, PA-1SmT. Before introducing these method, some parameters should be defined. Assuming that the source domain is defined as $\{{x_i}, y_i\}_{i = 1}^{n_S}$ and the target domain is defined as $\{x_i\}_{i = 1}^{n_T}$. The joint distribution of the source domain is represented as $P_S(x, y)$. Similarly, the joint distribution for target domain can be defined by $P_T(x, y)$ where $P_T(x, y) \neq P_S(x, y)$. The other parameters in our model are defined in Table 1.

2.1.   Clustering method

Soft Large-Margin Clustering (SLMC) ^[23] is typical clustering method from the viewpoint of label space along the large-margin principle. It combines the advantages of soft label and large-margin clustering and achieves great performance in many scenes of clustering. SLMC enjoys the advantages of large-margin modeling and soft-label assignment. On the one hand, it could obtain the maximum of between-class boundaries. On the other hand, it could capture the data inherent structure by clustering each instance with soft-label assignment. However, The SLMC method suffer from the way of clustering encoding. In addition, as an clustering algorithm, SLMC cannot be suitable for domain adaptation due to the missing source domain. In fact, the instances in SLMC is the target domain indeed which limits the use of the method.

In addition, Self-Taught Clustering (STC) ^[24] performs clustering through co-clustering using both the target instances and massive unlabeled auxiliary instances, so that these unlabeled data desirably affects the target clustering through their shared feature representation. STC makes use of the joint knowledge from both the target domain and the auxiliary samples for co-clustering, so as to obtain better clustering results. However, this method does not apply the correlations between those auxiliary samples and the target samples, meanwhile is not available to multiple domains clustering.

Transfer Spectral Clustering (TSC) ^[29] performs co-clustering for multiple tasks simultaneously by embedding their shared knowledge. In this way, each of the clustering tasks can be improved by the cooperative clustering. However, TSC characterizes the correlations between the tasks, it requires the number of classes in different task domains must be equal, which is not applicable for scenarios where the task domains do not share the same classes.

Additionly, Transfer Fuzzy Subspace Clustering (TFSC) ^[25] implements clustering for both the source and target domains through minimizing the distance gap from the target instances to source domain centers. Although the TFSC method has taken into account the relationships between source and target domains, it is not suitable for handling scenarios with multi-target domains.

2.2.   PA-1SmT

PA-1SmT ^[26] was constructed based on the SLMC model by additionally incorporating cross-domain knowledge transferring terms between the source and target domains, under the assumption that the target domains are covered by the source domain. Although PA-1SmT has made use of the relationships between the source and target domains, it does not consider the domain shift factors and the common information when aligning the target domains to the source domain and the modeling motivation is seemingly conflicting with the fact that the target domain space is subset of the source domain.

2.3.   Deep domain adaptation

Recently, the methods of deep DA ^{[30,31,32,33,34]}, which is combined with deep neural networks, have achieved better performance than traditional shallow methods. For example, Ganin et al.^[30] designed the simplest DANN model, in which three components (label predictor, domain classifier, and feature extractor) are embedded, and they also illustrated the efficacy of their method. In iDANN ^[31], the authors adopted a different strategy and addressed the DA problem incrementally, by adapting the model to a new domain in an iterative manner. In addition, the CORAL ^[32] model was extended with deep nonlinear networks to match alignments between the layer activations. Moreover, DTA^[33] was proposed to leverage adversarial dropout to learn strongly discriminative features by enforcing the cluster assumption. Besides, DeepJDOT ^[34] was designed to align the data representations between the source and target domains while preserving the discriminative information of the domain classifier. However, the above methods focus merely on the 1S1T scenario and does not carry out in-depth exploration from perspective of the 1SmT scenario.

3.   Unsupervised domain adaptation through transferring source-knowledge and target-relatedness

3.1.   Motivation

In the scenario of 1SmT UDA learning, single source domain together with multiple target domains are involved. On the one hand, it definitely needs to transfer knowledge from the source domain to the target domains in the procedure of UDA learning. On the other hand, the target domains typically share some similar characteristics with each other, meaning that relatedness exists among them. As a result, we should perform 1SmT UDA learning for each of the target domains by transferring not only the knowledge from the source domain, but also the relatedness from the other target domains. Similar with other work above, the source domain is defined as $\{x_i, y_i\}_{i = 1}^{n_S}$ and the target domain is defined as $\{x_i^m\}_{i = 1}^{n_T^m}$ where ${n_S}$ means the number of source domain instances and $n_T^m$ means the number of the $m$-th target domain instances. Assume the joint distribution of the source domain instances is represented as $P_S(x, y)$, with $x$ and $y$ denote the instance and its label from the source domain task. Similarly, the joint distribution for the $m$th of $M$ target domains can be defined by $P_T^m(x, y)$. Generally, $P_T^m(x, y) \neq P_T^k(x, y) \neq P_S(x, y), (m\neq k)$. Considering that the class spaces of the target domains are usually a subset of the source domains ^[26], which is equal that $K_T^m \subseteq K_S$ where $K_S$ denote the classes set of the source domain and $\{K_T^m\}_{m = 1}^M$ indicate the classes set of $M$ target domains, we can propose to construct the target model by representing it using the source model. However, the domain shift from the source domain to the target domains may be large. To handle it, we need to introduce a shared matrix to extract the common information and increase the matching flexibility from the source to target domains. In addition, to exploit the relatedness among the target domains, we can establish a common knowledge dictionary to relate the target domains. The complete scheme is shown in Figure 2.

3.2.   Formulation

3.2.1.   Knowledge transfer from source domain to target domains

To perform 1SmT UDA, we obviously need to transfer knowledge from the source domain to the target domains. Considering that the target domains are not consistent with each other and the domain shift from the source to these targets, it is necessary to introduce a target-shared transfer component matrix (denoted as ${\textbf{Q}}$) to extract the common information and increase their matching flexibility. Along this line, we can mathematically formulate the scheme above as

In Eq (3.1), the first term is responsible for knowledge transfer from the source domain ${\textbf{W}}_S$ to the target domains ${\textbf{W}}_T^m$, due to the fact that the target domain space is subset of the source domain. In addition, it may lead to poor matching effect, if we only focus on the specific information. Therefore, in this model, both target-shared transfer component matrix ${\textbf{Q}}$ and target-individual transfer component matrix ${\textbf{V}}^m$ are characterized to construct each of the target models, which focus on both specific and common information, thus improving the classification accuracy in the respective target domain tasks. ${\textbf{W}}_S$ is learned from K-means algorithm. The second term encourages the UDA to select the most related knowledge components from the source domain to the targets. $\alpha$ is a nonnegative parameter to keep a balance between the two terms. In order to prevent degenerated solutions of the target-shared transfer component matrix ${\textbf{Q}}$, we restrict it column-orthogonal by ${{{\textbf{Q}}^T}{\textbf{Q}} = I}$.

3.2.2.   Knowledge transfer between target domains

In the 1SmT scenario, multiple target domains are involved, which frequently exhibit potential promising correlations among them. Such relatedness may contribute to the target models training. To exploit these shared knowledge among these target domains, we propose to establish a over-complete representation dictionary to potentially extract such target-relatedness. Along this line, we can consequently construct the formulation for knowledge transfer between the multi-target domains as follows,

In Eq (3.2), the shared dictionary ${\textbf{D}}$ among the $M$ target domains plays the role of bridging them and exploring their relatedness to facilitate their learning. More specifically, the shared dictionary ${\textbf{D}}$ is established in the targets common space and it is over-complete to cover each of the target domains. That is, the projection matrix ${\textbf{W}}_T^m$ for the $m$th target domain can be recovered by ${\textbf{D}}$ with its reconstruction coefficient ${\textbf{V}}_T^m$. In this way, the potential relatedness among the target domains is integrated into their learning. The second term of Eq (3.2) aims to select the most knowledge components from the dictionary to corresponding target domain.

3.2.3.   Unsupervised domain adaptation by considering both source-knowledge and target-relatedness

Through taking into account both the considerations aforementioned in Sections 3.2.1 and 3.2.2, we achieve incorporating both the source-knowledge and target-relatedness into 1SmT UDA. In consideration that we concentrate on supervised source domain and unsupervised target domains, without loss of generality, we readily take the SLMC objective function for target domain clustering. Eventually, we can consequently build the complete objective function of the UDA model through transferring source-knowledge and target-relatedness, UDA-SKTR for short, as follows,

where $\lambda_1$ to $\lambda_4$ are nonnegative tradeoff parameters. The first two terms are the objective w.r.t. SLMC on the target domains, the third term models target-relatedness among the $M$ target domains, while the fourth term transfers knowledge from the source to the target domains.

3.3.   Remark

At first glance, the proposed UDA-SKTR method (see Eq (3.3)) is seemingly similar to PA-UDA ^[26]. Nevertheless, there are two significant differences between them as follows. On one hand, the source domain model of PA-UDA is modeled to be reconstructed completely by each of the target domains with respective components, which is seemingly conflicting with the fact that the target domain space is subset of the source domain. In contrast, we construct the target models in UDA-SKTR through leveraging knowledge from the source model (see the third term of Eq (3.3)), which is consistent with the UDA setting. On the other hand, the PA-UDA approach relates each of the target models to the source model merely through the target's individual components. By comparison, in UDA-SKTR, each of the target models is constructed by transferring both target-shared (via the target-shared transfer component matrix ${\textbf{Q}}$) and target-specific (via the target-individual transfer component matrix ${\textbf{V}}^m$) knowledge from the source domain model (also see the third term of Eq (3.3)), which is more flexible to better cater for real UDA scenarios.

3.4.   Optimization

We can obviously see that, six variables are involved in Eq (3.3) such that it is marginal convex with each of the variables, i.e., ${\textbf{W}}_T^m, {{\textbf{D}}}, {{\textbf{V}}}_T^m, {{{\textbf{V}}}^m}, {\textbf{Q}}, u_{k, i}^m$. To this end, we establish an alternating optimization algorithm to solve for each of the variables by fixing all the others until the objective converges. The specific steps are as follows:

3.4.1.   Solve ${\textbf{W}}_T^m$ by fixing ${{\textbf{D}}}, {{\textbf{V}}}_T^m, {{{\textbf{V}}}^m}, {\textbf{Q}}, u_{k, i}^m$

Let

Then, calculating the derivative of ${J_{{\textbf{W}}_T^m}}$ w.r.t. ${\textbf{W}}_T^m$ and making it to zero yields:

where $\widehat {\bf{U}}_k^m = {\mathop{\rm diag}\nolimits} \left({{{\left({u_{k, 1}^m} \right)}^2}, \cdots, {{\left({u_{k, N_k^m}^m} \right)}^2}} \right)$ and ${\bf{L}}_k^m \in {\mathbb{R}^{K_T^m \times N_T^m}}$ stands for a matrix with all entities being 0 except for the $k$-th row being 1. From Eq (3.5), we can get the closed-form solution of ${\textbf{W}}_T^m$ as:

where ${\textbf{I}}$ is an identity matrix of proper size.

3.4.2.   Solve $u_{k, i}^m$ by fixing ${\textbf{W}}_T^m, {\bf{D}}, {\bf{V}}_T^m, {{\bf{V}}^m}, {\textbf{Q}}$

When ${\textbf{W}}_T^m, {\textbf{D}}, {\textbf{V}}_T^m, {{\textbf{V}}^m}, {\textbf{Q}}$ are fixed, Eq (3.3) can be equivalently transformed as:

Let

where $\xi _i^m$ represent the Lagrangian multipliers. Then, calculating the derivative of ${J_{u_{k, i}^m}}$ w.r.t. $u_{k, i}^m$ and making it to zero yields

Combining Eq (3.9) with $\sum\limits_{k = 1}^{{K^m}} {u_{k, i}^m} = 1$, we can get the closed-form solution as

3.4.3.   Solve ${{{\textbf{V}}}^m}$ by fixing ${\textbf{W}}_T^m, {{\textbf{D}}}, {{\textbf{V}}}_T^m, {\textbf{Q}}, u_{k, i}^m$

Let

Calculating the derivative of ${J_{{{\textbf{V}}^m}}}$ w.r.t. ${\textbf{V}}^m$ and making it to zero, we can obtain the following analytic solution for ${\textbf{V}}^m$

where ${{\bf{M}}^m}$ is a diagonal matrix whose $i$th diagonal element ${{\textbf{M}}^m}_{(i, i)} = \frac{1}{{2{{\left\| {{{\left({{{\textbf{V}}^{m{\rm{*}}}}} \right)}^i}} \right\|}_2}}}$ in which ${{\textbf{V}}^{m{\rm{*}}}}$ denotes the solution of ${\textbf{V}}^m$ obtained in previous iteration.

3.4.4.   Solve ${\textbf{V}}_T^m$ by fixing ${\textbf{W}}_T^m, {\bf{D}}, {\textbf{V}}^m, {\textbf{Q}}, u_{k, i}^m$

Similar to ${\textbf{V}}^m$, the solution of ${\textbf{V}}_T^m$ can be generated as:

where ${\textbf{M}}_T^m$ is a diagonal matrix whose $i$-th diagonal element ${\textbf{M}}_T^m\left({i, i} \right) = \frac{1}{{2{{\left\| {{{\left({{\textbf{V}}_T^{m{\rm{*}}}} \right)}^i}} \right\|}_2}}}$ in which ${\textbf{V}}_T^{m{\rm{*}}}$ denotes the solution of ${\textbf{V}}_T^m$ updated in last iteration.

3.4.5.   Solve ${{\textbf{D}}}$ by fixing ${\textbf{W}}_T^m, {\textbf{V}}_T^m, {\textbf{V}}^m, {\textbf{Q}}, u_{k, i}^m$

Let

Calculating the derivative of $J_{\textbf{D}}$ w.r.t ${\textbf{D}}$ and letting it to zero yields

3.4.6.   Solve ${\textbf{Q}}$ by fixing ${\textbf{W}}_T^m, {\bf{D}}, {\textbf{V}}_T^m, {\textbf{V}}^m, u_{k, i}^m$

When ${\textbf{W}}_T^m, {\textbf{D}}, {\textbf{V}}_T^m, {\textbf{V}}^m, u_{k, i}^m$ are fixed, Eq (3.3) can be equivalently rewritten as

The objective function of Eq (3.16) is an orthogonal ProCrustes problem ^[35]. To solve it, we first perform SVD on ${ {\sum\limits_{m = 1}^M }\left({{\textbf{W}}_T^m} \left({{\textbf{W}}_S{{\textbf{V}}^m}}\right)^T\right)}$ as

Then, ${\textbf{Q}}$ can be constructed as

We repeat the above six steps alternately until convergence and can eventually generate the optimal solutions of the variables of Eq (3.3). The complete optimization procedure is elaborated in Algorithm 1.

3.5.   Time complexity analysis

The time complexity of UDA-SKTR in Algorithm 1 is mainly consisted of the alternating optimization steps. More specifically, the complexity of Eq (3.6) is ${\mathcal{O}}({d^2}N_T^m + {d^3} + dN_T^mK_T^m)$. The complexity of Eq (3.10) is ${\mathcal{O}}(d{(K_T^m)^2})$. The complexity of Eq (3.12) is ${\mathcal{O}}({{d^2}{K_S} + {d^3}})$. The complexity of Eq (3.13) is ${\mathcal{O}}({d{r^2} + drK_T^m})$. The complexity of Eq (3.15) is ${\mathcal{O}}({\sum_{m = 1}^M {K_T^mdr} + {K_S}dr})$ and the complexity of Eq (3.18) is ${\mathcal{O}}({{d^3}{\rm{ + }}{d^2}{K_S}})$. Assume that Algorithm 1 converges after ${L^{\max}}$ iterations, and let ${N^{\max}}$ and ${K^{\max}}$ denote the maximum sample number and class number of the target domains. Taking into accounts all the time costs, we conclude that the total time complexity in Algorithm 1 is ${\mathcal{O}}({LN_T^{\max}{d^2} + LN_T^{\max}d{{(K_T^{\max})}^3}{\rm{ + }}L{d^3}})$.

3.6.   Convergence analysis

In this section, we provide the convergence analysis for Algorithm 1. For convenience of clarification, we denote the objective value of Eq (3.3), at the $t$-th optimization iteration, as $J({u_{ki}^{m(t)}, {\bf{W}}_T^{m(t)}, {{\textbf{V}}^{m(t)}}, {\textbf{V}}_T^{m(t)}, {{\textbf{D}}^{(t)}}, {{\textbf{Q}}^{(t)}}})$. At the beginning of iteration $t$+1, we fix ${u_{ki}^{m(t)}, {{\textbf{V}}^{m(t)}}, {\textbf{V}}_T^{m(t)}, {{\textbf{D}}^{(t)}}, {{\textbf{Q}}^{(t)}}}$, then Eq (3.3) is convex w.r.t. ${\textbf{W}}_T^m$ and we denote the objective value after solving it as $J\left({u_{ki}^{m(t)}, {{\textbf{W}}}_T^{m(t + 1)}, {{\textbf{V}}^{m(t)}}, {\textbf{V}}_T^{m(t)}, {{\textbf{D}}^{(t)}}, {{\textbf{Q}}^{(t)}}} \right)$, and it holds that

Then, we fix ${{\textbf{W}}}_T^{m(t+1)}, {{\textbf{V}}^{m(t)}}, {\textbf{V}}_T^{m(t)}, {{\textbf{D}}^{(t)}}, {{\textbf{Q}}^{(t)}}$ and then Eq (3.3) is convex w.r.t. $u_{ki}^m$, and we denote the objective value of Eq (3.3) after updating $u_{ki}^m$, based on Eq (3.10), as $J\left(u_{ki}^{m(t+1)}, {{\textbf{W}}}_T^{m(t+1)}\right.$, $\left.{{\textbf{V}}^{m(t)}}, {\textbf{V}}_T^{m(t)}, {{\textbf{D}}^{(t)}}, {{\textbf{Q}}^{(t)}} \right)$, then it holds that

For both ${\textbf{V}}^m$ and ${\textbf{V}}_T^m$, the objective function of Eq (3.3) is convex w.r.t each of them when fixing all the other variables. After updating them based on Eqs (3.12) and (3.13), the overall objective value of Eq (3.3) can be updated, and it respectively holds that

and

Also, Eq (3.3) is convex w.r.t ${\textbf{D}}$ and ${\textbf{Q}}$. After respectively updating them, we similarly have

and

All in all, we can draw the following conclusions

It means that the entire objective value of Eq (3.3) descends with increased iterations. In addition, since the squared Frobenius-norm and the 2, 1-norm of matrices are both non-negative, so that the objective value of Eq (3.3) is totally non-negative and is consequently lower bounded. Moreover, the entire objective of Eq (3.3) is marginal-convex with each of the variables, so the objective value descends monotonously after optimizing each of the variables. Eventually, the value of Eq (3.3) will definitely converge after finite iterations, which concludes the proof.

4.   Experiment

To evaluate the proposed methods, we performed evaluation experiments on both benchmark and real datasets from the viewpoints of performance comparison, parameter analysis and convergence validation.

4.1.   Dataset

We chose the widely used benchmark dataset, i.e., Office + Caltech, as well as real face datasets, i.e., AgeDB, Morph2 and CACD, for UDA evaluation. Moreover, an additional experiment on PIE datasets for evaluating the accuracy on the scene of unequal classes. For the benchmark datasets, the Office dataset includes three different subdatasets, that is Amazon, DSLR and Webcam. These subdatasets share ten common object categories. For Caltech, it is widely used for object recognition. In experiments, we apply the version of their ten classes to generate four different data fields. According to ^[36], we extracted SURF features from these datasets. For convenience, we denote the Amazon dataset as 'A', DSLR as 'D', Webcam as 'W' and Caltech as 'C'. As for the real face datasets, the AgeDB dataset ^[37] contains more than 16,000 face images of 568 people aged between 0 and 100 years. The Morph2 dataset ^[38] is composed of over 55,000 face images aged between 16 and 77 years. The CACD database ^[39] is the largest cross age face dataset, containing more than 160,000 images with face aged from 14 to 62 years. {The PIE database, which is constructed at Carnegie Mellon University in the year 2000, has been proven particularly important to the progression of research in face recognition across a variety of poses and lighting conditions. The PIE datasets include 40,000 photos from 68 people, including 13 postural conditions, 43 lighting conditions and 4 facial expressions for each person. }Image examples of these databases are demonstrated in Figure 3.

4.2.   Setup

In the experiments, we compared the proposed method with the related methods, i.e., SLMC, LSS ^[40], STC, TSC, TFSC, PA-1SmT, MT-MTDA ^[41]. In addition, we took the Normalized Mutual Information (NMI) and the Rand Index (RI) ^[42] criteria as performance measure on the benchmark datasets, while the Mean Absolute Errors (MAE) ^[43] and the Cumulative Scores (CS) ^[44] as evaluation measures on the real face datasets. Moreover, all hyper-parameters were set through five-fold cross-validation which assigned the value of ${\lambda _1}$ to ${\lambda _4}$ in the range of {1e-4, 1e-3, 1e-2, 1e-1, 1, 1e1, 1e2} and the dictionary dimension $r$ in the range of {5, 10, 15, 20, 25}. With these setting, we adopted the feature representations associated with the Office + Caltech datasets for evaluation. For face datasets, we extracted BIF coefficients ^[45] as feature representation from normalized 32 $\times$ 32 face region and reduced their dimensions to 200 through PCA ^[46]. For each evaluation, we run it thirty times with random data partitions.

4.3.   Results and analysis

4.3.1.   On benchmark datasets

We first performed evaluations on the Office and Caltech datasets. Specifically, we took the Amazon and Caltech datasets as source domain while DSLR and Webcam as target domain. Besides, in hte scenario settings, the source domain categories contains the target ones while target domain may not share the same categories necessarily. Therefore, in this experiment, the first four categories in each target domain are selected while all the source samples are intended to experiments. The results are reported in Tables 2 and 3 where 'w/o' means without. Here, the best results are boldfaced, while those underlined have statistical significance after statistical t-test under p-level: 0.05.

From the results in Tables 2 and 3 we can observe the following findings. Firstly, compared with those 1S1T methods, the 1SmT methods achieved much higher NMI and RI results. It shows that performing joint learning among the target domains benefit its UDA performance than single target domain learning. Secondly, in nearly all cases, the proposed UDA-SKTR method generated the best results with statistical significance than the PA-1SmT approach, in terms of both NMI and RI. It demonstrates the solidness and superiority of the proposed method in modeling both source-knowledge and target-relatedness.

Besides, we also evaluated the sensitivity of the hyper-parameters involved in our method. The results are shown in Figures 4 and 5. We can observe the following findings. Firstly, in terms of both NMI and RI, with increased $\lambda_1$, the estimation results gets worse when $\lambda_1 >$ 1e-2. It reason is that excessive weight on the unsupervised clustering on the target domains overwhelms the other terms for knowledge transfer. Secondly, the performance consistently increases with growing values of $\lambda_2$ and $\lambda_3$. It validates the effectiveness and rationality of the proposed model in exploiting knowledge from both the source domain and the target-relatedness. Thirdly, the performance is not sensitive to the value of $\lambda_4$, which therefore can be set to a constant, e.g., 1 in the comparison experiments. Fourthly, as for the dimension $r$ of the target-shared dictionary ${\textbf{D}}$, the estimation results become better with increased $r$ but keep level when $r > 10$. In view of this observation, in practice we can set $r$ to the value that meets the performance reduction point.

4.3.2.   On PIE datasets

For evaluating the accuracy on the scene where the classes in source domain is not equal with target domains, we perform additional experiments on PIE datasets. In this experiment, all the target domains will be selected the first four classes and the source domain will be selected all. The results is shown in Tables 4 and 5.

From Tables 4 and 5 we can find that with the incomplete class involved, our method achieved the best results compared the other method in both NMI and RI. Moreover, when PIE29 is added to the target domains, the results improved a lot, nearly 8% to 10%. The results when PIE07 is thought as target domain don't get notable improvement, which may be because the features are relatively independent. In addition, the sensitivity of parameters are tested in the below. From Figures 6 and 7 we can observe that $\lambda_1$ has the strong sensitivity about the model, while other parameters has weak sensitivity. Not only that, the parameter $r$ show a disorderly sensitivity which shows that dictionary parameters have great influence on the model and need to be adjusted carefully.

4.3.3.   On real human face datasets

We also performed UDA evaluation on the real face datasets, i.e., AgeDB, Morph2 and CACD. For convenience of evaluation, we chose the shared common age range of 16 to 62 years from the three face datasets. Specifically, we set each neighboring five ages as one group, for example, 16–20 years as the first group, 21–25 years as the second group, and so on.

In order to embody the ordering of age, $W_{S}$ would be learned by ridge regression from labeled source data and CA coding was also used on labelling. The results are shown in Table 6 and Figure 9(a). We can observe the following findings. Firstly, the MAE results of both PA-1SmT and the proposed UDA-SKTR with 1SmT setting are correspondingly better than that in 1S1T setting. It shows that exploiting the target-relatedness can brings estimation accuracy improvement to age estimation. Secondly, the proposed method UDA-SKTR consistently achieves the best results with statistical significance among all the methods. These results once again verifies the superiority of the proposed methodology on handling UDA tasks.

We also evaluated the sensitivity of the model hyper-parameters on the cross-database UDA with results shown in Figure 8. Similar findings can be observed as in Figures 4 and 5. It once again verifies the effectiveness and solidness of the proposed methodology in transferring knowledge from both the source and target domains simultaneously.

To empirically evaluate the convergence efficiency of the proposed Algorithm 1, without loss of generality, we conducted evaluations on CACD$\rightarrow${AgeDB, Morph2}. The convergence result is shown in Figure 9(b). We can see that the algorithm converges within ten iterations, which validates its high convergence efficiency.

4.3.4.   t-SNE visualization

To intuitively visualize our method, we provide a visualization of the A$\rightarrow$W, D task on the Office and Caltech datasets. Specifically, we perform t-SNE comparisons with the typical PA-1SmT and UDA-SKTR as shown in Figure 10. We can observe that the source and target samples adapted by PA-1SmT, is not aligned well. In contrast, samples adapted by our method are aligned preferably, which supports the superiority of our UDA-SKTR method to PA-1SmT.

5.   Conclusions

In this paper, we proposed a kind of 1SmT UDA model through transferring both the source-knowledge and target-relatedness, i.e., UDA-SKTR. In this way, not only the supervision knowledge from the source domain but also the potential relatedness among the target domains are simultaneously modeled for exploitation in 1SmT UDA. In addition, we constructed an alternating optimization algorithm to solve the variables of the proposed model with a convergence guarantee. Finally, through extensive experiments on both benchmark and real datasets, we validated the effectiveness and superiority of the proposed method. In the future, we will consider extending the model to more challenging multi-source multi-target (mSmT) scenarios and extend it to practical application, such as disease detection^[47] in medical field and fault detection^[48] in industry.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grant 62176128, the Open Projects Program of State Key Laboratory for Novel Software Technology of Nanjing University under Grant KFKT2022B06, the Fundamental Research Funds for the Central Universities No. NJ2022028, the Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions (PAPD) fund, as well as the Qing Lan Project.

Conflict of interest

All of the authors declare that they have no conflict of interest.