Inverse distillation for source-free unsupervised domain adaptation

Di Wu; Hui Jiang; Xing Wei; Junlong Xu; Zhaoxin Ji; Di Wu; Hui Jiang; Xing Wei; Junlong Xu; Zhaoxin Ji

doi:10.3934/era.2026204

Electronic Research Archive

2026, Volume 34, Issue 7: 4626-4647. doi: 10.3934/era.2026204

Previous Article Next Article

Research article

Inverse distillation for source-free unsupervised domain adaptation

Di Wu ^{1,2
,
,},
Hui Jiang ^1,2,
Xing Wei ^{3
,
,},
Junlong Xu ³,
Zhaoxin Ji ³

1.
Key Laboratory of Philosophy and Social Science of Anhui Province on Adolescent Mental Health and Crisis Intelligence Intervention, Hefei Normal University, Hefei 230601, China
2.
School of Computer and Artificial Intelligence, Hefei Normal University, Hefei 230601, China
3.
School of Computer and Information, Hefei University of Technology, Hefei 230601, China

Received: 04 April 2026 Revised: 03 May 2026 Accepted: 20 May 2026 Published: 29 May 2026

Unsupervised domain adaptation (UDA) aims to leverage labeled source domain knowledge to improve the target domain's performance. Source-free domain adaptation (SFDA), a recent research focus, addresses challenges such as data privacy by relying solely on a pretrained source model and unlabeled target data, thus eliminating the need for direct access to source domain data. Although many studies have proposed methods such as generating a source domain, using a proxy source domain, or using pseudo-label training, these approaches directly fine-tune the source model, overlooking the excessive bias towards the source domain data in SFDA. The source domain model contains numerous source domain-specific features, and directly updating it to shift towards the target domain is hindered by these domain-specific features. To address this issue, we propose an inverse distillation-based SFDA method. By constructing an initial target domain model, the method extracts pure target domain features and distills them back into the source model, facilitating a smoother transition towards the target domain. Additionally, it identifies stable and active target samples from both structural and scoring perspectives, applying distinct matching strategies for pseudo-label selection. Extensive experiments and ablation studies on public datasets (Digits, Office-31, Office-Home and VisDA-2017) demonstrate the superior performance of our approach in SFDA tasks.
- source-free domain adaptation,
- knowledge distillation,
- image classification
Citation: Di Wu, Hui Jiang, Xing Wei, Junlong Xu, Zhaoxin Ji. Inverse distillation for source-free unsupervised domain adaptation[J]. Electronic Research Archive, 2026, 34(7): 4626-4647. doi: 10.3934/era.2026204

Related Papers:

Abstract

Unsupervised domain adaptation (UDA) aims to leverage labeled source domain knowledge to improve the target domain's performance. Source-free domain adaptation (SFDA), a recent research focus, addresses challenges such as data privacy by relying solely on a pretrained source model and unlabeled target data, thus eliminating the need for direct access to source domain data. Although many studies have proposed methods such as generating a source domain, using a proxy source domain, or using pseudo-label training, these approaches directly fine-tune the source model, overlooking the excessive bias towards the source domain data in SFDA. The source domain model contains numerous source domain-specific features, and directly updating it to shift towards the target domain is hindered by these domain-specific features. To address this issue, we propose an inverse distillation-based SFDA method. By constructing an initial target domain model, the method extracts pure target domain features and distills them back into the source model, facilitating a smoother transition towards the target domain. Additionally, it identifies stable and active target samples from both structural and scoring perspectives, applying distinct matching strategies for pseudo-label selection. Extensive experiments and ablation studies on public datasets (Digits, Office-31, Office-Home and VisDA-2017) demonstrate the superior performance of our approach in SFDA tasks.

References

[1]	V. K. Kurmi, V. K. Subramanian, V. P. Namboodiri, Domain impression: A source data free domain adaptation method, in 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), (2021), 615–625. https://doi.org/10.1109/WACV48630.2021.00066
[2]	Y. Hou, L. Zheng, Visualizing adapted knowledge in domain transfer, in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2021), 13819–13828. https://doi.org/10.1109/CVPR46437.2021.01361
[3]	Y. Ding, L. Sheng, J. Liang, A. Zheng, R. He, Proxymix: Proxy-based mixup training with label refinery for source-free domain adaptation, Neural Netw., 167 (2023), 92–103. https://doi.org/10.1016/j.neunet.2023.08.005 doi: 10.1016/j.neunet.2023.08.005
[4]	Y. Ganin, V. Lempitsky, Unsupervised domain adaptation by backpropagation, in Proceedings of the 32nd International Conference on Machine Learning, 37 (2015), 1180–1189.
[5]	H. Liu, J. Wang, M. Long, Cycle self-training for domain adaptation, in Advances in Neural Information Processing Systems, 34 (2021), 22968–22981.
[6]	W. Zhou, G. Guan, Y. Yi, W. Cui, Y. Chen, Progressive pseudo-labels enhancement for source-free domain adaptation medical image segmentation, Biomed. Signal Process. Control, 109 (2025), 108053. https://doi.org/10.1016/j.bspc.2025.108053 doi: 10.1016/j.bspc.2025.108053
[7]	M. Long, Y. Cao, J. Wang, M. Jordan, Learning transferable features with deep adaptation networks, in Proceedings of the 32nd International Conference on Machine Learning, 37 (2015), 97–105.
[8]	S. Li, S. Song, G. Huang, Z. Ding, C. Wu, Domain invariant and class discriminative feature learning for visual domain adaptation, IEEE Trans. Image Process., 27 (2018), 4260–4273. https://doi.org/10.1109/TIP.2018.2839528 doi: 10.1109/TIP.2018.2839528
[9]	B. Xu, Z. Zeng, C. Lian, Z. Ding, Few-shot domain adaptation via mixup optimal transport, IEEE Trans. Image Process., 31 (2022), 2518–2528. https://doi.org/10.1109/TIP.2022.3157139 doi: 10.1109/TIP.2022.3157139
[10]	Y. H. Tsai, W. C. Hung, S. Schulter, K. Sohn, M. H. Yang, M. Chandraker, Learning to adapt structured output space for semantic segmentation, in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2018), 7472–7481. https://doi.org/10.1109/CVPR.2018.00780
[11]	Y. Zou, Z. Yu, B. V. K. Kumar, J. Wang, Unsupervised domain adaptation for semantic segmentation via class-balanced self-training, in Proceedings of the European Conference on Computer Vision (ECCV), (2018), 289–305.
[12]	J. Peng, Y. Huang, W. Sun, N. Chen, Y. Ning, Q. Du, Domain adaptation in remote sensing image classification: A survey, IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., 15 (2022), 9842–9859. https://doi.org/10.1109/JSTARS.2022.3220875 doi: 10.1109/JSTARS.2022.3220875
[13]	Y. Huang, J. Peng, W. Sun, N. Chen, Q. Du, Y. Ning, et al., Two-branch attention adversarial domain adaptation network for hyperspectral image classification, IEEE Trans. Geosci. Remote Sens., 60 (2022), 1–13. https://doi.org/10.1109/TGRS.2022.3215677 doi: 10.1109/TGRS.2022.3215677
[14]	Y. Ning, J. Peng, L. Sun, Y. Huang, W. Sun, Q. Du, Adaptive local discriminant analysis and distribution matching for domain adaptation in hyperspectral image classification, IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., 15 (2022), 4797–4808. https://doi.org/10.1109/JSTARS.2022.3181577 doi: 10.1109/JSTARS.2022.3181577
[15]	L. Zha, Y. Chen, P. Zhou, Y. Zhang, Intensifying the consistency of pseudo label refinement for unsupervised domain adaptation person re-identification, in 2023 IEEE International Conference on Multimedia and Expo (ICME), (2023), 1547–1552. https://doi.org/10.1109/ICME55011.2023.00267
[16]	M. Long, H. Zhu, J. Wang, M. I. Jordan, Deep transfer learning with joint adaptation networks, in Proceedings of the 34th International Conference on Machine Learning, 70 (2017), 2208–2217.
[17]	K. Mei, C. Zhu, J. Zou, S. Zhang, Instance adaptive self-training for unsupervised domain adaptation, in Computer Vision–ECCV 2020: 16th European Conference, (2020), 415–430. https://doi.org/10.1007/978-3-030-58574-7_25
[18]	W. Zhang, W. Ouyang, W. Li, D. Xu, Collaborative and adversarial network for unsupervised domain adaptation, in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2018), 3801–3809. https://doi.org/10.1109/CVPR.2018.00400
[19]	Z. Zheng, Y. Yang, Rectifying pseudo label learning via uncertainty estimation for domain adaptive semantic segmentation, Int. J. Comput. Vis., 129 (2021), 1106–1120. https://doi.org/10.1007/s11263-020-01395-y doi: 10.1007/s11263-020-01395-y
[20]	M. Shao, S. Chen, F. Wang, L. Zhang, Adaptive pseudo-label threshold for source-free domain adaptation, Neural Comput. Appl., 37 (2025), 1875–1887. https://doi.org/10.1007/s00521-024-10697-y doi: 10.1007/s00521-024-10697-y
[21]	Z. Qiu, Y. Zhang, H. Lin, S. Niu, Y. Liu, Q. Du, et al., Source-free domain adaptation via avatar prototype generation and adaptation, in Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI-21), (2021), 2921–2927. https://doi.org/10.24963/ijcai.2021/402
[22]	R. Li, Q. Jiao, W. Cao, H. S. Wong, S. Wu, Model adaptation: Unsupervised domain adaptation without source data, in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2020), 9638–9647. https://doi.org/10.1109/CVPR42600.2020.00966
[23]	H. Yan, Y. Guo, C. Yang, Source-free unsupervised domain adaptation with surrogate data generation, in BMVC, (2021), 198. https://doi.org/10.5244/c.35.324
[24]	Y. Du, H. Yang, M. Chen, H. Luo, J. Jiang, Y. Xin, et al., Generation, augmentation, and alignment: A pseudo-source domain based method for source-free domain adaptation, Mach. Learn., 113 (2024), 3611–3631. https://doi.org/10.1007/s10994-023-06432-8 doi: 10.1007/s10994-023-06432-8
[25]	J. Liang, D. Hu, J. Feng, Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation, in Proceedings of the 37th International Conference on Machine Learning, 119 (2020), 6028–6039.
[26]	S. Yang, J. Van de Weijer, L. Herranz, S. Jui, Exploiting the intrinsic neighborhood structure for source-free domain adaptation, in Advances in Neural Information Processing Systems, 34 (2021), 29393–29405.
[27]	H. Xia, H. Zhao, Z. Ding, Adaptive adversarial network for source-free domain adaptation, in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), (2021), 8990–8999. https://doi.org/10.1109/iccv48922.2021.00888
[28]	J. Huang, D. Guan, A. Xiao, S. Lu, Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data, in Advances in Neural Information Processing Systems, 34 (2021), 3635–3649.
[29]	S. Qu, G. Chen, J. Zhang, Z. Li, W. He, D. Tao, General class-balanced multicentric dynamic prototype pseudo-labeling for source-free domain adaptation, Int. J. Comput. Vis., 133 (2025), 3327–3348. https://doi.org/10.1007/s11263-024-02335-w doi: 10.1007/s11263-024-02335-w
[30]	J. Liang, D. Hu, Y. Wang, R. He, J. Feng, Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer, IEEE Trans. Pattern Anal. Mach. Intell., 44 (2021), 8602–8617. https://doi.org/10.1109/tpami.2021.3103390 doi: 10.1109/tpami.2021.3103390
[31]	K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, et al., Fixmatch: Simplifying semi-supervised learning with consistency and confidence, in Advances in Neural Information Processing Systems, 33 (2020), 596–608.
[32]	B. Zhang, Y. Wang, W. Hou, H. Wu, J. Wang, M. Okumura, et al., Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling, in Advances in Neural Information Processing Systems, 34 (2021), 18408–18419.
[33]	C. Chen, W. Xie, W. Huang, Y. Rong, X. Ding, Y. Huang, et al., Progressive feature alignment for unsupervised domain adaptation, in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2019), 627–636. https://doi.org/10.1109/CVPR.2019.00072
[34]	K. Tanwisuth, X. Fan, H. Zheng, S. Zhang, H. Zhang, B. Chen, et al., A prototype-oriented framework for unsupervised domain adaptation, in Advances in Neural Information Processing Systems, 34 (2021), 17194–17208.
[35]	Y. Yang, S. Chen, X. Li, L. Xie, Z. Lin, D. Tao, Inducing neural collapse in imbalanced learning: Do we really need a learnable classifier at the end of deep neural network?, in Advances in Neural Information Processing Systems, 35 (2022), 37991–38002.
[36]	J. Hoffman, E. Tzeng, T. Park, J. Y. Zhu, P. Isola, K. Saenko, et al., Cycada: Cycle-consistent adversarial domain adaptation, in International Conference on Machine Learning, (2018), 1989–1998.
[37]	K. Saenko, B. Kulis, M. Fritz, T. Darrell, Adapting visual category models to new domains, in Computer Vision–ECCV 2010: 11th European Conference, (2010), 213–226. https://doi.org/10.1007/978-3-642-15561-1_16
[38]	H. Venkateswara, J. Eusebio, S. Chakraborty, S. Panchanathan, Deep hashing network for unsupervised domain adaptation, in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2017), 5385–5394. https://doi.org/10.1109/CVPR.2017.572
[39]	X. Peng, B. Usman, N. Kaushik, J. Hoffman, D. Wang, K. Saenko, Visda: The visual domain adaptation challenge, preprint, arXiv: 1710.06924.
[40]	A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, et al., Pytorch: An imperative style, high-performance deep learning library, in Advances in Neural Information Processing Systems, 32 (2019), 1–12.
[41]	K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2016), 770–778. https://doi.org/10.1109/CVPR.2016.90
[42]	J. Deng, W. Dong, R. Socher, L. J. Li, K. Li, L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in 2009 IEEE Conference on Computer Vision and Pattern Recognition, (2009), 248–255. https://doi.org/10.1109/CVPR.2009.5206848
[43]	E. Tzeng, J. Hoffman, K. Saenko, T. Darrell, Adversarial discriminative domain adaptation, in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2017), 2962–2971. https://doi.org/10.1109/CVPR.2017.316
[44]	K. Saito, Y. Ushiku, T. Harada, K. Saenko, Adversarial dropout regularization, preprint, arXiv: 1711.01575.
[45]	Z. Deng, Y. Luo, J. Zhu, Cluster alignment with a teacher for unsupervised domain adaptation, in 2019 IEEE/CVF International Conference on Computer Vision (ICCV), (2019), 9943–9952. https://doi.org/10.1109/ICCV.2019.01004
[46]	C. Y. Lee, T. Batra, M. H. Baig, D. Ulbricht, Sliced wasserstein discrepancy for unsupervised domain adaptation, in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2019), 10277–10287. https://doi.org/10.1109/CVPR.2019.01053
[47]	Y. Kim, D. Cho, K. Han, P. Panda, S. Hong, Domain adaptation without source data, IEEE Trans. Artif. Intell., 2 (2021), 508–518. https://doi.org/10.1109/TAI.2021.3110179 doi: 10.1109/TAI.2021.3110179
[48]	S. Roy, M. Trapp, A. Pilzer, J. Kannala, N. Sebe, E. Ricci, et al., Uncertainty-guided source-free domain adaptation, in Computer Vision – ECCV 2022, (2022), 537–555. https://doi.org/10.1007/978-3-031-19806-9_31
[49]	S. Yang, Y. Wang, L. Herranz, S. Jui, J. Van de Weijer, Casting a BAIT for offline and online source-free domain adaptation, Comput. Vis. Image Underst., 234 (2023), 103747. https://doi.org/10.1016/j.cviu.2023.103747 doi: 10.1016/j.cviu.2023.103747
[50]	T. Chu, Y. Liu, J. Deng, W. Li, L. Duan, Denoised maximum classifier discrepancy for source-free unsupervised domain adaptation, in Proceedings of the AAAI Conference on Artificial Intelligence, 36 (2022), 472–480. https://doi.org/10.1609/aaai.v36i1.19925
[51]	N. Ding, Y. Xu, Y. Tang, C. Xu, Y. Wang, D. Tao, Source-free domain adaptation via distribution estimation, in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2022), 7202–7212. https://doi.org/10.1109/CVPR52688.2022.00707
[52]	F. Wang, Z. Han, Y. Gong, Y. Yin, Exploring domain-invariant parameters for source free domain adaptation, in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2022), 7141–7150. https://doi.org/10.1109/CVPR52688.2022.00701
[53]	J. He, L. Wu, C. Tao, F. Lv, Source-free domain adaptation with unrestricted source hypothesis, Pattern Recognit., 149 (2024), 110246. https://doi.org/10.1016/j.patcog.2023.110246 doi: 10.1016/j.patcog.2023.110246
[54]	M. Jing, J. Li, K. Lu, L. Zhu, H. T. Shen, Visually source-free domain adaptation via adversarial style matching, IEEE Trans. Image Process., 33 (2024), 1032–1044. https://doi.org/10.1109/TIP.2024.3353539 doi: 10.1109/TIP.2024.3353539
[55]	L. Zhou, N. Li, M. Ye, X. Zhu, S. Tang, Source-free domain adaptation with class prototype discovery, Pattern Recognit., 145 (2024), 109974. https://doi.org/10.1016/j.patcog.2023.109974 doi: 10.1016/j.patcog.2023.109974
[56]	S. Yang, Y. Wang, K. Wang, S. Jui, J. Van de Weijer, Attracting and dispersing: A simple approach for source-free domain adaptation, in Advances in Neural Information Processing Systems, 35 (2022), 5802–5815. https://doi.org/10.52202/068431-0420
[57]	J. Lee, D. Jung, J. Yim, S. Yoon, Confidence score for source-free unsupervised domain adaptation, in International Conference on Machine Learning, 162 (2022), 12365–12377.
[58]	L. Yi, G. Xu, P. Xu, J. Li, R. Pu, C. Ling, et al., When source-free domain adaptation meets learning with noisy labels, preprint, arXiv: 2301.13381.
[59]	M. Litrico, A. Del Bue, P. Morerio, Guiding pseudo-labels with uncertainty estimation for source-free unsupervised domain adaptation, in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2023), 7640–7650. https://doi.org/10.1109/cvpr52729.2023.00738
[60]	S. Tang, A. Chang, F. Zhang, X. Zhu, M. Ye, C. Zhang, Source-free domain adaptation via target prediction distribution searching, Int. J. Comput. Vis., 132 (2024), 654–672. https://doi.org/10.1007/s11263-023-01892-w doi: 10.1007/s11263-023-01892-w
[61]	R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, Grad-CAM: Visual explanations from deep networks via gradient-based localization, Int. J. Comput. Vis., 128 (2020), 336–359. https://doi.org/10.1007/s11263-019-01228-7 doi: 10.1007/s11263-019-01228-7

Reader Comments

Your name:*

Email:*
© 2026 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)