Stochastic three-term conjugate gradient: a tensor correction algorithmic framework for nonconvex optimization in machine learning

Jiazhen Liu; Gonglin Yuan; Jiahuan Tang; Jiazhen Liu; Gonglin Yuan; Jiahuan Tang

doi:10.3934/era.2026311

Electronic Research Archive

2026, Volume 34, Issue 10: 7201-7221. doi: 10.3934/era.2026311

Previous Article Next Article

Research article

Stochastic three-term conjugate gradient: a tensor correction algorithmic framework for nonconvex optimization in machine learning

1.
Postdoctoral Research Workstation, Guangxi Rural Commercial United Bank, Nanning 530022, China
2.
School of Mathematics & Center for Applied Mathematics of Guangxi (Guangxi University) & Post-doctoral Research Station of the First-level Discipline in Mathematics, Guangxi University, Nanning 530004, China

Received: 01 June 2026 Revised: 22 July 2026 Accepted: 24 July 2026 Published: 18 August 2026

In large-scale, nonconvex optimization for machine learning, such as the training of spine large models, the balance between efficient high-order geometric features of nonconvex landscapes identification and computational stability remains challenging. This paper proposes a tensor stochastic three-term conjugate gradient (TSTCG) algorithm to address two issues under a stochastic CG framework, the insufficient curvature awareness of the stochastic gradient descent and the heavy reliance on line searches. By introducing an implicit correction mechanism based on third-order tensor information, the algorithm fuses function values and gradient differences to reconstruct evolution vectors, effectively enhancing search precision in ill-conditioned curvature regions. Simultaneously, leveraging a specially designed three-term denominator structure allows the algorithm to process sufficient descent property, line search reliance reduction, and efficient fixed step-size updates. Finally, the nonconvexity proofs and numerical experiments demonstrate that TSTCG outperforms other mainstream stochastic optimization algorithms in terms of convergence efficiency and stability.
- machine learning,
- nonconvex optimization,
- spine large models,
- tensor,
- three-term conjugate gradient,
- sufficient descent
Citation: Jiazhen Liu, Gonglin Yuan, Jiahuan Tang. Stochastic three-term conjugate gradient: a tensor correction algorithmic framework for nonconvex optimization in machine learning[J]. Electronic Research Archive, 2026, 34(10): 7201-7221. doi: 10.3934/era.2026311

Related Papers:

Abstract

In large-scale, nonconvex optimization for machine learning, such as the training of spine large models, the balance between efficient high-order geometric features of nonconvex landscapes identification and computational stability remains challenging. This paper proposes a tensor stochastic three-term conjugate gradient (TSTCG) algorithm to address two issues under a stochastic CG framework, the insufficient curvature awareness of the stochastic gradient descent and the heavy reliance on line searches. By introducing an implicit correction mechanism based on third-order tensor information, the algorithm fuses function values and gradient differences to reconstruct evolution vectors, effectively enhancing search precision in ill-conditioned curvature regions. Simultaneously, leveraging a specially designed three-term denominator structure allows the algorithm to process sufficient descent property, line search reliance reduction, and efficient fixed step-size updates. Finally, the nonconvexity proofs and numerical experiments demonstrate that TSTCG outperforms other mainstream stochastic optimization algorithms in terms of convergence efficiency and stability.

References

[1]	K. J. Bergen, P. A. Johnson, M. V. de Hoop, G. C. Beroza, Machine learning for data-driven discovery in solid Earth geoscience, Science, 363 (2019), eaau0323. https://doi.org/10.1126/science.aau0323 doi: 10.1126/science.aau0323
[2]	S. Malley, C. Reina, S. Nacy, J. Gilles, B. Koohbor, G. Youssef, Predictability of mechanical behavior of additively manufactured particulate composites using machine learning and data-driven approaches, Comput. Ind., 142 (2022), 103739. https://doi.org/10.1016/j.compind.2022.103739 doi: 10.1016/j.compind.2022.103739
[3]	D. V. Akman, M. Malekipirbazari, Z. D. Yenice, A. Yeo, N. Adhikari, Y. K. Wong, et al., K-best feature selection and ranking via stochastic approximation, Expert Syst. Appl., 213 (2023), 118864. https://doi.org/10.1016/j.eswa.2022.118864 doi: 10.1016/j.eswa.2022.118864
[4]	N. Khan, M. R. Saleem, D. Lee, M. W. Park, C. Park, Utilizing safety rule correlation for mobile scaffolds monitoring leveraging deep convolution neural networks, Comput. Ind., 129 (2021), 103448. https://doi.org/10.1016/j.compind.2021.103448 doi: 10.1016/j.compind.2021.103448
[5]	Y. Ma, F. Rusu, K. Wu, A. Sim, Adaptive stochastic gradient descent for deep learning on heterogeneous CPU+ GPU architectures, in 2021 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), (2021), 6–15. https://doi.org/10.1109/IPDPSW50202.2021.00012
[6]	Y. LeCun, Y. Bengio, G. Hinton, Deep learning, Nature, 521 (2015), 436–444. https://doi.org/10.1038/nature14539 doi: 10.1038/nature14539
[7]	S. P. Awate, R. T. Whitaker, Unsupervised, information-theoretic, adaptive image filtering for image restoration, IEEE Trans. Pattern Anal. Mach. Intell., 28 (2006), 364–376. https://doi.org/10.1109/TPAMI.2006.52 doi: 10.1109/TPAMI.2006.52
[8]	Y. Zhang, M. J. Wainwright, J. C. Duchi, Communication-efficient algorithms for statistical optimization, Adv. Neural Inf. Process. Syst., 25 (2012), 1502–1510.
[9]	I. Goodfellow, Y. Bengio, A. Courville, Y. Bengio, Deep Learning, MIT Press, Cambridge, 2016.
[10]	S. Klein, M. Staring, J. P. Pluim, Evaluation of optimization methods for nonrigid medical image registration using mutual information and B-splines, IEEE Trans. Image Process., 16 (2007), 2879–2890. https://doi.org/10.1109/TIP.2007.909315 doi: 10.1109/TIP.2007.909315
[11]	Z. Allen-Zhu, Y. Yuan, Improved SVRG for non-strongly-convex or sum-of-nonconvex objectives, in International Conference on Machine Learning, (2016), 1080–1089.
[12]	H. Robbins, S. Monro, A stochastic approximation method, Ann. Math. Statist., 22 (1951), 400–407. https://doi.org/10.1214/aoms/1177729586 doi: 10.1214/aoms/1177729586
[13]	L. Bottou, Large-scale machine learning with stochastic gradient descent, in Proceedings of COMPSTAT'2010, (2010), 177–186. https://doi.org/10.1007/978-3-7908-2604-3_16
[14]	R. Johnson, T. Zhang, Accelerating stochastic gradient descent using predictive variance reduction, Adv. Neural Inf. Process. Syst., 26 (2013), 315–323.
[15]	A. Defazio, F. Bach, S. Lacoste-Julien, SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives, Adv. Neural Inf. Process. Syst., 27 (2014), 1646–1654.
[16]	L. M. Nguyen, J. Liu, K. Scheinberg, M. Takáč, SARAH: A novel method for machine learning problems using stochastic recursive gradient, in International Conference on Machine Learning, (2017), 2613–2621.
[17]	J. Duchi, E. Hazan, Y. Singer, Adaptive subgradient methods for online learning and stochastic optimization, J. Mach. Learn. Res., 12 (2011), 2121–2159.
[18]	G. Hinton, N. Srivastava, K. Swersky, Neural Networks for Machine Learning, 2012. Available from: https://www.cs.toronto.edu/tijmen/csc321/slides/lecture_slides_lec6.pdf.
[19]	D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, preprint, arXiv: 1412.6980. https://doi.org/10.48550/arXiv.1412.6980
[20]	S. J. Reddi, S. Kale, S. Kumar, On the convergence of Adam and beyond, preprint, arXiv: 1904.09237. https://doi.org/10.48550/arXiv.1904.09237
[21]	M. Yousefi, Á. Martínez Calomardo, A stochastic modified limited memory BFGS for training deep neural networks, in Science and Information Conference, (2022), 9–28. https://doi.org/10.1007/978-3-031-10464-0_2
[22]	M. R. Hestenes, E. Stiefel, Methods of conjugate gradients for solving linear systems, J. Res. Natl. Bur. Stand., 49 (1952), 409–436. https://doi.org/10.6028/jres.049.044 doi: 10.6028/jres.049.044
[23]	R. Fletcher, C. M. Reeves, Function minimization by conjugate gradients, Comput. J., 7 (1964), 149–154. https://doi.org/10.1093/comjnl/7.2.149 doi: 10.1093/comjnl/7.2.149
[24]	E. Polak, G. Ribiere, Note sur la convergence de méthodes de directions conjuguées, Rev. Fr. Inf. Rech. Oper., 3 (1969), 35–43. https://doi.org/10.1051/m2an/196903R100351 doi: 10.1051/m2an/196903R100351
[25]	B. T. Polyak, The conjugate gradient method in extremal problems, USSR Comput. Math. Math. Phys., 9 (1969), 94–112. https://doi.org/10.1016/0041-5553(69)90035-4 doi: 10.1016/0041-5553(69)90035-4
[26]	N. N. Schraudolph, T. Graepel, Combining conjugate direction methods with stochastic approximation of gradients, in Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics, (2003), 248–253.
[27]	H. Jiang, P. Wilford, A stochastic conjugate gradient method for the approximation of functions, J. Comput. Appl. Math., 236 (2012), 2529–2544. https://doi.org/10.1016/j.cam.2011.12.012 doi: 10.1016/j.cam.2011.12.012
[28]	X. B. Jin, X. Y. Zhang, K. Huang, G. G. Geng, Stochastic conjugate gradient algorithm with variance reduction, IEEE Trans. Neural Netw. Learn. Syst., 30 (2018), 1360–1369. https://doi.org/10.1109/TNNLS.2018.2868841 doi: 10.1109/TNNLS.2018.2868841
[29]	C. Kou, H. Yang, A mini-batch stochastic conjugate gradient algorithm with variance reduction, J. Global Optim., 87 (2023), 1009–1025. https://doi.org/10.1007/s10898-023-01306-0 doi: 10.1007/s10898-023-01306-0
[30]	J. Z. Zhang, N. Y. Deng, L. Chen, New quasi-Newton equation and related methods for unconstrained optimization, J. Optim. Theory Appl., 102 (1999), 147–167. https://doi.org/10.1023/A:1021798319692 doi: 10.1023/A:1021798319692
[31]	A. Bouaricha, Tensor methods for large, sparse unconstrained optimization, SIAM J. Optim., 7 (1997), 732–756. https://doi.org/10.1137/S105262349427218X doi: 10.1137/S105262349427218X
[32]	J. Zhang, C. Xu, Properties and numerical performance of quasi-Newton methods with modified quasi-Newton equations, J. Comput. Appl. Math., 137 (2001), 269–278. https://doi.org/10.1016/S0377-0427(00)00713-1 doi: 10.1016/S0377-0427(00)00713-1
[33]	F. Biglari, M. A. Hassan, W. J. Leong, New quasi-Newton methods via higher order tensor models, J. Comput. Appl. Math., 235 (2011), 2412–2422. https://doi.org/10.1016/j.cam.2010.10.040 doi: 10.1016/j.cam.2010.10.040
[34]	L. Zhang, W. Zhou, D. Li, Some descent three-term conjugate gradient methods and their global convergence, Optim. Methods Softw., 22 (2007), 697–711. https://doi.org/10.1080/10556780600812232 doi: 10.1080/10556780600812232
[35]	K. Sugiki, Y. Narushima, H. Yabe, Globally convergent three-term conjugate gradient methods that use secant conditions and generate descent search directions for unconstrained optimization, J. Optim. Theory Appl., 153 (2012), 733–757. https://doi.org/10.1007/s10957-011-9964-1 doi: 10.1007/s10957-011-9964-1
[36]	M. Kelly, R. Landreville, K. Nottingham, The UCI Machine Learning Repository, 2023. Available from: https://archive.ics.uci.edu.
[37]	C. C. Chang, C. J. Lin, LIBSVM: A library for support vector machines, ACM Trans. Intell. Syst. Technol., 2 (2011), 1–27. https://doi.org/10.1145/1961189.1961199 doi: 10.1145/1961189.1961199

Reader Comments

Your name:*

Email:*
© 2026 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)