Decent directions generator conjugate gradient method with its application to train a two-layer neural network model

Osman Omer Osman Yousif; Mohammed A. Saleh; Abdulgader Z. Almaymuni; Osman Omer Osman Yousif; Mohammed A. Saleh; Abdulgader Z. Almaymuni

doi:10.3934/math.20251180

AIMS Mathematics

2025, Volume 10, Issue 11: 26844-26866. doi: 10.3934/math.20251180

Previous Article Next Article

Research article Special Issues

Decent directions generator conjugate gradient method with its application to train a two-layer neural network model

1.
Department of Mathematics, Faculty of Mathematical and Computer Science, University of Gezira, Wad Madani, Sudan
2.
Department of Cybersecurity, College of Computer, Qassim University, Saudi Arabia

Received: 12 September 2025 Revised: 25 October 2025 Accepted: 06 November 2025 Published: 19 November 2025
MSC : 65K05, 90C30, 90C56

In the last decades, conjugate gradient methods have gained important applications in various scientific areas due to their low memory requirements and ability to solve problems of high dimensions. When analyzing a conjugate gradient method, the descent property of the search directions is always required, as it ensures that the search for the minimizer is in the correct direction. In this paper, we proposed a conjugate gradient method that always generates descent search directions under all line searches techniques. Moreover, we established the global convergence of the proposed method when it is applied under Wolfe or strong Wolfe line search. At the same time, to show the performance of the proposed method in practical computation, we compared it with other well-known methods and then applied it to train two-layer neural network models. The numerical results show that the proposed method is efficient.
- optimization method,
- conjugate gradient methods,
- sufficient descent property,
- strong Wolfe line search,
- global convergence,
- neural networks
Citation: Osman Omer Osman Yousif, Mohammed A. Saleh, Abdulgader Z. Almaymuni. Decent directions generator conjugate gradient method with its application to train a two-layer neural network model[J]. AIMS Mathematics, 2025, 10(11): 26844-26866. doi: 10.3934/math.20251180

Related Papers:

Abstract

In the last decades, conjugate gradient methods have gained important applications in various scientific areas due to their low memory requirements and ability to solve problems of high dimensions. When analyzing a conjugate gradient method, the descent property of the search directions is always required, as it ensures that the search for the minimizer is in the correct direction. In this paper, we proposed a conjugate gradient method that always generates descent search directions under all line searches techniques. Moreover, we established the global convergence of the proposed method when it is applied under Wolfe or strong Wolfe line search. At the same time, to show the performance of the proposed method in practical computation, we compared it with other well-known methods and then applied it to train two-layer neural network models. The numerical results show that the proposed method is efficient.

References

[1]	P. Wolfe, Convergence conditions for ascent methods, SIAM Rev., 11 (1969), 226–235. https://doi.org/10.1137/1011036 doi: 10.1137/1011036
[2]	P. Wolfe, Convergence conditions for ascent methods. Ⅱ: Some corrections, SIAM Rev., 13 (1971), 185–188. https://doi.org/10.1137/1013035 doi: 10.1137/1013035
[3]	R. Fletcher, C. M. Reeves, Function minimization by conjugate gradients, Compute. J., 7 (1964), 149–154. https://doi.org/10.1093/comjnl/7.2.149 doi: 10.1093/comjnl/7.2.149
[4]	R. Fletcher, Practical method of optimization, New York: Wiley Inter science Publication, 2000. https://doi.org/10.1002/9781118723203
[5]	Y. H. Dai, Y. Yuan, A nonlinear conjugate gradient with strong global convergence properties, SIAM J. Optim., 10 (1999), 177–182. https://doi.org/10.1137/S1052623497318992 doi: 10.1137/S1052623497318992
[6]	M. R. Hestenes, E. Steifel, Method of conjugate gradient for solving linear equations, J. Res. Nat. Bur. Stand., 49 (1952), 409–436. https://doi.org/10.6028/jres.049.044 doi: 10.6028/jres.049.044
[7]	B. T. Polyak, The conjugate gradient method in extreme problems, USSR Comp. Math. Math. Phys., 9 (1969), 94–112. https://doi.org/10.1016/0041-5553(69)90035-4 doi: 10.1016/0041-5553(69)90035-4
[8]	E. Polak, G. Ribiere, Note sur la convergence de directions conjugees, Revue franç aise d'informatique et de recherche opérationnelle. Série rouge, 3 (1969), 35–43.
[9]	Y. Liu, C. Storey, Efficient generalized conjugate gradient algorithms, part 1: theory, J. Comput. Appl. Math., 69 (1991), 129–137. https://doi.org/10.1007/BF00940464 doi: 10.1007/BF00940464
[10]	A. Abdelrahman, O. O. O. Yousif, M. Mogtaba, M. K. Elbahir, Global convergence of nonlinear conjugate gradient coefficients with inexact line search, Sci. J. King Faisal Univ., 22 (2021), 86–91. https://doi.org/10.37575/b/sci/210058 doi: 10.37575/b/sci/210058
[11]	A. Abdelrahman, M. Mohammed, O. O. O. Yousif, M. K. Elbahir, Nonlinear conjugate gradient coefficients with exact and strong Wolfe line searches techniques, J. Math., 2022 (2022), 1383129. https://doi.org/10.1155/2022/1383129 doi: 10.1155/2022/1383129
[12]	O. Omer, M. Mamat, A. Abashar, M. Rivaie, The global convergence properties of a conjugate gradient method, AIP Conf. Proc., 1602 (2014), 286–295. https://doi.org/10.1063/1.4882501 doi: 10.1063/1.4882501
[13]	O. Omer, M. Rivaie, M. Mamat, Z. Amani, A new conjugate gradient method with sufficient descent without any line search for unconstrained optimization, AIP Conf. Proc., 1643 (2015), 602–608. https://doi.org/10.1063/1.4907500 doi: 10.1063/1.4907500
[14]	M. Rivaie, M. Mamat, L. W. June, I. Mohd, A new class of nonlinear conjugate gradient coefficient with global convergence properties, Appl. Math. Comput., 218 (2012), 11323–11332. https://doi.org/10.1016/j.amc.2012.05.030 doi: 10.1016/j.amc.2012.05.030
[15]	O. Omer, M. Rivaie, M. Mamat, A. Abdalla, A new conjugate gradient method and its global convergence under the exact line search, AIP Conf. Proc., 1635 (2014), 639–646. https://doi.org/10.1063/1.4903649 doi: 10.1063/1.4903649
[16]	Z. Dai, F. Wen, A modified CG-DESCENT method for unconstrained optimization, J. Comput. Appl. Math., 235 (2011), 3332–3341. https://doi.org/10.1016/j.cam.2011.01.046 doi: 10.1016/j.cam.2011.01.046
[17]	O. O. O. Yousif, R. Ziadi, M. A. Saleh, A. Z. Almaymuni, Another updated parameter for the Hestenes-Stiefel conjugate gradient method, Int. J. Anal. Appl., 23 (2025), 10. https://doi.org/10.28924/2291-8639-23-2025-10 doi: 10.28924/2291-8639-23-2025-10
[18]	O. O. O. Yousif, M. A. Saleh, Another modified version of RMIL conjugate gradient method, Appl. Nume. Math., 202 (2024), 120–126. https://doi.org/10.1016/j.apnum.2024.04.014 doi: 10.1016/j.apnum.2024.04.014
[19]	D. Touati-Ahmed, C. Storey, Efficient hybrid conjugate gradient techniques, J. Optim. Theory Appl., 64 (1990), 379–397. https://doi.org/10.1007/BF00939455 doi: 10.1007/BF00939455
[20]	Y. H. Dai, Y. Yuan, An efficient hybrid conjugate gradient method for unconstrained optimization, Ann. Oper. Res., 103 (2001), 33–47. https://doi.org/10.1023/A:1012930416777 doi: 10.1023/A:1012930416777
[21]	N. Andrei, Another hybrid conjugate gradient algorithm for unconstrained optimization, Numer. Algor., 47 (2008), 143–156. https://doi.org/10.1007/s11075-007-9152-9 doi: 10.1007/s11075-007-9152-9
[22]	Z. Meng, B. Keshtegar, Adaptive conjugate single-loop method for efficient reliability-based design and topology optimization, Comput. Methods Appl. Mech. Eng., 344 (2019), 95–119. https://doi.org/10.1016/j.cma.2018.10.009 doi: 10.1016/j.cma.2018.10.009
[23]	G. Zoutendijk, Nonlinear programming computational methods, In: Integer and nonlinear programming, North Hollad, Amsterdam, 1970, 37–86.
[24]	M. J. D. Powell, Convergence properties of algorithm for nonlinear optimization, SIAM Rev., 28 (1986), 487–500. https://doi.org/10.1137/1028154 doi: 10.1137/1028154
[25]	M. Al-Baali, Descent property and global convergence of Fletcher-Reeves method with inexact line search, IMA. J. Numer. Anal., 5 (1985), 121–124. https://doi.org/10.1093/imanum/5.1.121 doi: 10.1093/imanum/5.1.121
[26]	G. Liu, J. Han, H. Yin, Global convergence of the fletcher-reeves algorithm with inexact line search. Appl. Math., 10 (1995), 75–82. https://doi.org/10.1007/BF02663897 doi: 10.1007/BF02663897
[27]	L. Zhang, W. Zhou, D. Li, Global convergence of a modified Fletcher–Reeves conjugate gradient method with Armijo-type line search, Numer. Math., 104 (2006), 561–572. https://doi.org/10.1007/s00211-006-0028-z doi: 10.1007/s00211-006-0028-z
[28]	X. Li, W. Zhang, X. Dong, A class of modified FR conjugate gradient method and applications to non-negative matrix factorization, Comput. Math. Appl., 73 (2017), 270–276. https://doi.org/10.1016/j.camwa.2016.11.017 doi: 10.1016/j.camwa.2016.11.017
[29]	G. Yuan, X. Lu, A modified PRP conjugate gradient method, Ann. Oper. Res., 166 (2009), 73–90. https://doi.org/10.1007/s10479-008-0420-4 doi: 10.1007/s10479-008-0420-4
[30]	J. C. Gilbert, J. Nocedal, Global convergence properties of conjugate gradient methods for optimization, SIAM J. Optim., 2 (1992), 21–42. https://doi.org/10.1137/0802003 doi: 10.1137/0802003
[31]	Z. Wei, G. Li, L. Qi, New nonlinear conjugate gradient formulas for large-scale unconstrained optimization problems, Appl. Math. Comput., 179 (2006), 407–430. https://doi.org/10.1016/j.amc.2005.11.150 doi: 10.1016/j.amc.2005.11.150
[32]	W. W. Hager, H. Zhang, A new conjugate gradient method with guaranteed descent and an efficient line search, SIAM J. Optim., 16 (2005), 170–192. https://doi.org/10.1137/030601880 doi: 10.1137/030601880
[33]	J. Sun, J. Zhang, Global convergence of conjugate gradient methods without line search, Ann. Oper. Res., 103 (2001), 161–173. https://doi.org/10.1023/A:1012903105391 doi: 10.1023/A:1012903105391
[34]	O. O. O. Yousif, A. Abdelrahman, M. Mohammed, M. A. Saleh, A sufficient condition for the global convergence of conjugate gradient methods for solving unconstrained optimisation problems, Sci. J. King Faisal Univ., 23 (2022), 106–112. https://doi.org/10.37575/b/sci/220013 doi: 10.37575/b/sci/220013
[35]	O. O. O. Yousif, M. A. Y. Mohammed, M. A. Saleh, M. K. Elbashir, A criterion for the global convergence of conjugate gradient methods under strong Wolfe line search, J. King Saud Univ.-Sci., 34 (2022), 1–7. https://doi.org/10.1016/j.jksus.2022.102281 doi: 10.1016/j.jksus.2022.102281
[36]	N. Andrei, An unconstrained optimization test functions collection, Adv. Model. Optim., 10 (2008), 147–161.
[37]	E. D. Dolan, J. J. Moré, Benchmarking optimization software with performance profile, Math. Prog., 91 (2002), 201–213. https://doi.org/10.1007/s101070100263 doi: 10.1007/s101070100263
[38]	Z. Sabir, H. A. Wahab, S. Javeed, H. M. Baskonus, An efficient stochastic numerical computing framework for the nonlinear higher order singular models, Fractal Fract., 5 (2021), 176. https://doi.org/10.3390/fractalfract5040176 doi: 10.3390/fractalfract5040176
[39]	Z. Sabir, M. A. Z. Raja, C. M. Khalique, C. Unlu, Neuro-evolution computing for nonlinear multi-singular system of third order Emden–Fowler equation, Math. Comput. Simul., 185 (2021), 799–812. https://doi.org/10.1016/j.matcom.2021.02.004 doi: 10.1016/j.matcom.2021.02.004
[40]	B. O. Wang, J. F. Gomez-Aguilar, Z. Sabir, M. A. Z. Raja, W. F. Xia, H. A. D. I. Jahansh, et al., Numerical computing to solve the nonlinear corneal system of eye surgery using the capability of Morlet wavelet artificial neural networks, Fractals, 30 (2022), 2240147. https://doi.org/10.1142/S0218348X22401478 doi: 10.1142/S0218348X22401478
[41]	I. A. T. Hashem, F. A. Alaba, M. H. Jumare, A. O. Ibrahim, A. W. Abulfaraj, Adaptive stochastic conjugate gradient optimization for backpropagation neural networks, IEEE Access, 12 (2024), 33757–33768. https://doi.org/10.1109/ACCESS.2024.3370859 doi: 10.1109/ACCESS.2024.3370859
[42]	O. O. O. Yousif, R. Ziadi, A. Z. Almaymuni, M. A. Saleh, An improved version of Polak-Ribière-Polyak conjugate gradient method with its applications in neural networks training, Electron. Res. Arch., 33 (2025), 4799–4815. https://doi.org/10.3934/era.2025216 doi: 10.3934/era.2025216

Reader Comments

Your name:*

Email:*
© 2025 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)