Physics-informed neural networks (PINNs) have become a powerful meshless technique for solving partial differential equations (PDEs). However, the network architecture and the selection of hyperparameters significantly influence their performance. The majority of existing research works are based on heuristic methods for hyperparameter tuning. This often results in lower accuracy and unstable convergence, especially for nonlinear stiff PDEs. In this study, we address this limitation by introducing a comprehensive and systematic tune-first PINN framework. We employ the tree-structured Parzen estimator (TPE) within a Bayesian optimization scheme to fine-tune key hyperparameters, including network depth, width, collocation points, weights associated with PDE and initial conditions, Fourier features, and learning rates. The optimized configuration is subsequently used for high-fidelity PINN training. We validate the proposed methodology using challenging one- and two-dimensional nonlinear, stiff PDEs. Numerical results indicate that systematic tuning of PINN architecture improves performance by up to 85% compared to a standard PINN with manually selected hyperparameters. These findings underscore that hyperparameter optimization provides a reliable and reproducible approach for improving the accuracy and stability of PINNs, and it is adaptable to higher-dimensional problems and other stiff nonlinear PDEs.
Citation: Safeer Hamza Tahir, Sadia Saeed, Reham Alahmadi, Saifudin Hafiz Yahaya, Umair Ali. Improving physics-informed neural networks performance with tree-structured Parzen estimator (TPE)-based Bayesian hyperparameters tuning: Application to nonlinear partial differential equations[J]. AIMS Mathematics, 2026, 11(10): 34780-34804. doi: 10.3934/math.20261355
Physics-informed neural networks (PINNs) have become a powerful meshless technique for solving partial differential equations (PDEs). However, the network architecture and the selection of hyperparameters significantly influence their performance. The majority of existing research works are based on heuristic methods for hyperparameter tuning. This often results in lower accuracy and unstable convergence, especially for nonlinear stiff PDEs. In this study, we address this limitation by introducing a comprehensive and systematic tune-first PINN framework. We employ the tree-structured Parzen estimator (TPE) within a Bayesian optimization scheme to fine-tune key hyperparameters, including network depth, width, collocation points, weights associated with PDE and initial conditions, Fourier features, and learning rates. The optimized configuration is subsequently used for high-fidelity PINN training. We validate the proposed methodology using challenging one- and two-dimensional nonlinear, stiff PDEs. Numerical results indicate that systematic tuning of PINN architecture improves performance by up to 85% compared to a standard PINN with manually selected hyperparameters. These findings underscore that hyperparameter optimization provides a reliable and reproducible approach for improving the accuracy and stability of PINNs, and it is adaptable to higher-dimensional problems and other stiff nonlinear PDEs.
| [1] |
A. H. Ganie, L. H. Sadek, M. M. Tharwat, M. A. Iqbal, M. M. Miah, M. M. Rasid, et al., New investigation of the analytical behaviors for some nonlinear PDEs in mathematical physics and modern engineering, Part. Differ. Equal. Appl. Math., 9 (2024), 100608. https://doi.org/10.1016/j.padiff.2023.100608 doi: 10.1016/j.padiff.2023.100608
|
| [2] |
K. Velten, D. M. Schmidt, K. Kahlen, Mathematical modeling and simu lation: Introduction for scientists and engineers, John Wiley & Sons, 2024. https://doi.org/10.1002/9783527849604 |
| [3] |
A. Shah, S. Ayub, M. Sohaib, S. Saeed, S. A. Khan, S. Abbas, et al., An operator splitting scheme for numerical simulation of spinodal decomposition and microstructure evolution of binary alloys, Heliyon, 9 (2023). https://doi.org/10.1016/j.heliyon.2023.e16597 |
| [4] |
A. Shah, S. Saeed, S. A. Khan, Numerical investigation of bubbles coales cence in a shear flow with diffuse-interface model, Heliyon, 4 (2018). https://doi.org/10.1016/j.heliyon.2018.e01024 |
| [5] |
W. Barsukow, R. Loubere, P. H. Maire, A node-conservative vorticity pre serving finite volume method for linear acoustics on unstructured grids, Math. Comput., 94 (2025), 2299–2343. https://doi.org/10.1090/mcom/4020 doi: 10.1090/mcom/4020
|
| [6] |
J. N. Reddy, An introduction to the finite element method, In Dynamics of Earth's Fluid System, CRC Press, 2026,199–226. https://doi.org/10.1201/9781003762133-22 |
| [7] |
S. A. Khan, A. Shah, S. Saeed, Numerical simulation of the interaction between three equal-sized rising bubbles using the phase-field method, AIP Adv., 10 (2020). https://doi.org/10.1063/1.5144963 |
| [8] |
A. Shah, S. Saeed, L. Yuan, An artificial compressibility method for 3D phase-field model and its application to two-phase flows, Int. J. Comput. Method., 14 (2017), 1750059. https://doi.org/10.1142/S0219876217500591 doi: 10.1142/S0219876217500591
|
| [9] |
Z. Yang, J. Sun, Y. Yang, Y. Chai, Y. Liu, Rapid screening strategy of 2D materials for li-ion battery (lib) electrode based on deep neural networks (DNN), ACS Appl. Energ. Mater., 8 (2025), 3058–3065. https://doi.org/10.1021/acsaem.4c03209 doi: 10.1021/acsaem.4c03209
|
| [10] |
Z. Zhou, Y. Wang, G. Zhou, X. Liu, M. Wu, K. Dai, Vehicle lateral dynamics-inspired hybrid model using neural network for parameter identification and error characterization, IEEE T. Veh. Technol., 73 (2024), 16173–16186. https://doi.org/10.1109/TVT.2024.3416317 doi: 10.1109/TVT.2024.3416317
|
| [11] |
M. D. Kumar, C. S. K. Raju, K. Sajjan, G. Dharmaiah, N. A. Shah, S. J. Yook, Deep neural network-based prediction and computational fluid dynamics analysis of convective heat transfer in dusty fluid flow over heated surface, Phys. Fluids, 37 (2025). https://doi.org/10.1063/5.0250396 |
| [12] |
Z. Cai, X. Lin, T. Liu, Physics-informed neural network for estimating surface heat flux from surface temperature measurements, Int. J. Heat Mass Tran., 253 (2025), 127580. https://doi.org/10.1016/j.ijheatmasstransfer.2025.127580 doi: 10.1016/j.ijheatmasstransfer.2025.127580
|
| [13] |
D. Zhang, L. Lu, L. Guo, G. E. Karniadakis, Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems, J. Comput. Phys., 397 (2019), 108850. https://doi.org/10.1016/j.jcp.2019.07.048 doi: 10.1016/j.jcp.2019.07.048
|
| [14] |
H. Yang, Z. Liu, W. Liu, H. Wang, Y. Zhang, H. Wang, Graph-MDETR: A graph-guided mamba-DETR network for UAV catenary support components detection in electrified railways, IEEE T. Intell. Transp., 27 (2026), 6319–6332. https://doi.org/10.1109/TITS.2026.3680498 doi: 10.1109/TITS.2026.3680498
|
| [15] |
M. Raissi, P. Perdikaris, G. E. Karniadakis, Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, J. Comput. Phys., 378 (2019), 686–707. https://doi.org/10.1016/j.jcp.2018.10.045 doi: 10.1016/j.jcp.2018.10.045
|
| [16] |
R. Mattey, S. Ghosh, A novel sequential method to train physics informed neural networks for Allen Cahn and Cahn Hilliard equations, Comput. Methods Appl. Mech. Eng., 390 (2022), 114474. https://doi.org/10.1016/j.cma.2021.114474 doi: 10.1016/j.cma.2021.114474
|
| [17] |
H. Sadaf, N. Iftikhar, S. Saeed, Physics-informed neural networks for simulating nonlinear peristaltic flow dynamics: A comparative analysis, Int. Commun. Heat Mass, 169 (2025), 109708. https://doi.org/10.1016/j.icheatmasstransfer.2025.109708 doi: 10.1016/j.icheatmasstransfer.2025.109708
|
| [18] |
A. D. Jagtap, G. E. Karniadakis, Extended physics-informed neural networks (XPINNs): A generalized space-time domain decomposition based deep learning framework for nonlinear partial differential equations, Commun. Comput. Phys., 28 (2020), 2002–2041. https://doi.org/10.4208/cicp.OA-2020-0164 doi: 10.4208/cicp.OA-2020-0164
|
| [19] |
Z. Zhou, Y. Wang, X. Liu, Z. Li, M. Wu, G. Zhou, Hybrid of neural network and physics-based estimator for vehicle longitudinal dynamics modeling using limited driving data, IEEE T. Intell. Transp., 2025, 1–12. https://doi.org/10.1109/TITS.2025.3585346 |
| [20] |
M. Li, L. Li, Z. Zhang, Q. Zhang, A novel method for generating permutation mass functions using permutations to represent class bias, Expert Syst. Appl., 319 (2026), 132004. https://doi.org/10.1016/j.eswa.2026.132004 doi: 10.1016/j.eswa.2026.132004
|
| [21] |
J. Chen, C. Yu, Y. Wang, Z. Zhou, Z. Liu, Hybrid modeling for vehicle lateral dynamics via AGRU with a dual-attention mechanism under limited data, Control Eng. Pract., 151 (2024), 106015. https://doi.org/10.1016/j.conengprac.2024.106015 doi: 10.1016/j.conengprac.2024.106015
|
| [22] |
S. Wang, X. Yu, P. Perdikaris, When and why PINNs fail to train: A neural tangent kernel perspective, J. Comput. Phys., 449 (2022), 110768. https://doi.org/10.1016/j.jcp.2021.110768 doi: 10.1016/j.jcp.2021.110768
|
| [23] |
S. Markidis, The old and the new: Can physics-informed deep-learning replace traditional linear solvers?, Front. Big Data, 4 (2021), 669097. https://doi.org/10.3389/fdata.2021.669097 doi: 10.3389/fdata.2021.669097
|
| [24] |
K. Luo, J. Zhao, Y. Wang, J. Li, J. Wen, J. Liang, et al., Physics informed neural networks for PDE problems: A comprehensive review, Artif. Intell. Rev., 58 (2025), 323. https://doi.org/10.1007/s10462-025-11322-7 doi: 10.1007/s10462-025-11322-7
|
| [25] |
J. Wu, X. Y. Chen, H. Zhang, L. D. Xiong, H. Lei, S. H. Deng, Hyperparameter optimization for machine learning models based on Bayesian optimization, J. Electron. Sci. Technol., 17 (2019), 26–40. https://doi.org/10.1016/j.jnlest.2020.100007 doi: 10.1016/j.jnlest.2020.100007
|
| [26] |
J. Bergstra, Y. Bengio, Random search for hyper-parameter optimization, J. Mach. Learn. Res., 13 (2012), 281305. |
| [27] |
J. Snoek, H. Larochelle, R. P. Adams, Practical Bayesian optimization of machine learning algorithms, Adv. Neural Inform. Proc. Syst., 25 (2012). |
| [28] |
C. E. Rasmussen, C. K. I. Williams, Gaussian processes for machine learning, MIT Press, 2006. https://doi.org/10.7551/mitpress/3206.001.0001 |
| [29] |
Y. Ramdhani, N. K. Hikmawati, R. T. Prasetio, Sentiment analysis on the JMO application using random forest with grid search optimization, In 2025 Tenth International Conference on Informatics and Computing (ICIC), 2025, 1–6. https://doi.org/10.1109/ICIC68054.2025.11309556 |
| [30] |
T. Akiba, S. Sano, T. Yanase, T. Ohta, M. Koyama, Optuna: A next generation hyperparameter optimization framework, In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery data mining, 2019, 2623–2631. https://doi.org/10.1145/3292500.3330701 |
| [31] |
J. Bergstra, R. Bardenet, Y. Bengio, B. Kegl, Algorithms for hyperparameter optimization, Adv. Neural Inform. Proc. Syst., 2011, 24. |
| [32] |
Y. F. Chen, B. Hosseini, H. Owhadi, A. M. Stuart, Solving and learning nonlinear pdes with gaussian processes, J. Comput. Phys., 447 (2021), 110668. https://doi.org/10.1016/j.jcp.2021.110668 doi: 10.1016/j.jcp.2021.110668
|
| [33] |
T. Ohwada, Cole-Hopf transformation as numerical tool for the Burgers equation, Appl. Comput. Math., 8 (2009), 107–113. |
| [34] |
T. A. Driscoll, N. Hale, L. N. Trefethen, Chebfun guide, Pafnuty Publications, Oxford, 2014. |
Figures(11) / Tables(7)
Safeer Hamza Tahir, Sadia Saeed, Reham Alahmadi, Saifudin Hafiz Yahaya, Umair Ali. Improving physics-informed neural networks performance with tree-structured Parzen estimator (TPE)-based Bayesian hyperparameters tuning: Application to nonlinear partial differential equations[J]. AIMS Mathematics, 2026, 11(10): 34780-34804. doi: 10.3934/math.20261355
TPE-based Bayesian optimization architecture
Optuna plots: (a) Optimization history plot, (b) Parallel coordinates plot, (c) Optimization timeline
(a–c) Solution snapshots of the Burgers' equation at $ t = 0, 0.50, 1.0. $ Heatmaps of (d) Reference solution, (e) Standard PINN solution, and (f) Optimized PINN solution. (g) Absolute error of the Standard PINN. (h) Absolute error of the Optimized PINN
(a) Training loss curves of standard and optimized PINN. (b) Prediction vs reference solution. (c) Performance metrics
Optuna plots for one-dimensional Allen-Cahn equation: (a) Optimization history plot, (b) Hyperparameter importance plot, (c) Optuna completed and pruned timeline plot
(a) Standard PINN solution at t = 0, 0.25, 0.5, 0.75, 1.0. (b) Heatmaps of the reference solution. (c) Heatmaps of the standard PINN solution. (d) Absolute error standard PINN
(a) Optimized PINN solution at t = 0, 0.25, 0.5, 0.75, 1.0. (b) Heatmaps of the reference solution. (c) Heatmaps of the optimized PINN solution. (d) Absolute error optimized PINN
(a) Training loss curves of standard and optimized PINN. (b) Performance metrics
Optuna plots for 2D elliptic equation (a) Optimization history plot (b) Parallel coordinate plot (c) Optimization timeline
Nonlinear elliptic two-dimensional equation: (a) Reference solution, (b) Standard PINN solution, (c) Optimized PINN solution, (d) Absolute error standard PINN, (e) Absolute error optimized PINN
(a) Performance metrics of standard and optimized PINN (b) Training loss curves of standard and optimized PINN
DownLoad: