An evaluation framework for integrated learning and optimization by revisiting parameters and hyperparameters

Bo Jiang; Xuecheng Tian; Shuaian Wang; Bo Jiang; Xuecheng Tian; Shuaian Wang

doi:10.3934/era.2026293

Electronic Research Archive

2026, Volume 34, Issue 9: 6702-6733. doi: 10.3934/era.2026293

Previous Article Next Article

Research article Special Issues

An evaluation framework for integrated learning and optimization by revisiting parameters and hyperparameters

Faculty of Business, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong 999077, China

Received: 26 April 2026 Revised: 16 July 2026 Accepted: 21 July 2026 Published: 10 August 2026

Integrated learning and optimization (ILO), an emerging paradigm for contextual optimization also known as "smart predict-then-optimize" or "end-to-end optimization", typically presents major challenges for practical applications in the community of operations research and management science (OR/MS). This study developed a general evaluation framework that systematically assesses the difficulty of implementing ILO for contextual optimization problems. It distinguishes between computationally tractable and challenging problems, offering the corresponding exact and approximate solution strategies to obtain the final prescriptions, respectively. Our main contributions are threefold. First, we identified conceptual ambiguities in traditional definitions of parameters and hyperparameters in machine learning (ML) models, proposing new complexity-based definitions that strengthen the theoretical foundations. Second, building on this, we developed optimization frameworks to identify the optimal ML models and prescriptions under sequential learning and optimization (SLO) and ILO, respectively. The ILO evaluation framework was then established, which is structured around several key questions regarding the specific characteristics of the ML model and the downstream optimization problem. Third, we illustrated the applicability of the proposed evaluation framework through a numerical case study on a stochastic binary decision-making problem and provided a cautious discussion of the observed numerical comparison. Moreover, we analyzed and compared the efficacy and efficiency of SLO and ILO. This study provides valuable insights and practical guidance for implementing ILO in solving real-world problems in OR/MS. Furthermore, it enriches the literature on the comparative effectiveness of SLO and ILO.
- machine learning,
- contextual optimization,
- sequential learning and optimization,
- integrated learning and optimization,
- evaluation framework
Citation: Bo Jiang, Xuecheng Tian, Shuaian Wang. An evaluation framework for integrated learning and optimization by revisiting parameters and hyperparameters[J]. Electronic Research Archive, 2026, 34(9): 6702-6733. doi: 10.3934/era.2026293

Related Papers:

Abstract

Integrated learning and optimization (ILO), an emerging paradigm for contextual optimization also known as "smart predict-then-optimize" or "end-to-end optimization", typically presents major challenges for practical applications in the community of operations research and management science (OR/MS). This study developed a general evaluation framework that systematically assesses the difficulty of implementing ILO for contextual optimization problems. It distinguishes between computationally tractable and challenging problems, offering the corresponding exact and approximate solution strategies to obtain the final prescriptions, respectively. Our main contributions are threefold. First, we identified conceptual ambiguities in traditional definitions of parameters and hyperparameters in machine learning (ML) models, proposing new complexity-based definitions that strengthen the theoretical foundations. Second, building on this, we developed optimization frameworks to identify the optimal ML models and prescriptions under sequential learning and optimization (SLO) and ILO, respectively. The ILO evaluation framework was then established, which is structured around several key questions regarding the specific characteristics of the ML model and the downstream optimization problem. Third, we illustrated the applicability of the proposed evaluation framework through a numerical case study on a stochastic binary decision-making problem and provided a cautious discussion of the observed numerical comparison. Moreover, we analyzed and compared the efficacy and efficiency of SLO and ILO. This study provides valuable insights and practical guidance for implementing ILO in solving real-world problems in OR/MS. Furthermore, it enriches the literature on the comparative effectiveness of SLO and ILO.

References

[1]	Z. Hu, M. Xu, Q. Cheng, Multimodal large-language model empowering next-generation autonomous driving systems, J. Intell. Connected Veh., 8 (2025), 9210059. https://doi.org/10.26599/jicv.2025.9210059 doi: 10.26599/jicv.2025.9210059
[2]	Z. Gao, B. Jia, D. Xie, W. Wang, J. Wu, A discussion on the complexity and transit mechanisms of urban traffic systems, Engineering, 44 (2025), 24–29. https://doi.org/10.1016/j.eng.2024.12.006 doi: 10.1016/j.eng.2024.12.006
[3]	U. Sadana, A. Chenreddy, E. Delage, A. Forel, E. Frejinger, T. Vidal, A survey of contextual optimization methods for decision-making under uncertainty, Eur. J. Oper. Res., 320 (2025), 271–289. https://doi.org/10.1016/j.ejor.2024.03.020 doi: 10.1016/j.ejor.2024.03.020
[4]	D. Bertsimas, N. Kallus, From predictive to prescriptive analytics, Manage. Sci., 66 (2020), 1025–1044. https://doi.org/10.1287/mnsc.2018.3253 doi: 10.1287/mnsc.2018.3253
[5]	A. N. Elmachtoub, P. Grigas, Smart "predict, then optimize", Manage. Sci., 68 (2022), 9–26. https://doi.org/10.1287/mnsc.2020.3922
[6]	Y. Deng, S. Sen, Predictive stochastic programming, Comput. Manage. Sci., 19 (2022), 65–98. https://doi.org/10.1007/s10287-021-00400-0 doi: 10.1007/s10287-021-00400-0
[7]	N. Ho-Nguyen, F. Kılınç-Karzan, Risk guarantees for end-to-end prediction and optimization processes, Manage. Sci., 68 (2022), 8680–8698. https://doi.org/10.1287/mnsc.2022.4321 doi: 10.1287/mnsc.2022.4321
[8]	B. Bischl, M. Binder, M. Lang, T. Pielok, J. Richter, S. Coors, et al., Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges, Wiley Interdiscip. Rev.: Data Min. Knowl. Discovery, 13 (2023), e1484. https://doi.org/10.1002/widm.1484 doi: 10.1002/widm.1484
[9]	T. Yu, H. Zhu, Hyper-parameter optimization: A review of algorithms and applications, preprint, arXiv: 2003.05689.
[10]	M. Feurer, F. Hutter, Hyperparameter Optimization, Springer International Publishing, Cham, 2019. https://doi.org/10.1007/978-3-030-05318-5_1
[11]	J. Bergstra, R. Bardenet, Y. Bengio, B. Kégl, Algorithms for hyper-parameter optimization, Adv. Neural Inf. Process. Syst., 24 (2011).
[12]	J. Bergstra, Y. Bengio, Random search for hyper-parameter optimization, J. Mach. Learn. Res., 13 (2012), 281–305.
[13]	J. Snoek, H. Larochelle, R. P. Adams, Practical Bayesian optimization of machine learning algorithms, Adv. Inf. Process. Syst., 25 (2012).
[14]	B. Shahriari, K. Swersky, Z. Wang, R. P. Adams, N. de Freitas, Taking the human out of the loop: A review of Bayesian optimization, Proc. IEEE, 104 (2015), 148–175. https://doi.org/10.1109/JPROC.2015.2494218 doi: 10.1109/JPROC.2015.2494218
[15]	L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, A. Talwalkar, Hyperband: A novel bandit-based approach to hyperparameter optimization, J. Mach. Learn. Res., 18 (2018), 1–52.
[16]	C. Thornton, F. Hutter, H. H. Hoos, K. Leyton-Brown, Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms, in Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, (2013), 847–855. https://doi.org/10.1145/2487575.2487629
[17]	M. Feurer, A. Klein, K. Eggensperger, J. Springenberg, M. Blum, F. Hutter, Efficient and robust automated machine learning, Adv. Neural Inf. Process. Syst., 28 (2015).
[18]	L. Yang, A. Shami, On hyperparameter optimization of machine learning algorithms: Theory and practice, Neurocomputing, 415 (2020), 295–316. https://doi.org/10.1016/j.neucom.2020.07.061
[19]	F. Hutter, H. H. Hoos, K. Leyton-Brown, Sequential model-based optimization for general algorithm configuration, in Proceedings of the International Conference on Learning and Intelligent Optimization, Springer, (2011), 507–523. https://doi.org/10.1007/978-3-642-25566-3_40
[20]	A. Klein, S. Falkner, S. Bartels, P. Hennig, F. Hutter, Fast Bayesian optimization of machine learning hyperparameters on large datasets, in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, PMLR, (2017), 528–536.
[21]	M. Feurer, B. Letham, F. Hutter, E. Bakshy, Practical transfer learning for Bayesian optimization, preprint, arXiv: 1802.02219.
[22]	S. Falkner, A. Klein, F. Hutter, BOHB: Robust and efficient hyperparameter optimization at scale, in Proceedings of the 35th International Conference on Machine Learning, PMLR, (2018), 1437–1446.
[23]	E. Real, A. Aggarwal, Y. Huang, Q. V. Le, Regularized evolution for image classifier architecture search, in Proceedings of the AAAI Conference on Artificial Intelligence, (2019), 4780–4789. https://doi.org/10.1609/aaai.v33i01.33014780
[24]	D. Hernández-Lobato, J. M. Hernández-Lobato, A. Shah, R. P. Adams, Predictive entropy search for multi-objective Bayesian optimization, in Proceedings of the 33rd International Conference on Machine Learning, PMLR, (2016), 1492–1501.
[25]	B. Paria, K. Kandasamy, B. Póczos, A flexible framework for multi-objective Bayesian optimization using random scalarizations, in Proceedings of the 35th Uncertainty in Artificial Intelligence Conference, PMLR, (2020), 766–776.
[26]	B. Zoph, Q. V. Le, Neural architecture search with reinforcement learning, in Proceedings of the International Conference on Learning Representations, 2017.
[27]	T. Elsken, J. H. Metzen, F. Hutter, Neural architecture search: A survey, J. Mach. Learn. Res., 20 (2019), 1–21.
[28]	M. Baratchi, C. Wang, S. Limmer, J. N. van Rijn, H. Hoos, T. Bäck, et al., Automated machine learning: Past, present and future, Artif. Intell. Rev., 57 (2024), 122. https://doi.org/10.1007/s10462-024-10726-1 doi: 10.1007/s10462-024-10726-1
[29]	L. Franceschi, M. Donini, V. Perrone, A. Klein, C. Archambeau, M. Seeger, et al., Hyperparameter optimization in machine learning, Found. Trends Mach. Learn., 18 (2025), 975–1109. https://doi.org/10.1561/2200000088 doi: 10.1561/2200000088
[30]	P. Gijsbers, M. L. P. Bueno, S. Coors, E. LeDell, S. Poirier, J. Thomas, et al., AMLB: An AutoML benchmark, J. Mach. Learn. Res., 25 (2024), 1–65.
[31]	F. Mohr, M. Wever, Naive automated machine learning, Mach. Learn., 112 (2023), 1131–1170. https://doi.org/10.1007/s10994-022-06200-0 doi: 10.1007/s10994-022-06200-0
[32]	N. Mallik, E. Bergman, C. Hvarfner, D. Stoll, M. Janowski, M. Lindauer, et al., PriorBand: Practical hyperparameter optimization in the age of deep learning, Adv. Neural Inf. Process. Syst., 36 (2023), 7377–7391. https://doi.org/10.52202/075280-0323 doi: 10.52202/075280-0323
[33]	S. Müller, M. Feurer, N. Hollmann, F. Hutter, PFNs4BO: In-context learning for Bayesian optimization, preprint, arXiv: 2305.17535.
[34]	M. Kuhn, K. Johnson, Applied Predictive Modeling, Springer, New York, 2013. https://doi.org/10.1007/978-1-4614-6849-3
[35]	P. Probst, A. L. Boulesteix, B. Bischl, Tunability: Importance of hyperparameters of machine learning algorithms, preprint, arXiv: 1802.09596.
[36]	V. Kumar, M. L. Garg, Predictive analytics: A review of trends and techniques, Int. J. Comput. Appl., 182 (2018), 31–37. https://doi.org/10.5120/ijca2018917434 doi: 10.5120/ijca2018917434
[37]	N. Mishra, S. Silakari, Predictive analytics: A survey, trends, applications, oppurtunities & challenges, Int. J. Comput. Sci. Inf. Technol., 3 (2012), 4434–4438.
[38]	M. Raghu, B. Poole, J. Kleinberg, S. Ganguli, J. Sohl-Dickstein, On the expressive power of deep neural networks, preprint, arXiv: 1606.05336.
[39]	V. Vapnik, The Nature of Statistical Learning Theory, Springer Science & Business Media, Berlin, 2013. https://doi.org/10.1007/978-1-4757-3264-1
[40]	P. L. Bartlett, S. Mendelson, Rademacher and Gaussian complexities: Risk bounds and structural results, J. Mach. Learn. Res., 3 (2002), 463–482.
[41]	H. Buhrman, R. De Wolf, Complexity measures and decision tree complexity: A survey, Theor. Comput. Sci., 288 (2002), 21–43. https://doi.org/10.1016/S0304-3975(01)00144-X doi: 10.1016/S0304-3975(01)00144-X
[42]	N. Bulso, M. Marsili, Y. Roudi, On the complexity of logistic regression models, Neural Comput., 31 (2019), 1592–1623. https://doi.org/10.1162/neco_a_01207 doi: 10.1162/neco_a_01207
[43]	T. Liang, T. Poggio, A. Rakhlin, J. Stokes, Fisher-rao metric, geometry, and complexity of neural networks, preprint, arXiv: 1711.01530.
[44]	X. Hu, L. Chu, J. Pei, W. Liu, J. Bian, Model complexity of deep learning: A survey, Knowl. Inf. Syst., 63 (2021), 2585–2619. https://doi.org/10.1007/s10115-021-01605-0 doi: 10.1007/s10115-021-01605-0
[45]	S. Liu, S. Cong, L. Yang, Survey of research on autonomous driving testing with large models, Commun. Transp. Res., 5 (2025), 100179. https://doi.org/10.1016/j.commtr.2025.100179 doi: 10.1016/j.commtr.2025.100179
[46]	R. Kannan, G. Bayraksan, J. R. Luedtke, Residuals-based distributionally robust optimization with covariate information, preprint, arXiv: 2012.01088.
[47]	D. Bertsimas, B. Van Parys, Bootstrap robust prescriptive analytics, Math. Program., 195 (2022), 39–78. https://doi.org/10.1007/s10107-021-01679-2 doi: 10.1007/s10107-021-01679-2
[48]	G. Y. Ban, C. Rudin, The big data newsvendor: Practical insights from machine learning, Oper. Res., 67 (2019), 90–108. https://doi.org/10.1287/opre.2018.1757 doi: 10.1287/opre.2018.1757
[49]	S. Liu, L. He, Z. Shen, On-time last-mile delivery: Order assignment with travel-time predictors, Manage. Sci., 67 (2021), 4095–4119. https://doi.org/10.1287/mnsc.2020.3741 doi: 10.1287/mnsc.2020.3741
[50]	A. Esteban-Pérez, J. M. Morales, Distributionally robust optimal power flow with contextual information, Eur. J. Oper. Res., 306 (2023), 1047–1058. https://doi.org/10.1016/j.ejor.2022.10.024 doi: 10.1016/j.ejor.2022.10.024
[51]	Z. Y. Chen, M. Sun, X. X. Han, Prediction-driven collaborative emergency medical resource allocation with deep learning and optimization, J. Oper. Res. Soc., 74 (2023), 590–603. https://doi.org/10.1080/01605682.2022.2101953 doi: 10.1080/01605682.2022.2101953
[52]	G. Perakis, M. Sim, Q. Tang, P. Xiong, Robust pricing and production with information partitioning and adaptation, Manage. Sci., 69 (2023), 1398–1419. https://doi.org/10.1287/mnsc.2022.4446 doi: 10.1287/mnsc.2022.4446
[53]	V. Monga, Y. Li, Y. C. Eldar, Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing, IEEE Signal Process Mag., 38 (2021), 18–44. https://doi.org/10.1109/MSP.2020.3016905 doi: 10.1109/MSP.2020.3016905
[54]	M. Blondel, Q. Berthet, M. Cuturi, R. Frostig, S. Hoyer, F. Llinares-López, et al., Efficient and modular implicit differentiation, preprint, arXiv: 2105.15183.
[55]	Q. Berthet, M. Blondel, O. Teboul, M. Cuturi, J. P. Vert, F. Bach, Learning with differentiable pertubed optimizers, preprint, arXiv: 2002.08676.
[56]	C. Wang, D. Jia, W. Wang, D. Ngoduy, B. Peng, J. Wang, A knowledge-informed deep learning paradigm for generalizable and stability-optimized car-following models, Commun. Transp. Res., 5 (2025), 100211. https://doi.org/10.1016/j.commtr.2025.100211 doi: 10.1016/j.commtr.2025.100211
[57]	H. Chu, W. Zhang, P. Bai, Y. Chen, Data-driven optimization for last-mile delivery, Complex Intell. Syst., 9 (2023), 2271–2284. https://doi.org/10.1007/s40747-021-00293-1 doi: 10.1007/s40747-021-00293-1
[58]	X. Tian, R. Yan, Y. Liu, S. Wang, A smart predict-then-optimize method for targeted and cost-effective maritime transportation, Transp. Res. Part B Methodol., 172 (2023), 32–52. https://doi.org/10.1016/j.trb.2023.03.009 doi: 10.1016/j.trb.2023.03.009
[59]	R. Yan, S. Wang, K. Fagerholt, A semi-"smart predict then optimize"(semi-SPO) method for efficient ship inspection, Transp. Res. Part B Methodol., 142 (2020), 100–125. https://doi.org/10.1016/j.trb.2020.09.014 doi: 10.1016/j.trb.2020.09.014
[60]	S. Cong, M. Zheng, W. Zhang, Y. Bie, E. Szczepaàski, Closed-loop coordination of connected and automated vehicles in structured road networks, Commun. Transp. Res., 5 (2025), 100182. https://doi.org/10.1016/j.commtr.2025.100182 doi: 10.1016/j.commtr.2025.100182
[61]	L. Han, X. Ma, Y. Wang, L. He, Y. Li, L. Zhang, et al., Safe and efficient DRL driving policies using fuzzy logic for urban lane changing scenarios, J. Intell. Connected Veh., 8 (2025), 9210054. https://doi.org/10.26599/jicv.2024.9210054 doi: 10.26599/jicv.2024.9210054
[62]	C. Ma, X. Huang, Y. Zhao, T. Wang, B. Du, GRU-LSTM model based on the SSA for short-term traffic flow prediction, J. Intell. Connected Veh., 8 (2025), 9210051. https://doi.org/10.26599/jicv.2024.9210051 doi: 10.26599/jicv.2024.9210051
[63]	Y. Hu, N. Kallus, X. Mao, Fast rates for contextual linear optimization, Manage. Sci., 68 (2022), 4236–4245. https://doi.org/10.1287/mnsc.2022.4383 doi: 10.1287/mnsc.2022.4383
[64]	A. N. Elmachtoub, H. Lam, H. Zhang, Y. Zhao, Estimate-then-optimize versus integrated-estimation-optimization versus sample average approximation: A stochastic dominance perspective, preprint, arXiv: 2304.06833.
[65]	T. Vanderschueren, T. Verdonck, B. Baesens, W. Verbeke, Predict-then-optimize or predict-and-optimize? An empirical evaluation of cost-sensitive learning strategies, Inf. Sci., 594 (2022), 400–415. https://doi.org/10.1016/j.ins.2022.02.021 doi: 10.1016/j.ins.2022.02.021
[66]	J. Mandi, J. Kotary, S. Berden, M. Mulamba, V. Bucarey, T. Guns, et al., Decision-focused learning: Foundations, state of the art, benchmark and future opportunities, J. Artif. Intell. Res., 80 (2024), 1623–1701. https://doi.org/10.1613/jair.1.15320 doi: 10.1613/jair.1.15320
[67]	S. Fujita, G. Ramey, Exogenous versus endogenous separation, Am. Econ. J. Macroecon., 4 (2012), 68–93. https://doi.org/10.1257/mac.4.4.68 doi: 10.1257/mac.4.4.68
[68]	B. Hssina, A. Merbouha, H. Ezzikouri, M. Erritali, A comparative study of decision tree ID3 and C4.5, Int. J. Adv. Comput. Sci. Appl., 4 (2014), 13–19. https://doi.org/10.14569/SpecialIssue.2014.040203 doi: 10.14569/SpecialIssue.2014.040203
[69]	J. R. Cano, Analysis of data complexity measures for classification, Expert Syst. Appl., 40 (2013), 4820–4831. https://doi.org/10.1016/j.eswa.2013.02.025 doi: 10.1016/j.eswa.2013.02.025
[70]	E. G. Talbi, A taxonomy of metaheuristics for bi-level optimization, in Metaheuristics for Bi-level Optimization, Springer, (2013), 1–39. https://doi.org/10.1007/978-3-642-37838-6_1

Reader Comments

Your name:*

Email:*
© 2026 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)