Dynamic hybrid flow shops must simultaneously balance production efficiency, volatile energy prices, carbon emissions, and stochastic shop-floor disruptions. Optimization-based schedulers cannot react to real-time events, whereas purely learning-based dispatchers converge slowly and struggle to satisfy operational constraints. In this study, we proposed a Hybrid Push–Pull Reinforcement Learning (HP2-RL) framework for energy-aware dynamic scheduling in hybrid flow shops with parallel heterogeneous machines, finite buffers, and dynamic lot splitting. An upper-layer MOEA/D-DE optimizer with Tchebycheff decomposition handled seven objectives (makespan, tardiness, energy cost, carbon emissions, WIP dwell time, machine idle time, and material-delay penalty) and generated Pareto-oriented scheduling templates; a lower-layer Actor–Critic agent performed event-driven dispatching and lot splitting guided by the selected template; and a closed-loop feedback mechanism triggered rolling-horizon re-optimization every 4 hours or when KPI deviations exceeded 15%. Simulation results over 30 independent runs showed that HP2-RL reduced makespan by 14.5% relative to rule-based dispatching (1240.5 min) and total energy cost by 11.5% relative to static MOEA/D-DE (, 450), achieved the lowest carbon footprint (1,241 kg), and converged within approximately 300 training episodes. Under machine failures, recovery took 58.6 minutes versus 115 minutes for rule-based dispatching. Ablation experiments and Wilcoxon signed-rank tests (p < 0.05) confirmed that each component contributes significantly, and an inference time below 3.8 ms per decision supports real-time deployment.
Citation: Kuo-Feng Huang. Energy-aware dynamic scheduling in hybrid flow shops via integrated multi-objective evolutionary optimization and Actor–Critic reinforcement learning[J]. Journal of Industrial and Management Optimization, 2026, 22(12): 6339-6373. doi: 10.3934/jimo.2026213
Dynamic hybrid flow shops must simultaneously balance production efficiency, volatile energy prices, carbon emissions, and stochastic shop-floor disruptions. Optimization-based schedulers cannot react to real-time events, whereas purely learning-based dispatchers converge slowly and struggle to satisfy operational constraints. In this study, we proposed a Hybrid Push–Pull Reinforcement Learning (HP2-RL) framework for energy-aware dynamic scheduling in hybrid flow shops with parallel heterogeneous machines, finite buffers, and dynamic lot splitting. An upper-layer MOEA/D-DE optimizer with Tchebycheff decomposition handled seven objectives (makespan, tardiness, energy cost, carbon emissions, WIP dwell time, machine idle time, and material-delay penalty) and generated Pareto-oriented scheduling templates; a lower-layer Actor–Critic agent performed event-driven dispatching and lot splitting guided by the selected template; and a closed-loop feedback mechanism triggered rolling-horizon re-optimization every 4 hours or when KPI deviations exceeded 15%. Simulation results over 30 independent runs showed that HP2-RL reduced makespan by 14.5% relative to rule-based dispatching (1240.5 min) and total energy cost by 11.5% relative to static MOEA/D-DE (, 450), achieved the lowest carbon footprint (1,241 kg), and converged within approximately 300 training episodes. Under machine failures, recovery took 58.6 minutes versus 115 minutes for rule-based dispatching. Ablation experiments and Wilcoxon signed-rank tests (p < 0.05) confirmed that each component contributes significantly, and an inference time below 3.8 ms per decision supports real-time deployment.
| [1] |
F. Tao, J. F. Cheng, Q. L. Qi, M. Zhang, F. Y. Sui, Digital twin-driven product design, manufacturing and service with big data, Int. J. Adv. Manuf. Technol., 94 (2017), 3563–3576. https://doi.org/10.1007/s00170-017-0233-1 doi: 10.1007/s00170-017-0233-1
|
| [2] |
L. R. Lu, Q. W. Deng, Y. Y. Hu, Q. Luo, S. C. Gao, J. X. Zhang, Deep reinforcement learning for dynamic scheduling in distributed heterogeneous flexible job shop with integrated inventory allocation and product delivery, Eng. Appl. Artif. Intell., 162 (2025), 112681. https://doi.org/10.1016/j.engappai.2025.112681 doi: 10.1016/j.engappai.2025.112681
|
| [3] |
Y. J. Qu, X. G. Ming, Z. W. Liu, X. Y. Zhang, Z. T. Hou, Smart manufacturing systems: state of the art and future trends, Int. J. Adv. Manuf. Technol., 103 (2019), 3751–3768. https://doi.org/10.1007/s00170-019-03754-7 doi: 10.1007/s00170-019-03754-7
|
| [4] |
M. Füchtenhans, C. H. Glock, Energy-efficient job shop scheduling considering processing speed and incentive-based programs, IFAC-PapersOnLine, 55 (2022), 192–197. https://doi.org/10.1016/j.ifacol.2022.09.389 doi: 10.1016/j.ifacol.2022.09.389
|
| [5] |
G. Mouzon, M. B. Yildirim, J. Twomey, Operational methods for minimization of energy consumption of manufacturing equipment, Int. J. Prod. Res., 45 (2007), 4247–4271. https://doi.org/10.1080/00207540701450013 doi: 10.1080/00207540701450013
|
| [6] |
R. Li, L. Wang, W. Gong, J. F. Chen, Z. X. Pan, Y. Wu, et al., Evolutionary computation and reinforcement learning integrated algorithm for distributed heterogeneous flowshop scheduling, Eng. Appl. Artif. Intell., 135 (2024), 108775. https://doi.org/10.1016/j.engappai.2024.108775 doi: 10.1016/j.engappai.2024.108775
|
| [7] |
Y. Tang, L. Shen, S. Han, Low-carbon flexible job shop scheduling problem based on deep reinforcement learning, Sustainability, 16 (2024), 4544. https://doi.org/10.3390/su16114544 doi: 10.3390/su16114544
|
| [8] |
M. L. Pinedo, Scheduling: Theory, algorithms, and systems, Springer Cham, 2022. https://doi.org/10.1007/978-3-031-05921-6 |
| [9] |
D. Ouelhadj, S. Petrovic, A survey of dynamic scheduling in manufacturing systems, J. Sched., 12 (2009), 417–431. https://doi.org/10.1007/s10951-008-0090-8 doi: 10.1007/s10951-008-0090-8
|
| [10] |
R. Ruiz, J. A. Vázquez-Rodríguez, The hybrid flow shop scheduling problem, Eur. J. Oper. Res., 205 (2010), 1–18. https://doi.org/10.1016/j.ejor.2009.09.024 doi: 10.1016/j.ejor.2009.09.024
|
| [11] |
P. Godinho, F. G. Branco, Adaptive policies for multi-mode project scheduling under uncertainty, Eur. J. Oper. Res., 216 (2012), 553–562. https://doi.org/10.1016/j.ejor.2011.08.010 doi: 10.1016/j.ejor.2011.08.010
|
| [12] |
R. L. Haupt, S. E. Haupt, Practical genetic algorithms, Wiley, 2003. https://doi.org/10.1002/0471671746 |
| [13] |
M. F. Ashby, Materials and the environment, ScienceDirect, 2013. https://doi.org/10.1016/C2010-0-66554-0 |
| [14] |
Y. Dong, H. Zhang, C. Wang, X. Zhou, Soft actor-critic DRL algorithm for interval optimal dispatch of integrated energy systems with uncertainty in demand response and renewable energy, Eng. Appl. Artif. Intell., 127 (2024), 107230. https://doi.org/10.1016/j.engappai.2023.107230 doi: 10.1016/j.engappai.2023.107230
|
| [15] |
A. Mencaroni, P. Leyman, B. Raa, S. D. Vuyst, D. Claeys, Towards net-zero manufacturing: Carbon-aware scheduling for GHG emissions reduction, J. Clean. Prod., 529 (2025), 146787. https://doi.org/10.1016/j.jclepro.2025.146787 doi: 10.1016/j.jclepro.2025.146787
|
| [16] |
K. Gao, Y. Huang, A. Sadollah, L. Wang, A review of energy-efficient scheduling in intelligent production systems, Complex Intell. Syst., 6 (2020), 237–249. https://doi.org/10.1007/s40747-019-00122-6 doi: 10.1007/s40747-019-00122-6
|
| [17] |
L. Glomb, F. Liers, F. Rösel, A rolling-horizon approach for multi-period optimization, Eur. J. Oper. Res., 300 (2022), 189–206. https://doi.org/10.1016/j.ejor.2021.07.043 doi: 10.1016/j.ejor.2021.07.043
|
| [18] |
K. Deb, A. Pratap, S. Agarwal, T. Meyarivan, A fast and elitist multiobjective genetic algorithm: NSGA-Ⅱ, IEEE Trans. Evol. Comput., 6 (2002), 182–197. https://doi.org/10.1109/4235.996017 doi: 10.1109/4235.996017
|
| [19] |
Q. Zhang, H. Li, MOEA/D: A multiobjective evolutionary algorithm based on decomposition, IEEE Trans. Evol. Comput., 11 (2007), 712–731. https://doi.org/10.1109/TEVC.2007.892759 doi: 10.1109/TEVC.2007.892759
|
| [20] |
C. A. C. Coello, G. B. Lamont, D. A. V. Veldhuizen, Evolutionary algorithms for solving multi-objective problems, Springer, 2007. https://doi.org/10.1007/978-0-387-36797-2 |
| [21] |
K. Tamssaouet, S. Dauzère-Pérès, S. Knopp, A. Bitar, C. Yugma, Multiobjective optimization for complex flexible job-shop scheduling problems, Eur. J. Oper. Res., 296 (2022), 87–100. https://doi.org/10.1016/j.ejor.2021.03.069 doi: 10.1016/j.ejor.2021.03.069
|
| [22] |
G. Fu, T. Yu, W. Li, Q. Deng, B. Yang, A decomposition-based multiobjective optimization evolutionary algorithm with adaptive weight generation strategy, Math. Probl. Eng., 2021 (2021), 2764558. https://doi.org/10.1155/2021/2764558 doi: 10.1155/2021/2764558
|
| [23] |
W. Chen, J. Wang, G. Yu, Y. Hu, Energy-efficient hybrid flow-shop scheduling under time-of-use and ladder electricity tariffs, Appl. Sci., 12 (2022), 6456. https://doi.org/10.3390/app12136456 doi: 10.3390/app12136456
|
| [24] |
D. Schweisgut, A. Benoit, Y. Robert, H. Meyerhenke, Carbon-aware workflow scheduling with fixed mapping and deadline constraint, Proc. Int. Conf. Parallel Process., 2025,627–637. https://doi.org/10.1145/3754598.3754673 doi: 10.1145/3754598.3754673
|
| [25] |
M. S. Cavallieri, E. Viles, J. R. Montoya-Torresb, Sustainable production scheduling: insights from real-life applications, Prod. Plan. Control., 36 (2025), 1595–1622. https://doi.org/10.1080/09537287.2024.2389879 doi: 10.1080/09537287.2024.2389879
|
| [26] |
B. Wang, Q. Li, Rolling horizon procedure for large-scale job-shop scheduling problems, IEEE Int. Conf. Autom. Logist., 2007, 18–21. https://doi.org/10.1109/ICAL.2007.4338679 doi: 10.1109/ICAL.2007.4338679
|
| [27] |
R. Yu, D. Oron, Single-machine scheduling with fixed energy recharging times to minimize the number of late jobs and the number of just-in-time jobs: A parameterized complexity analysis, Eur. J. Oper. Res., 324 (2025), 40–48. https://doi.org/10.1016/j.ejor.2025.01.007 doi: 10.1016/j.ejor.2025.01.007
|
| [28] |
B. Kocot, P. Czarnul, J. Proficz, Energy-aware scheduling for high-performance computing systems: A survey, Energies, 16 (2023), 890. https://doi.org/10.3390/en16020890 doi: 10.3390/en16020890
|
| [29] |
R. S. Sutton, A. G. Barto, Reinforcement learning: An introduction, MIT Press, 2018. |
| [30] |
Z. Wang, M. Gombolay, Learning scheduling policies for multi-robot coordination with graph attention networks, IEEE Robot. Autom. Lett., 5 (2020), 4509–4516. https://doi.org/10.1109/LRA.2020.3002198 doi: 10.1109/LRA.2020.3002198
|
| [31] |
Z. Luo, C. Jiang, L. Liu, X. Zheng, H. Ma, F. Dong, Deep-reinforcement-learning-based production scheduling in industrial internet of things, IEEE Internet Things J., 10 (2023), 19725–19739. https://doi.org/10.1109/JIOT.2023.3283056 doi: 10.1109/JIOT.2023.3283056
|
| [32] |
M. Monaci, V. Agasucci, G. Grani, An actor-critic algorithm with policy gradients to solve the job shop scheduling problem using deep double recurrent agents, Eur. J. Oper. Res., 312 (2024), 910–926. https://doi.org/10.1016/j.ejor.2023.07.037 doi: 10.1016/j.ejor.2023.07.037
|
| [33] |
P. Stöckermann, S. Feudel, A. Immordino, N. Hayen, T. Altenmüller, M. Gebser, et al., Reinforcement learning based dispatching solutions in semiconductor manufacturing: a literature review on validation and deployment, Prod. Manuf. Res., 13 (2025), 2582472. https://doi.org/10.1080/21693277.2025.2582472 doi: 10.1080/21693277.2025.2582472
|
| [34] |
J. Jang, D. Klabjan, H. Liu, N. S. Patel, X. Li, B. Ananthanarayanan, et al., Scalable multi-agent reinforcement learning for factory-wide dynamic scheduling in semiconductor manufacturing, Eng. Appl. Artif. Intell., 161 (2025), 112168. https://doi.org/10.1016/j.engappai.2025.112168 doi: 10.1016/j.engappai.2025.112168
|
| [35] |
F. Eberle, F. Fischer, J. Matuschke, N. Megow, On index policies for stochastic minsum scheduling, Oper. Res. Lett., 47 (2019), 213–218. https://doi.org/10.1016/j.orl.2019.03.007 doi: 10.1016/j.orl.2019.03.007
|
| [36] |
B. Wang, H. Wang, Q. Yan, E. Ma, Multi-agent reinforcement learning-aided evolutionary algorithm for a many-objective distributed hybrid flow shop scheduling problem, Swarm Evol. Comput., 97 (2025), 101991. https://doi.org/10.1016/j.swevo.2025.101991 doi: 10.1016/j.swevo.2025.101991
|
| [37] |
Z. Pan, D. Lei, L. Wang, A knowledge-based two-population optimization algorithm for distributed energy-efficient parallel machines scheduling, IEEE Trans. Cybern., 52 (2022), 5051–5063. https://doi.org/10.1109/TCYB.2020.3026571 doi: 10.1109/TCYB.2020.3026571
|
| [38] |
H. Wang, B. R. Sarker, J. Li, J. Li, Adaptive scheduling for assembly job shop with uncertain assembly times based on dual Q-learning, Int. J. Prod. Res., 59 (2021), 5867–5883. https://doi.org/10.1080/00207543.2020.1794075 doi: 10.1080/00207543.2020.1794075
|
| [39] |
F. Zhao, Z. Fu, L. Wang, H. Sang, A heterogeneous graph reinforcement learning framework with question-aware neighborhood aggregation and interoption prompt attention for dynamic flexible job shop scheduling problem, IEEE Trans. Ind. Inform., 22 (2026), 2863–2874. https://doi.org/10.1109/TII.2025.3646962 doi: 10.1109/TII.2025.3646962
|
| [40] |
S. Chen, X. Wang, Y. Wang, X. Gu, A knowledge-driven many-objective algorithm for energy-efficient distributed heterogeneous hybrid flowshop scheduling with lot-streaming, Swarm Evol. Comput., 91 (2024), 101771. https://doi.org/10.1016/j.swevo.2024.101771 doi: 10.1016/j.swevo.2024.101771
|
Overall research framework of the proposed hybrid push–pull scheduling system
Data and process flow within the proposed framework
Mechanism of dynamic lot splitting and compatible job merging
Upper-layer optimization workflow using MOEA/D–DE
Lower-layer adaptive pull control via reinforcement learning
Feedback learning and performance monitoring mechanism
Comparative performance across scheduling scenarios
Analysis of total energy cost and peak power demand
Carbon emissions and sustainability score assessment
Training dynamics and algorithm convergence
System adaptation efficiency under varying constraints and price multipliers
Recovery time and stability index across disturbance types
Performance degradation rate relative to disturbance intensity