Automated container terminals rely on automated guided vehicle (AGV) fleets to connect quay cranes and yard blocks. Existing AGV dispatching studies mainly address task assignment, routing, charging, or equipment coordination, while maintenance decisions are often modeled as exogenous availability changes. This separation limits policy responses to health degradation, failure risk, maintenance windows, and maintenance resources. This study investigates a joint AGV dispatching and predictive maintenance problem that integrates transport assignment, charging, preventive and corrective maintenance, health degradation, stochastic failures, and shared resource capacities in a unified framework. A health-aware extension of QMIX (monotonic mixing of individual agents' action values), named HA-QMIX, is proposed to learn decentralized AGV dispatching policies under centralized training. HA-QMIX incorporates the health index, remaining useful life, failure risk, and state of charge into agent observations and global states. A top-K task candidate mechanism handles dynamic task arrivals within a fixed action space, and action masking excludes infeasible actions caused by task status, vehicle status, charging conditions, maintenance windows, and resource capacities. A reward transformation links net-cost minimization with cooperative learning, while constrained checkpoint selection balances operational performance and safety. In the reported preventive maintenance pressure experiments, HA-QMIX exhibits relatively consistent risk-related indicators across five training seed runs, while throughput varies substantially across these runs. The results therefore identify throughput stability as a remaining limitation of the current approach.
Citation: Ang Yang, Qing Xu, Xingchun Li, Keshuang Du, Jing Gao. Joint AGV dispatching and predictive maintenance in automated container terminals: A health-aware multi-agent reinforcement learning approach[J]. Journal of Industrial and Management Optimization, 2026, 22(11): 5855-5908. doi: 10.3934/jimo.2026198
Automated container terminals rely on automated guided vehicle (AGV) fleets to connect quay cranes and yard blocks. Existing AGV dispatching studies mainly address task assignment, routing, charging, or equipment coordination, while maintenance decisions are often modeled as exogenous availability changes. This separation limits policy responses to health degradation, failure risk, maintenance windows, and maintenance resources. This study investigates a joint AGV dispatching and predictive maintenance problem that integrates transport assignment, charging, preventive and corrective maintenance, health degradation, stochastic failures, and shared resource capacities in a unified framework. A health-aware extension of QMIX (monotonic mixing of individual agents' action values), named HA-QMIX, is proposed to learn decentralized AGV dispatching policies under centralized training. HA-QMIX incorporates the health index, remaining useful life, failure risk, and state of charge into agent observations and global states. A top-K task candidate mechanism handles dynamic task arrivals within a fixed action space, and action masking excludes infeasible actions caused by task status, vehicle status, charging conditions, maintenance windows, and resource capacities. A reward transformation links net-cost minimization with cooperative learning, while constrained checkpoint selection balances operational performance and safety. In the reported preventive maintenance pressure experiments, HA-QMIX exhibits relatively consistent risk-related indicators across five training seed runs, while throughput varies substantially across these runs. The results therefore identify throughput stability as a remaining limitation of the current approach.
| [1] |
X. Chen, S. He, Y. Zhang, L. Tong, P. Shang, X. Zhou, Yard crane and agv scheduling in automated container terminal: A multi-robot task allocation framework, Transp. Res. Part C Emerg. Technol., 114 (2020), 241–271. https://doi.org/10.1016/j.trc.2020.02.012 doi: 10.1016/j.trc.2020.02.012
|
| [2] |
S. Li, L. Fan, S. Jia, A hierarchical solution framework for dynamic and conflict-free agv scheduling in an automated container terminal, Transp. Res. Part C Emerg. Technol., 165 (2024), 104724. https://doi.org/10.1016/j.trc.2024.104724 doi: 10.1016/j.trc.2024.104724
|
| [3] |
J. Xin, R. R. Negenborn, F. Corman, G. Lodewijks, Control of interacting machines in automated container terminals using a sequential planning approach for collision avoidance, Transp. Res. Part C Emerg. Technol., 60 (2015), 377–396. https://doi.org/10.1016/j.trc.2015.09.002 doi: 10.1016/j.trc.2015.09.002
|
| [4] |
X. Zhang, H. Li, J. B. Sheu, Integrated scheduling optimization of agv and double yard cranes in automated container terminals, Transp. Res. Part B Methodol., 179 (2024), 102871. https://doi.org/10.1016/j.trb.2023.102871 doi: 10.1016/j.trb.2023.102871
|
| [5] |
X. Yang, H. Hu, Y. Wang, C. Cheng, Agv scheduling in automated container terminals considering multi-load strategy and charging requirements, Int. J. Prod. Res., 63 (2025), 9269–9297. https://doi.org/10.1080/00207543.2025.2536728 doi: 10.1080/00207543.2025.2536728
|
| [6] |
C. Hu, X. Qian, S. Lu, X. Liu, P. M. Pardalos, Coordinated optimization of production scheduling and maintenance activities with machine reliability deterioration, J. Ind. Manag. Optim., 18 (2022), 3953–3973. https://doi.org/10.3934/jimo.2021142 doi: 10.3934/jimo.2021142
|
| [7] |
J. Zhou, J. Yang, S. Xiang, Y. Qin, Remaining useful life prediction methodologies with health indicator dependence for rotating machinery: A comprehensive review, IEEE Trans. Instrum. Meas., 74 (2025), 1–19. https://doi.org/10.1109/TIM.2025.3556919 doi: 10.1109/TIM.2025.3556919
|
| [8] |
M. Ghaleb, S. Taghipour, M. Sharifi, H. Zolfagharinia, Integrated production and maintenance scheduling for a single degrading machine with deterioration-based failures, Comput. Ind. Eng., 143 (2020), 106432. https://doi.org/10.1016/j.cie.2020.106432 doi: 10.1016/j.cie.2020.106432
|
| [9] |
M. Geurtsen, J. B. H. C. Didden, J. Adan, Z. Atan, I. J. B. F. Adan, Production, maintenance and resource scheduling: A review, Eur. J. Oper. Res., 305 (2023), 501–529. https://doi.org/10.1016/j.ejor.2022.03.045 doi: 10.1016/j.ejor.2022.03.045
|
| [10] |
J. C. Chen, T. L. Chen, B. Y. Kew, P. C. Chang, Simultaneous scheduling optimization for machines and multiple-load agvs in a digital-twin-based environment, J. Ind. Manag. Optim., 21 (2025), 5914–5937. https://doi.org/10.3934/jimo.2025117 doi: 10.3934/jimo.2025117
|
| [11] |
X. S. Si, W. Wang, C. H. Hu, D. H. Zhou, Remaining useful life estimation – a review on the statistical data driven approaches, Eur. J. Oper. Res., 213 (2011), 1–14. https://doi.org/10.1016/j.ejor.2010.11.018 doi: 10.1016/j.ejor.2010.11.018
|
| [12] |
S. Alaswad, Y. Xiang, A review on condition-based maintenance optimization models for stochastically deteriorating system, Reliab. Eng. Syst. Saf., 157 (2017), 54–63. https://doi.org/10.1016/j.ress.2016.08.009 doi: 10.1016/j.ress.2016.08.009
|
| [13] |
Y. Shi, W. Zhu, Y. Xiang, Q. Feng, Condition-based maintenance optimization for multi-component systems subject to a system reliability requirement, Reliab. Eng. Syst. Saf., 202 (2020), 107042. https://doi.org/10.1016/j.ress.2020.107042 doi: 10.1016/j.ress.2020.107042
|
| [14] |
Z. Lu, W. Cui, X. Han, Integrated production and preventive maintenance scheduling for a single machine with failure uncertainty, Comput. Ind. Eng., 80 (2015), 236–244. https://doi.org/10.1016/j.cie.2014.12.017 doi: 10.1016/j.cie.2014.12.017
|
| [15] |
Q. Liu, M. Dong, F. F. Chen, Single-machine-based joint optimization of predictive maintenance planning and production scheduling, Robot. Comput. Integr. Manuf., 51 (2018), 238–247. https://doi.org/10.1016/j.rcim.2018.09.007 doi: 10.1016/j.rcim.2018.09.007
|
| [16] |
T. Qin, Y. Du, J. H. Chen, M. Sha, Combining mixed integer programming and constraint programming to solve the integrated scheduling problem of container handling operations of a single vessel, Eur. J. Oper. Res., 285 (2020), 884–901. https://doi.org/10.1016/j.ejor.2020.02.021 doi: 10.1016/j.ejor.2020.02.021
|
| [17] |
N. Zhang, W. Si, Deep reinforcement learning for condition-based maintenance planning of multi-component systems under dependent competing risks, Reliab. Eng. Syst. Saf., 203 (2020), 107094. https://doi.org/10.1016/j.ress.2020.107094 doi: 10.1016/j.ress.2020.107094
|
| [18] |
M. L. R. Rodríguez, S. Kubler, A. de Giorgio, M. Cordy, J. Robert, Y. L. Traon, Multi-agent deep reinforcement learning based predictive maintenance on parallel machines, Robot. Comput. Integr. Manuf., 78 (2022), 102406. https://doi.org/10.1016/j.rcim.2022.102406 doi: 10.1016/j.rcim.2022.102406
|
| [19] |
H. Li, J. Peng, X. Wang, J. Wan, Integrated resource assignment and scheduling optimization with limited critical equipment constraints at an automated container terminal, IEEE Trans. Intell. Transp. Syst., 22 (2021), 7607–7618. https://doi.org/10.1109/TITS.2020.3005854 doi: 10.1109/TITS.2020.3005854
|
| [20] |
C. Yang, Y. Zhang, J. Wang, L. He, H. Wu, A deep reinforcement learning based multi-agent simulation optimization approach for igv bidirectional task allocation and charging joint scheduling in automated container terminals, Comput. Oper. Res., 183 (2025), 107189. https://doi.org/10.1016/j.cor.2025.107189 doi: 10.1016/j.cor.2025.107189
|
| [21] |
J. C. Chen, T. L. Chen, Y. C. Teng, Meta-model based simulation optimization for automated guided vehicle system under different charging mechanisms, Simul. Model. Pract. Theory, 106 (2021), 102208. https://doi.org/10.1016/j.simpat.2020.102208 doi: 10.1016/j.simpat.2020.102208
|
| [22] | T. Rashid, M. Samvelyan, C. S. de Witt, G. Farquhar, J. Foerster, S. Whiteson, Monotonic value function factorisation for deep multi-agent reinforcement learning, J. Mach. Learn. Res., 21 (2020), 1–51. |
Figures(11) / Tables(10)
Ang Yang, Qing Xu, Xingchun Li, Keshuang Du, Jing Gao. Joint AGV dispatching and predictive maintenance in automated container terminals: A health-aware multi-agent reinforcement learning approach[J]. Journal of Industrial and Management Optimization, 2026, 22(11): 5855-5908. doi: 10.3934/jimo.2026198
DownLoad: