Efficient cooperative target search with multiple unmanned aerial vehicles (UAVs) is fundamentally constrained by partial observability, sensing uncertainty, and limited interagent communication. We formulate this task as a communication-constrained decentralized partially observable Markov decision process with explicit detection and false-alarm models and target-motion prediction. To address these challenges, we propose hierarchical option-based communication-constrained attention (HOCCA), a multiagent reinforcement learning framework that integrates temporally extended options, message-restricted graph attention, and value-of-information-driven communication under centralized training and decentralized execution. Each UAV maintains a provenance-aware occupancy belief using local observations and received likelihood innovations, while unique identifiers prevent duplicate evidence fusion. A receiver-aware scheduler selects informative evidence subject to a hard communication budget, and the hierarchical controller jointly learns option selection, termination, and intraoption maneuvering. In the reference environment, HOCCA requires $ 25.1\% $ and $ 23.8\% $ fewer steps than the strongest hierarchical baseline to achieve $ 80\% $ coverage and detection, respectively, while recording the highest coverage and recall at 260 steps. Evaluations across varying target densities, map scales, and fleet sizes, together with a representative case study, demonstrate efficient coordination across the tested task configurations.
Citation: Jiancheng Liu, Wanlong Qi, Yihua Wang, Bo Li, Kexin Wang, Dingrui Xue. HOCCA: Hierarchical option-based communication-constrained attention for cooperative multi-UAV target search[J]. Journal of Industrial and Management Optimization, 2026, 22(12): 6213-6247. doi: 10.3934/jimo.2026209
Efficient cooperative target search with multiple unmanned aerial vehicles (UAVs) is fundamentally constrained by partial observability, sensing uncertainty, and limited interagent communication. We formulate this task as a communication-constrained decentralized partially observable Markov decision process with explicit detection and false-alarm models and target-motion prediction. To address these challenges, we propose hierarchical option-based communication-constrained attention (HOCCA), a multiagent reinforcement learning framework that integrates temporally extended options, message-restricted graph attention, and value-of-information-driven communication under centralized training and decentralized execution. Each UAV maintains a provenance-aware occupancy belief using local observations and received likelihood innovations, while unique identifiers prevent duplicate evidence fusion. A receiver-aware scheduler selects informative evidence subject to a hard communication budget, and the hierarchical controller jointly learns option selection, termination, and intraoption maneuvering. In the reference environment, HOCCA requires $ 25.1\% $ and $ 23.8\% $ fewer steps than the strongest hierarchical baseline to achieve $ 80\% $ coverage and detection, respectively, while recording the highest coverage and recall at 260 steps. Evaluations across varying target densities, map scales, and fleet sizes, together with a representative case study, demonstrate efficient coordination across the tested task configurations.
| [1] |
J. Wu, J. Luo, C. Jiang, L. Gao, A multiagent deep reinforcement learning approach for multi-UAV cooperative search in multilayered aerial computing networks, IEEE Internet Things J., 12 (2025), 5807–5821. https://doi.org/10.1109/JIOT.2024.3490761 doi: 10.1109/JIOT.2024.3490761
|
| [2] |
J. Liu, S. Wei, B. Li, T. Wang, W. Qi, X. Han, Dual-timescale hierarchical MADDPG for multi-UAV cooperative search, J. King Saud Univ. Comput. Inf. Sci., 37 (2025), 142. https://doi.org/10.1007/s44443-025-00156-6 doi: 10.1007/s44443-025-00156-6
|
| [3] |
B. O. Koopman, The theory of search. Ⅱ. Target detection, Oper. Res., 4 (1956), 503–531. https://doi.org/10.1287/opre.4.5.503 doi: 10.1287/opre.4.5.503
|
| [4] |
L. D. Stone, Theory of optimal search, Academic Press, 1975. https://shop.elsevier.com/books/theory-of-optimal-search/stone/978-0-12-672450-9 |
| [5] |
H. Yuan, Q. Wang, L. Cui, Z. Xu, Q. Li, S. Yan, Hierarchical reinforcement learning with cohesion-aware adaptive planning for large-scale multi-UAV coverage, Inf. Sci., 753 (2026), 123660. https://doi.org/10.1016/j.ins.2026.123660 doi: 10.1016/j.ins.2026.123660
|
| [6] |
R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, I. Mordatch, Multi-agent actor-critic for mixed cooperative-competitive environments, Adv. Neural Inf. Process. Syst., 30 (2017), 6379–6390. Available from: https://proceedings.neurips.cc/paper-files/paper/2017/hash/68a9750337a418a86fe06c1991a1d64c-Abstract.html. |
| [7] |
C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, The surprising effectiveness of PPO in cooperative multi-agent games, Adv. Neural Inf. Process. Syst., 35 (2022), 24611–24624. https://doi.org/10.52202/068431-1787 doi: 10.52202/068431-1787
|
| [8] |
T. Rashid, M. Samvelyan, C. S. D. Witt, G. Farquhar, J. Foerster, S. Whiteson, QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning, Proc. Int. Conf. Mach. Learn., 2018, 4295–4304. Available from: https://proceedings.mlr.press/v80/rashid18a.html. |
| [9] |
X. X. Yang, W. Q. Yao, Y. Zhang, H. Yu, C. Wang, Multi-UAV cooperative search in partially observable low-altitude environments based on deep reinforcement learning, Drones, 9 (2025), 825. https://doi.org/10.3390/drones9120825 doi: 10.3390/drones9120825
|
| [10] |
C. C. Ekechi, T. Elfouly, A. Alouani, T. Khattab, A survey on UAV control with multi-agent reinforcement learning, Drones, 9 (2025), 484. https://doi.org/10.3390/drones9070484 doi: 10.3390/drones9070484
|
| [11] |
X. Chi, Belief-driven decentralized reinforcement learning for cooperative UAV search in partially observable urban environments, Int. Acad. Exch. Conf. Sci. Technol. Innov., 2025. https://doi.org/10.1109/IAECST68792.2025.11415150 doi: 10.1109/IAECST68792.2025.11415150
|
| [12] |
X. Zhang, J. Bai, K. Chen, X. Chen, Hierarchical graph-attention multi-agent reinforcement learning for safe-separation-and-collision-avoidance coordination of heterogeneous UAV swarms, Drones, 10 (2026), 508. https://doi.org/10.3390/drones10070508 doi: 10.3390/drones10070508
|
| [13] |
Z. Yang, W. Du, X. Zhang, J. Wang, Y. Li, A spatiotemporal graph reasoning approach for pursuit-evasion game with communication limits, IEEE Trans. Veh. Technol., 75 (2026), 2069–2085. https://doi.org/10.1109/TVT.2025.3605736 doi: 10.1109/TVT.2025.3605736
|
| [14] |
B. Zhang, T. Wang, M. Li, Y. Cui, X. Lin, Z. Zhu, Multiple unmanned aerial vehicle (multi-UAV) reconnaissance and search with limited communication range using semantic episodic memory in reinforcement learning, Drones, 8 (2024), 393. https://doi.org/10.3390/drones8080393 doi: 10.3390/drones8080393
|
| [15] |
Y. Liu, X. Li, X. Jiang, H. Liu, J. Yang, Multi-UAV trajectory planning for dynamic target search: An LLM-enhanced multi-agent reinforcement learning algorithm, IEEE Trans. Cogn. Commun. Netw., 12 (2026), 10294–10310. https://doi.org/10.1109/TCCN.2026.3710519 doi: 10.1109/TCCN.2026.3710519
|
| [16] |
J. P. Carvalho, A. P. Aguiar, Multi-agent reinforcement learning for zero-shot coverage path planning with dynamic UAV networks, Robot. Auton. Syst., 195 (2026), 105163. https://doi.org/10.1016/j.robot.2025.105163 doi: 10.1016/j.robot.2025.105163
|
| [17] |
R. S. Sutton, D. Precup, S. Singh, Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning, Artif. Intell., 112 (1999), 181–211. https://doi.org/10.1016/S0004-3702(99)00052-1 doi: 10.1016/S0004-3702(99)00052-1
|
| [18] |
P. L. Bacon, J. Harb, D. Precup, The option-critic architecture, Proc. AAAI Conf. Artif. Intell., 2017, 1726–1734. https://doi.org/10.1609/aaai.v31i1.10916 doi: 10.1609/aaai.v31i1.10916
|
| [19] |
P. M. Dames, Distributed multi-target search and tracking using the PHD filter, Auton. Robots, 44 (2020), 673–689. https://doi.org/10.1007/s10514-019-09840-9 doi: 10.1007/s10514-019-09840-9
|
| [20] |
F. A. Oliehoek, C. Amato, A Concise introduction to decentralized POMDPs, SpringerBriefs in Intelligent Systems, Springer, Cham, Switzerland, 2016. https://doi.org/10.1007/978-3-319-28929-8 |
| [21] |
A. U. Rehman, X. Shi, F. Ullah, Z. Wang, C. Ma, Measuring student attention based on EEG brain signals using deep reinforcement learning, Expert Syst. Appl., 269 (2025), 126426. https://doi.org/10.1016/j.eswa.2025.126426 doi: 10.1016/j.eswa.2025.126426
|
| [22] |
A. U. Rehman, N. U. Eman, F. Ullah, D. Cacciagrano, L. Mostarda, N. Raza, Advanced EEG signal processing and deep Q-learning for accurate student attention monitoring, IEEE Access, 13 (2025), 47260–47270. https://doi.org/10.1109/ACCESS.2024.3523225 doi: 10.1109/ACCESS.2024.3523225
|
| [23] |
A. U. Rehman, K. Mahmood, E. AbdulKreem, Proactive threat hunting and anomaly detection in wearable EEG consumer devices using double deep reinforcement learning, IEEE Trans. Consum. Electron., 2026. https://doi.org/10.1109/TCE.2026.3695900 doi: 10.1109/TCE.2026.3695900
|
| [24] |
T. Ali, U. Draz, I. Razzak, M. Hijji, S. Yasin, M. Ayaz, Vision-driven collaborative path planning for autonomous multi-UAV swarms in maritime and coastal environments using deep reinforcement learning, Robot. Auton. Syst., 204 (2026), 105567. https://doi.org/10.1016/j.robot.2026.105567 doi: 10.1016/j.robot.2026.105567
|
| [25] |
U. Draz, S. Yasin, T. Ali, M. Hijji, M. Ayaz, E. H. M. Aggoune, Communication-aware multi-agent reinforcement learning for cooperative navigation of multiple USVs in dynamic ocean environments, Ocean Eng., 345 (2026), 123626. https://doi.org/10.1016/j.oceaneng.2025.123626 doi: 10.1016/j.oceaneng.2025.123626
|
| [26] |
U. Draz, S. Yasin, T. Ali, M. Hijji, M. Ayaz, E. H. M. Aggoune, Energy-efficient and interference-resilient optimization of continuous 3D UAV relays in IoT networks using deep deterministic policy gradient, IEEE Internet Things J., 2026. https://doi.org/10.1109/JIOT.2026.3720275 doi: 10.1109/JIOT.2026.3720275
|
| [27] |
H. Zhu, J. Hui, Z. Guo, Two-level-attention-based continuous trajectory design and computation offloading for multi-UAV cooperative target search, IEEE Trans. Mob. Comput., 25 (2026), 3196–3214. https://doi.org/10.1109/TMC.2025.3614596 doi: 10.1109/TMC.2025.3614596
|
| [28] |
T. D. Kulkarni, K. Narasimhan, A. Saeedi, J. Tenenbaum, Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation, Adv. neural inf. process. syst., 2016. Available from: https://proceedings.neurips.cc/paper-files/paper/2016/hash/f442d33fa06832082290ad8544a8da27-Abstract.html. |
| [29] |
A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, FeUdal networks for hierarchical reinforcement learning, Proc. Int. Conf. Mach. Learn., 2017, 3540–3549. https://proceedings.mlr.press/v70/vezhnevets17a.html |
| [30] |
C. Amato, G. Konidaris, A. Anders, G. Cruz, J. P. How, L. P. Kaelbling, Policy search for multi-robot coordination under uncertainty, Int. J. Robot. Res., 35 (2016), 1760–1778. https://doi.org/10.1177/0278364916679611. doi: 10.1177/0278364916679611
|
| [31] |
S. Sukhbaatar, A. Szlam, R. Fergus, Learning multiagent communication with backpropagation, Adv. Neural Inf. Process. Syst., 29 (2016), 2244–2252. Available from: https://proceedings.neurips.cc/paper/2016/hash/55b1927fdafef39c48e5b73b5d61ea60-Abstract.html. |
| [32] |
A. Das, T. Gervet, J. Romoff, D. Batra, D. Parikh, M. Rabbat, TarMAC: Targeted multi-agent communication, Proc. 36th Int. Conf. Mach. Learn., 97 (2019), 1538–1546. https://proceedings.mlr.press/v97/das19a.html |
| [33] |
D. Kim, S. Moon, D. Hostallero, W. J. Kang, T. Lee, K. Son, Learning to schedule communication in multi-agent reinforcement learning, Proc. Int. Conf. Learn. Represent., 2019. Available from: https://openreview.net/forum?id = HUAnBToP-a. |
| [34] |
Q. Sun, D. Steckelmacher, Y. Yao, A. Nowé, R. Avalos, Dynamic size message scheduling for multi-agent communication under limited bandwidth, IEEE Trans. Mob. Comput., 23 (2024), 15080–15097. https://doi.org/10.1109/TMC.2024.3452986 doi: 10.1109/TMC.2024.3452986
|
| [35] |
R. Zhao, D. Zhou, S. K. Jha, N. D. Bastian, A. Shah, HiComm: Hierarchical communication for multi-agent reinforcement learning, arXiv: 2606.29126, 2026. https://doi.org/10.48550/arXiv.2606.29126 |
| [36] |
J. You, Y. Wu, D. Wen, Y. Zhou, Y. Jiang, Y. Shi, Dynamic communication in multi-agent reinforcement learning via information bottleneck, Proc. IEEE Glob. Commun. Conf., 2024, 4884–4889. https://doi.org/10.1109/GLOBECOM52923.2024.10901017 doi: 10.1109/GLOBECOM52923.2024.10901017
|
| [37] |
V. Mnih, K. Kavukcuoglu, D. Silver, Human-level control through deep reinforcement learning, Nature, 518 (2015), 529–533. https://doi.org/10.1038/nature14236 doi: 10.1038/nature14236
|
| [38] |
H. V. Hasselt, A. Guez, D. Silver, Deep reinforcement learning with double Q-learning, Proc. AAAI Conf. Artif. Intell., 30 (2016), 2094–2100. https://doi.org/10.1609/aaai.v30i1.10295 doi: 10.1609/aaai.v30i1.10295
|
Figures(8) / Tables(14)
Jiancheng Liu, Wanlong Qi, Yihua Wang, Bo Li, Kexin Wang, Dingrui Xue. HOCCA: Hierarchical option-based communication-constrained attention for cooperative multi-UAV target search[J]. Journal of Industrial and Management Optimization, 2026, 22(12): 6213-6247. doi: 10.3934/jimo.2026209
Architecture and information flow of HOCCA. Blue (A): Local sensing, motion prediction, and uncertainty-aware occupancy belief updating. Orange (B): Receiver-aware evidence prioritization, budget-constrained transmission, duplicate-safe fusion, and message-restricted graph attention. Green (C): Decentralized option selection, learned termination, and intraoption execution over the primitive action set $ \{-30^\circ, 0^\circ, +30^\circ\} $. Centralized state information is used only by the training critics and is unavailable during decentralized execution
Measured training dynamics over 500 episodes. Solid curves show the smoothed mean episode reward across three independent seeds, and shaded regions indicate $ \pm $ one standard deviation. Color coding: HOCCA (brown), DTH-MADDPG (purple), MAPPO (red), MADDPG (green), and QMIX (orange). Higher values indicate better performance
Measured held-out search performance in the $ 48\times48 $ reference environment. Panel (a) reports footprint coverage, and panel (b) reports confirmed-target recall. Solid curves show the mean over 90 held-out evaluations, shaded regions indicate $ \pm $ one standard deviation, and the dashed horizontal line denotes the 0.8 performance threshold. Colors identify IQL (gray), QMIX (light blue), MADDPG (orange), MAPPO (green), DTH-MADDPG (purple), and HOCCA (dark blue). Higher values and earlier threshold crossings indicate better performance
Measured generalization performance under increasing task complexity. Panel (a) shows mission success rate as the target count increases from 10 to 40. Panel (b) shows the number of steps required to reach 80% footprint coverage as the grid side length increases from 32 to 80. Colors identify IQL (gray), QMIX (light blue), MADDPG (orange), MAPPO (green), DTH-MADDPG (purple), and HOCCA (dark blue). In Panel (a), higher values are better. In Panel (b), lower values are better
Measured fleet-size scaling on the $ 48\times48 $ reference environment with a 400-step horizon. The two panels report the numbers of steps required to reach 80% footprint coverage and 80% confirmed-target recall as the fleet size increases from two to eight UAVs. Colors identify IQL (gray), MADDPG (orange), MAPPO (green), DTH-MADDPG (purple), and HOCCA (dark blue). Lower values indicate faster cooperative searches
Single-component ablations of HOCCA. The bars report the environment steps required to reach $ 80\% $ coverage and confirmed target recall. Lower values indicate faster search
One-factor-at-a-time sensitivity analysis of HOCCA with respect to the maximum option age $ K_L $, option count $ |\Omega| $, number of GAT heads $ K_h $, and communication budget $ B $. All remaining hyperparameters are fixed at $ (K_L, |\Omega|, K_h, B) = (8, 4, 2, 2) $. Markers denote the environment steps required to reach $ 80\% $ footprint coverage, with lower values indicating faster exploration
Spatiotemporal evolution of cooperative search for fleets of $N = 4$, $6$, and $8$ UAVs on the same $48\times48$ target realization. Rows correspond to the three fleet sizes, while columns show the cumulative trajectories at $t = 80$, $180$, and $260$. The colored trajectory lines denote UAV-1, UAV-2, UAV-3, UAV-4, UAV-5, UAV-6, UAV-7, and UAV-8
DownLoad: