Human activity recognition (HAR) refers to the process of identifying human actions and activities using data collected from sensors. Neural networks, such as convolutional neural networks (CNNs), long short-term memory (LSTM) networks, convolutional LSTM, and their hybrid combinations, have demonstrated exceptional performance in various research domains. Developing a multilevel individual or hybrid model for HAR involves strategically integrating multiple networks to capitalize on their complementary strengths. The structural arrangement of these components is a critical factor influencing the overall performance. This study explored a novel framework of a two-level network architecture with dual-stage feature fusion: late fusion, which combines the outputs from the first network level, and intermediate fusion, which integrates the features from both the first and second levels. We evaluated $ 15 $ different network architectures of CNNs, LSTMs, and convolutional LSTMs, incorporating late fusion with and without intermediate fusion, to identify the optimal configuration. Experimental evaluation on two public benchmark datasets demonstrated that architectures incorporating both late and intermediate fusion achieve higher accuracy than those relying on late fusion alone. Moreover, the optimal configuration outperformed baseline models, thereby validating its effectiveness for HAR.
Citation: Abeer FathAllah Brery, Ascensión Gallardo-Antolín, Israel Gonzalez-Carrasco, Mahmoud Fakhry. Multilevel neural networks with dual-stage feature fusion for human activity recognition[J]. Applied Computing and Intelligence, 2025, 5(2): 286-300. doi: 10.3934/aci.2025016
Human activity recognition (HAR) refers to the process of identifying human actions and activities using data collected from sensors. Neural networks, such as convolutional neural networks (CNNs), long short-term memory (LSTM) networks, convolutional LSTM, and their hybrid combinations, have demonstrated exceptional performance in various research domains. Developing a multilevel individual or hybrid model for HAR involves strategically integrating multiple networks to capitalize on their complementary strengths. The structural arrangement of these components is a critical factor influencing the overall performance. This study explored a novel framework of a two-level network architecture with dual-stage feature fusion: late fusion, which combines the outputs from the first network level, and intermediate fusion, which integrates the features from both the first and second levels. We evaluated $ 15 $ different network architectures of CNNs, LSTMs, and convolutional LSTMs, incorporating late fusion with and without intermediate fusion, to identify the optimal configuration. Experimental evaluation on two public benchmark datasets demonstrated that architectures incorporating both late and intermediate fusion achieve higher accuracy than those relying on late fusion alone. Moreover, the optimal configuration outperformed baseline models, thereby validating its effectiveness for HAR.
| [1] | S. G. Dhekane, T. Ploetz, Transfer learning in human activity recognition: a survey, arXiv: 2401.10185. https://doi.org/10.48550/arXiv.2401.10185 |
| [2] | V. Soni, S. Jaiswal, V. B. Semwal, B. Roy, D. K. Choubey, D. K. Mallick, An enhanced deep learning approach for smartphone-based human activity recognition in ioht, In: Machine learning, image processing, network security and data sciences: select proceedings of 3rd international conference on MIND 2021, Singapore: Springer, 2023,505–516. https://doi.org/10.1007/978-981-19-5868-7_37 |
| [3] | S. Saini, A. Juneja, A. Shrivastava, Human activity recognition using deep learning: past, present and future, Proceedings of 1st International Conference on Intelligent Computing and Research Trends (ICRT), 2023, 1–6. https://doi.org/10.1109/ICRT57042.2023.10146621 |
| [4] |
S. Mekruksavanich, A. Jitpattanakul, The deep learning-based human activity recognition using smart wearable sensors: a tutorial, ReBICTE, 8 (2022), 1. https://doi.org/10.22667/ReBiCTE.2022.02.28.001 doi: 10.22667/ReBiCTE.2022.02.28.001
|
| [5] |
E. Ramanujam, T. Perumal, S. Padmavathi, Human activity recognition with smartphone and wearable sensors using deep learning techniques: a review, IEEE Sens. J., 21 (2021), 13029–13040. https://doi.org/10.1109/JSEN.2021.3069927 doi: 10.1109/JSEN.2021.3069927
|
| [6] | K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015,770–778. https://doi.org/10.1109/CVPR.2016.90 |
| [7] |
C. Han, L. Zhang, Y. Tang, W. Huang, F. Min, J. He, Human activity recognition using wearable sensors by heterogeneous convolutional neural networks, Expert Syst. Appl., 198 (2022), 116764. https://doi.org/10.1016/j.eswa.2022.116764 doi: 10.1016/j.eswa.2022.116764
|
| [8] |
Y. Li, J. Wu, W. Li, A. Fang, W. Dong, Temporal-spatial dynamic convolutional neural network for human activity recognition using wearable sensors, IEEE Trans. Instrum. Meas., 72 (2023), 2516912. https://doi.org/10.1109/TIM.2023.3279908 doi: 10.1109/TIM.2023.3279908
|
| [9] |
J. Sena, J. Barreto, C. Caetano, G. Cramer, W. R. Schwartz, Human activity recognition based on smartphone and wearable sensors using multiscale dcnn ensemble, Neurocomputing, 444 (2021), 226–243. https://doi.org/10.1016/j.neucom.2020.04.151 doi: 10.1016/j.neucom.2020.04.151
|
| [10] |
Q. Huang, W. Xie, C. Li, Y. Wang, Y. Liu, Human action recognition based on hierarchical multi-scale adaptive conv-long short-term memory network, Appl. Sci., 13 (2023), 10560. https://doi.org/10.3390/app131910560 doi: 10.3390/app131910560
|
| [11] | M. Sethi, M. Yadav, M. Singh, P. G. Shambharkar, Attnhar: human activity recognition using data collected from wearable sensors, Proceedings of 6th International Conference on Information Systems and Computer Networks (ISCON), 2023, 1–6. https://doi.org/10.1109/ISCON57294.2023.10112183 |
| [12] |
S. P. Singh, M. K. Sharma, A. Lay-Ekuakille, D. Gangwar, S. Gupta, Deep convlstm with self-attention for human activity decoding using wearable sensors, IEEE Sens. J., 21 (2021), 8575–8582. https://doi.org/10.1109/JSEN.2020.3045135 doi: 10.1109/JSEN.2020.3045135
|
| [13] |
L. Wang, R. Liu, Human activity recognition based on wearable sensor using hierarchical deep lstm networks, Circuits Syst. Signal Process., 39 (2020), 837–856. https://doi.org/10.1007/s00034-019-01116-y doi: 10.1007/s00034-019-01116-y
|
| [14] | W. Ahmad, M. Kazmi, H. Ali, Human activity recognition using multi-head cnn followed by lstm, Proceedings of 15th International Conference on Emerging Technologies (ICET), 2019, 1–6. https://doi.org/10.1109/ICET48972.2019.8994412 |
| [15] | R. Kolkar, V. Geetha, Human activity recognition in smart home using deep learning techniques, Proceedings of 13th International conference on information & communication technology and system (ICTS), 2021,230–234. https://doi.org/10.1109/ICTS52701.2021.9609044 |
| [16] | J. X. Goh, K. M. Lim, C. P. Lee, 1d convolutional neural network with long short-term memory for human activity recognition, Proceedings of IEEE International Conference on Artificial Intelligence in Engineering and Technology (IICAIET), 2021, 1–6. https://doi.org/10.1109/IICAIET51634.2021.9573979 |
| [17] |
M. M. Islam, S. Nooruddin, F. Karray, G. Muhammad, Multi-level feature fusion for multimodal human activity recognition in internet of healthcare things, Inform. Fusion, 94 (2023), 17–31. https://doi.org/10.1016/j.inffus.2023.01.015 doi: 10.1016/j.inffus.2023.01.015
|
| [18] | X. Shi, Z. Chen, H. Wang, D. Y. Yeung, W. K. Wong, W. C. Woo, Convolutional lstm network: a machine learning approach for precipitation nowcasting, Proceedings of 29th Annual Conference on Neural Information Processing Systems, 2015,802–810. |
| [19] |
G. Alam, I. McChesney, P. Nicholl, J. Rafferty, Open data sets in human activity recognition research-issues and challenges: a review, IEEE Sens. J., 23 (2023), 26952–26980. https://doi.org/10.1109/JSEN.2023.3317645 doi: 10.1109/JSEN.2023.3317645
|
| [20] | M. Zhang, A. A. Sawchuk, Usc-had: a daily activity dataset for ubiquitous activity recognition using wearable sensors, Proceedings of the 2012 ACM Conference on Ubiquitous Computing, 2012, 1036–1043. https://doi.org/10.1145/2370216.2370438 |
| [21] | D. Anguita, A. Ghio, L. Oneto, X. Parra, J. L. Reyes-Ortiz, A public domain dataset for human activity recognition using smartphones, Proceedings of European Symposium on Artificial Neural Networks, Computational Intelligenceand Machine Learning, 2013, 437–442. |
| [22] |
M. M. Islam, S. Nooruddin, F. Karray, G. Muhammad, Human activity recognition using tools of convolutional neural networks: a state of the art review, data sets, challenges, and future prospects, Comput. Biol. Med., 149 (2022), 106060. https://doi.org/10.1016/j.compbiomed.2022.106060 doi: 10.1016/j.compbiomed.2022.106060
|
| [23] | A. Ghosh, A. Sufian, F. Sultana, A. Chakrabarti, D. De, Fundamental concepts of convolutional neural network, In: Recent trends and advances in artificial intelligence and internet of things, Cham: Springer, 2019,519–567. https://doi.org/10.1007/978-3-030-32644-9_36 |
| [24] |
S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural Comput., 9 (1997), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735 doi: 10.1162/neco.1997.9.8.1735
|
| [25] |
J. Opitz, A closer look at classification evaluation metrics and a critical reflection of common evaluation practice, Transactions of the Association for Computational Linguistics, 12 (2024), 820–836. https://doi.org/10.1162/tacl_a_00675 doi: 10.1162/tacl_a_00675
|
Figures(6) / Tables(3)
Abeer FathAllah Brery, Ascensión Gallardo-Antolín, Israel Gonzalez-Carrasco, Mahmoud Fakhry. Multilevel neural networks with dual-stage feature fusion for human activity recognition[J]. Applied Computing and Intelligence, 2025, 5(2): 286-300. doi: 10.3934/aci.2025016
DownLoad: