Spatiotemporal and kinematic characteristics augmentation using Dual-GAN for ankle instability detection

Xin Liu; Chen Zhao; Bin Zheng; Qinwei Guo; Yuanyuan Yu; Dezheng Zhang; Aziguli Wulamu; Xin Liu; Chen Zhao; Bin Zheng; Qinwei Guo; Yuanyuan Yu; Dezheng Zhang; Aziguli Wulamu

doi:10.3934/mbe.2022469

Mathematical Biosciences and Engineering

2022, Volume 19, Issue 10: 10037-10059. doi: 10.3934/mbe.2022469

Previous Article Next Article

Research article Special Issues

Spatiotemporal and kinematic characteristics augmentation using Dual-GAN for ankle instability detection

1.
School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing, China
2.
Surgical Simulation Research Laboratory, Department of Surgery, University of Alberta, Edmonton, Alberta, Canada
3.
Beijing Key Laboratory of Knowledge Engineering for Materials Science, Beijing, China
4.
Institute of Sports Medicine, Peking University Third Hospital, Beijing, China

Academic Editor: Zhichang Zhang

Received: 09 May 2022 Revised: 29 June 2022 Accepted: 07 July 2022 Published: 14 July 2022

Obtaining massive amounts of training data is often crucial for computer-assisted diagnosis using deep learning. Unfortunately, patient data is often small due to varied constraints. We develop a new approach to extract significant features from a small clinical gait analysis dataset to improve computer-assisted diagnosis of Chronic Ankle Instability (CAI) patients. In this paper, we present an approach for augmenting spatiotemporal and kinematic characteristics using the Dual Generative Adversarial Networks (Dual-GAN) to train a series of modified Long Short-Term Memory (LSTM) detection models making the training process more data-efficient. Namely, we use LSTM-, LSTM-Fully Convolutional Networks (FCN)-, and Convolutional LSTM-based detection models to identify the patients with CAI. The Dual-GAN enables the synthesized data to approximate the real data distribution visualized by the t-distributed Stochastic Neighbor Embedding (t-SNE) algorithm. Then we trained the proposed detection models using real data collected from a controlled laboratory study and mixed data from real and synthesized gait features. The detection models were tested in real data to validate the positive role in data augmentation as well as to demonstrate the capability and effectiveness of the modified LSTM algorithm for CAI detection using spatiotemporal and kinematic characteristics in walking. Dual-GAN generated efficient spatiotemporal and kinematic characteristics to augment the training set promoting the performance of CAI detection and the modified LSTM algorithm yielded an enhanced classification outcome to identify those CAI patients from a group of control subjects based on gait analysis data than any previous reports.
- ankle instability,
- computer-assisted detection,
- gait features,
- Dual Generative Adversarial Networks (Dual-GAN),
- Long Short-Term Memory (LSTM)
Citation: Xin Liu, Chen Zhao, Bin Zheng, Qinwei Guo, Yuanyuan Yu, Dezheng Zhang, Aziguli Wulamu. Spatiotemporal and kinematic characteristics augmentation using Dual-GAN for ankle instability detection[J]. Mathematical Biosciences and Engineering, 2022, 19(10): 10037-10059. doi: 10.3934/mbe.2022469

Related Papers:

Abstract

Obtaining massive amounts of training data is often crucial for computer-assisted diagnosis using deep learning. Unfortunately, patient data is often small due to varied constraints. We develop a new approach to extract significant features from a small clinical gait analysis dataset to improve computer-assisted diagnosis of Chronic Ankle Instability (CAI) patients. In this paper, we present an approach for augmenting spatiotemporal and kinematic characteristics using the Dual Generative Adversarial Networks (Dual-GAN) to train a series of modified Long Short-Term Memory (LSTM) detection models making the training process more data-efficient. Namely, we use LSTM-, LSTM-Fully Convolutional Networks (FCN)-, and Convolutional LSTM-based detection models to identify the patients with CAI. The Dual-GAN enables the synthesized data to approximate the real data distribution visualized by the t-distributed Stochastic Neighbor Embedding (t-SNE) algorithm. Then we trained the proposed detection models using real data collected from a controlled laboratory study and mixed data from real and synthesized gait features. The detection models were tested in real data to validate the positive role in data augmentation as well as to demonstrate the capability and effectiveness of the modified LSTM algorithm for CAI detection using spatiotemporal and kinematic characteristics in walking. Dual-GAN generated efficient spatiotemporal and kinematic characteristics to augment the training set promoting the performance of CAI detection and the modified LSTM algorithm yielded an enhanced classification outcome to identify those CAI patients from a group of control subjects based on gait analysis data than any previous reports.

References

[1]	B. Burgesson, M. Glazebrook, S. Guillo, K. Matsui, M. D. Pastor, F. Peña, et al., Ankle instability (ICL 7), in ESSKA Instructional Course Lecture Book: Barcelona 2016 (eds. R. Becker, G. M. M. J. Kerkhoffs, P. E. Gelber, M. Denti, R. Seil), Springer Berlin Heidelberg, Berlin, Heidelberg, (2016), 89-99. https://doi.org/10.1007/978-3-662-49114-0_7
[2]	M. H. Leonard, Injuries of the lateral ligaments of the ankle-a clinical and experimental study, J. Bone Joint Surg. Am. , 31 (1949), 373-377. https://doi.org/10.2106/00004623-194931020-00013 doi: 10.2106/00004623-194931020-00013
[3]	E. Kemler, K. M. Thijs, I. Badenbroek, I. G. L. van de Port, A. W. Hoes, F. J. G. Backx, Long-term prognosis of acute lateral ankle ligamentous sprains: High incidence of recurrences and residual symptoms, Fam. Pract. , 33 (2016), 596-600. https://doi.org/10.1093/fampra/cmw076 doi: 10.1093/fampra/cmw076
[4]	C. J. Powden, J. M. Hoch, M. C. Hoch, Rehabilitation and improvement of health-related quality-of-life detriments in individuals with chronic ankle instability: A meta-analysis, J. Athl. Training, 52 (2017), 753-765. https://doi.org/10.4085/1062-6050-52.5.01 doi: 10.4085/1062-6050-52.5.01
[5]	R. Guo, X. Cheng, Z. C. Hou, J. Z. Ma, W. Q. Zheng, X. M. Wu, et al., A shoe-integrated sensor system for long-term center of pressure evaluation, IEEE Sens. J. , 21 (2021), 27037-27044. https://doi.org/10.1109/JSEN.2021.3116249 doi: 10.1109/JSEN.2021.3116249
[6]	S. Mollà-Casanova, M. Inglés, P. Serra-Añó, Effects of balance training on functionality, ankle instability, and dynamic balance outcomes in people with chronic ankle instability: Systematic review and meta-analysis, Clin. Rehabil. , 35 (2021), 1694-1709. https://doi.org/10.1177/02692155211022009 doi: 10.1177/02692155211022009
[7]	K. G. Migel, E. A. Wikstrom, Immediate effects of vibration biofeedback on ankle kinematics in people with chronic ankle instability, Clin. Biomech. , 90 (2021), 105495. https://doi.org/10.1016/j.clinbiomech.2021.105495 doi: 10.1016/j.clinbiomech.2021.105495
[8]	S. -W. Kim, H. G. Jung, J. S. Lee, Ligament stabilization improved clinical and radiographic outcomes for individuals with chronic ankle instability and medial ankle osteoarthritis, Knee Surg. Sports Tr. A. , 28 (2020), 3294-3300. https://doi.org/10.1007/s00167-020-05845-5 doi: 10.1007/s00167-020-05845-5
[9]	S. Ashkani-Esfahani, R. Mojahed-Yazdi, R. Bhimani, G. M. Kerkhoffs, M. Maas, C. W. DiGiovanni, et al., Deep learning algorithms improve the detection of subtle lisfranc malalignments on weightbearing radiographs, Foot Ankle Int. , 2022. https://doi.org/10.1177/10711007221093574 doi: 10.1177/10711007221093574
[10]	K. Kipp, R. M. Palmieri-Smith, Differences in kinematic control of ankle joint motions in people with chronic ankle instability, Clin. Biomech. , 28 (2013), 562-567. https://doi.org/10.1016/j.clinbiomech.2013.03.008 doi: 10.1016/j.clinbiomech.2013.03.008
[11]	R. M. Koldenhoven, J. Hart, S. Saliba, M. F. Abel, J. Hertel, Gait kinematics & kinetics at three walking speeds in individuals with chronic ankle instability and ankle sprain copers, Gait Posture, 74 (2019), 169-175. https://doi.org/10.1016/j.gaitpost.2019.09.010 doi: 10.1016/j.gaitpost.2019.09.010
[12]	T. Balasukumaran, U. Gottlieb, S. Springer, Spatiotemporal gait characteristics and ankle kinematics of backward walking in people with chronic ankle instability, Sci. Rep. , 10 (2020), 11515. https://doi.org/10.1038/s41598-020-68385-5 doi: 10.1038/s41598-020-68385-5
[13]	G. Andreopoulou, D. J. Mahad, T. H. Mercer, M. L. van der Linden, Test-retest reliability and minimal detectable change of ankle kinematics and spatiotemporal parameters in MS population, Gait Posture, 74 (2019), 218-222. https://doi.org/10.1016/j.gaitpost.2019.09.015 doi: 10.1016/j.gaitpost.2019.09.015
[14]	B. Stansfield, K. Hawkins, S. Adams, D. Church, Spatiotemporal and kinematic characteristics of gait initiation across a wide speed range, Gait Posture, 61 (2018), 331-338. https://doi.org/10.1016/j.gaitpost.2018.02.003 doi: 10.1016/j.gaitpost.2018.02.003
[15]	S. Ashkani-Esfahani, R. Mojahed Yazdi, R. Bhimani, G. M. Kerkhoffs, M. Maas, D. Guss, et al., Assessment of ankle fractures using deep learning algorithms and convolutional neural network, 7 (2021), 2473011421S00091. https://doi.org/10.1177/2473011421S00091
[16]	L. Xin, Z. Dezheng, Z. Bin, G. Qinwei, Z. Zhongshi, Gait kinematics of patients with lateral collateral ligament injuries of ankle, 2021. https://doi.org/10.21203/rs.3.rs-22139/v1
[17]	X. Liu, C. Zhao, B. Zheng, Q. Guo, Z. Zhang, A. Wulamu, et al., Synthesizing foot and ankle kinematic characteristics for lateral collateral ligament injuries detection, IEEE Access, 8 (2020), 188429-188440. https://doi.org/10.1109/ACCESS.2020.3029616 doi: 10.1109/ACCESS.2020.3029616
[18]	A. Radford, L. Metz, S. Chintala, Unsupervised representation learning with deep convolutional generative adversarial networks, preprint, arXiv: 1511.06434, 2015.
[19]	H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, et al., StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks, (2017), 5908-5916. https://doi.org/10.1109/ICCV.2017.629
[20]	Z. L. Yi, H. Zhang, P. Tan, M. L. Gong, DualGAN: Unsupervised dual learning for image-to-image translation, in Proceedings of the IEEE International Conference on Computer Vision, (2017), 2868-2876. https://doi.org/10.1109/Iccv.2017.310
[21]	X. Wang, K. Tan, Y. Chen, CapsNet and Triple-GANs towards hyperspectral classification, in 2018 Fifth International Workshop on Earth Observation and Remote Sensing Applications, (2018), 194-197. https://doi.org/10.1109/EORSA.2018.8598574
[22]	Y. Choi, M. Choi, M. Kim, J. W. Ha, S. Kim, J. Choo, StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2018), 8789-8797. https://doi.org/10.1109/Cvpr.2018.00916
[23]	J. Donahue, K. Simonyan, Large scale adversarial representation learning, in Advances in Neural Information Processing Systems 32 (NeurIPS 2019), 2019.
[24]	H. J. Tien, H. C. Yang, P. W. Shueng, J. C. Chen, Cone-beam CT image quality improvement using Cycle-Deblur consistent adversarial networks (Cycle-Deblur GAN) for chest CT imaging in breast cancer patients, Sci. Rep. , 11 (2021). https://doi.org/10.1038/s41598-020-80803-2 doi: 10.1038/s41598-020-80803-2
[25]	S. Nowozin, B. Cseke, R. Tomioka, f-GAN: Training generative neural samplers using variational divergence minimization, in Advances in Neural Information Processing Systems 29, 2016.
[26]	I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, A. Courville, Improved training of wasserstein GANs, in Advances in Neural Information Processing Systems 30 (Nips 2017), 2017.
[27]	M. Arjovsky, S. Chintala, L. Bottou, Wasserstein GAN, 2017. https://doi.org/abs/1701.07875
[28]	X. D. Mao, Q. Li, H. R. Xie, R. Y. K. Lau, Z. Wang, S. P. Smolley, Least squares generative adversarial networks, in Proceedings of the IEEE International Conference on Computer Vision, (2017), 2794-2802. https://doi.org/10.1109/Iccv.2017.304
[29]	S. Hu, Y. Shen, S. Wang, B. Lei, Brain MR to PET synthesis via bidirectional generative adversarial network, in International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, Cham, (2020), 698-707.
[30]	S. Hu, B. Lei, S. Wang, Y. Wang, Z. Feng, Y. Shen, Bidirectional mapping generative adversarial networks for brain MR to PET synthesis, IEEE. Trans. Med. Imaging, 41 (2022), 145-157. https://doi.org/10.1109/TMI.2021.3107013 doi: 10.1109/TMI.2021.3107013
[31]	W. Yu, B. Lei, S. Wang, Y. Liu, Z. Feng, Y. Hu, et al., Morphological feature visualization of alzheimer's disease via multidirectional perception GAN, IEEE Trans. Neural Networks Learn. Syst. , (2022), 1-15. https://doi.org/10.1109/TNNLS.2021.3118369 doi: 10.1109/TNNLS.2021.3118369
[32]	W. Yu, B. Lei, M. K. Ng, A. C. Cheung, Y. Shen, S. Wang, Tensorizing GAN with high-order pooling for alzheimer's disease assessment, IEEE Trans. Neural Networks Learn. Syst. , (2021), 1-15. https://doi.org/10.1109/TNNLS.2021.3063516 doi: 10.1109/TNNLS.2021.3063516
[33]	F. Pollastri, F. Bolelli, R. Paredes, C. Grana, Augmenting data with GANs to segment melanoma skin lesions, Multimedia Tools Appl. , 79 (2020), 15575-15592. https://doi.org/10.1007/s11042-019-7717-y doi: 10.1007/s11042-019-7717-y
[34]	J. Yoon, D. Jarrett, M. Schaar, Time-series generative adversarial networks, in Advances in Neural Information Processing Systems 32 (NeurIPS 2019), 2019.
[35]	J. Simon, L. Doederlein, A. S. McIntosh, D. Metaxiotis, H. G. Bock, S. I. Wolf, The Heidelberg foot measurement method: Development, description and assessment, Gait Posture, 23 (2006), 411-424. https://doi.org/10.1016/j.gaitpost.2005.07.003 doi: 10.1016/j.gaitpost.2005.07.003
[36]	A. Graves, Long short-term memory, in Supervised Sequence Labelling with Recurrent Neural Networks, Springer Berlin Heidelberg, Berlin, Heidelberg, (2012), 37-45. https://doi.org/10.1007/978-3-642-24797-2_4
[37]	F. Karim, S. Majumdar, H. Darabi, S. Chen, LSTM fully convolutional networks for time series classification, IEEE Access, 6 (2018), 1662-1669. https://doi.org/10.1109/ACCESS.2017.2779939 doi: 10.1109/ACCESS.2017.2779939
[38]	T. N. Sainath, O. Vinyals, A. Senior, H. Sak, Convolutional, long short-term memory, fully connected deep neural networks, in 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), (2015), 4580-4584. https://doi.org/10.1109/ICASSP.2015.7178838
[39]	E. Tsironi, P. Barros, C. Weber, S. Wermter, An analysis of convolutional long short-term memory recurrent neural networks for gesture recognition, Neurocomputing, 268 (2017), 76-86. https://doi.org/10.1016/j.neucom.2016.12.088 doi: 10.1016/j.neucom.2016.12.088
[40]	L. Van der Maaten, G. Hinton, Visualizing data using t-SNE, J. Mach. Learn. Res. , 9 (2008), 2579-2605.
[41]	M. Wattenberg, F. Viégas, I. Johnson, How to use t-SNE effectively, Distill, 1 (2016), e2. https://doi.org/10.23915/distill.00002 doi: 10.23915/distill.00002
[42]	S. Arora, W. Hu, P. K. Kothari, An analysis of the T-SNE algorithm for data visualization, in Conference On Learning Theory, (2018), 1455-1462.
[43]	G. Marta, F. Simona, C. Andrea, B. Dario, S. Stefano, V. Federico, et al., Wearable biofeedback suit to promote and monitor aquatic exercises: A feasibility study, IEEE Trans. Instrum. Meas., 69 (2020), 1219-1231. https://doi.org/10.1109/TIM.2019.2911756 doi: 10.1109/TIM.2019.2911756
[44]	A. R. Anwary, H. Yu, M. Vassallo, Optimal foot location for placing wearable IMU sensors and automatic feature extraction for gait analysis, IEEE Sens. J., 18 (2018), 2555-2567. https://doi.org/10.1109/JSEN.2017.2786587 doi: 10.1109/JSEN.2017.2786587
[45]	S. Qiu, H. Zhao, N. Jiang, Z. Wang, L. Liu, Y. An, et al., Multi-sensor information fusion based on machine learning for real applications in human activity recognition: State-of-the-art and research challenges, Inf. Fusion, 80 (2022), 241-265. https://doi.org/10.1016/j.inffus.2021.11.006 doi: 10.1016/j.inffus.2021.11.006
[46]	Z. Sun, Y. Tian, H. Li, J. Wang, A superlinear convergence feasible sequential quadratic programming algorithm for bipedal dynamic walking robot via discrete mechanics and optimal control, Optim. Control. Appl. Methods, 37 (2016), 1139-1161. https://doi.org/10.1002/oca.2228 doi: 10.1002/oca.2228
[47]	Z. Sun, T. Shi, L. Wei, Y. Sun, K. Liu, L. Jin, Noise-suppressing zeroing neural network for online solving time-varying nonlinear optimization problem: A control-based approach, Neural Comput. Appl., 32 (2020), 11505-11520. https://doi.org/10.1007/s00521-019-04639-2 doi: 10.1007/s00521-019-04639-2
[48]	Z. Sun, F. Li, B. Zhang, Y. Sun, L. Jin, Different modified zeroing neural dynamics with inherent tolerance to noises for time-varying reciprocal problems: A control-theoretic approach, Neurocomputing, 337 (2019), 165-179. https://doi.org/10.1016/j.neucom.2019.01.064 doi: 10.1016/j.neucom.2019.01.064

Reader Comments

Your name:*

Email:*
© 2022 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)