Enhancing autonomous vehicle safety in rain: a data centric approach for clear vision

Mark A. Seferian; Jidong J. Yang; Mark A. Seferian; Jidong J. Yang

doi:10.3934/aci.2024017

Applied Computing and Intelligence

2024, Volume 4, Issue 2: 282-299. doi: 10.3934/aci.2024017

Previous Article Next Article

Research article

Enhancing autonomous vehicle safety in rain: a data centric approach for clear vision

Mark A. Seferian ,
Jidong J. Yang ^,

Smart Mobility and Infrastructure Laboratory, College of Engineering, University of Georgia, Athens, GA, 30602, USA

Academic Editor: Pasi Fränti

Received: 23 November 2024 Revised: 25 December 2024 Accepted: 26 December 2024 Published: 30 December 2024

Autonomous vehicles (AV) face significant challenges in navigating adverse weather, particularly rain, due to the visual impairment of camera-based systems. In this study, we leveraged contemporary deep learning techniques to mitigate these challenges, aiming to develop a vision model that processes live vehicle camera feeds to eliminate rain-induced visual hindrances, yielding visuals closely resembling clear, rain-free scenes. Using the Car Learning to Act (CARLA) simulation environment, we generated a comprehensive dataset of clear and rainy images for model training and testing. In our model, we employed a classic encoder-decoder architecture with skip connections and concatenation operations. It was trained using novel batching schemes designed to effectively distinguish high-frequency rain patterns from low-frequency scene features across successive image frames. To evaluate the model's performance, we integrated it with a steering module that processes front-view images as input. The results demonstrated notable improvements in steering accuracy, underscoring the model's potential to enhance navigation safety and reliability in rainy weather conditions.
- autonomous vehicles,
- vehicle safety,
- adverse weather navigation,
- deep learning,
- computer vision,
- image deraining,
- batching schemes,
- CARLA simulator,
- steering performance
Citation: Mark A. Seferian, Jidong J. Yang. Enhancing autonomous vehicle safety in rain: a data centric approach for clear vision[J]. Applied Computing and Intelligence, 2024, 4(2): 282-299. doi: 10.3934/aci.2024017

Related Papers:

Abstract

Autonomous vehicles (AV) face significant challenges in navigating adverse weather, particularly rain, due to the visual impairment of camera-based systems. In this study, we leveraged contemporary deep learning techniques to mitigate these challenges, aiming to develop a vision model that processes live vehicle camera feeds to eliminate rain-induced visual hindrances, yielding visuals closely resembling clear, rain-free scenes. Using the Car Learning to Act (CARLA) simulation environment, we generated a comprehensive dataset of clear and rainy images for model training and testing. In our model, we employed a classic encoder-decoder architecture with skip connections and concatenation operations. It was trained using novel batching schemes designed to effectively distinguish high-frequency rain patterns from low-frequency scene features across successive image frames. To evaluate the model's performance, we integrated it with a steering module that processes front-view images as input. The results demonstrated notable improvements in steering accuracy, underscoring the model's potential to enhance navigation safety and reliability in rainy weather conditions.

References

[1]	P. Vincent, H. Larochelle, Y. Bengio, P. A. Manzagol, Extracting and composing robust features with denoising autoencoders, Proceedings of the 25th International Conference on Machine Learning, 2008, 1096–1103. https://doi.org/10.1145/1390156.1390294
[2]	A. Radford, L. Metz, S. Chintala, Unsupervised representation learning with deep convolutional generative adversarial networks, arXiv: 1511.06434. https://arXiv.org/abs/1511.06434
[3]	O. Ronneberger, P. Fischer, T. Brox, U-net: convolutional networks for biomedical image segmentation, In: Medical image computing and computer-assisted intervention–MICCAI 2015, Cham: Springer, 2015,234–241. https://doi.org/10.1007/978-3-319-24574-4_28
[4]	T. Karras, T. Aila, S. Laine, J. Lehtinen, Progressive growing of gans for improved quality, stability, and variation, arXiv: 1710.10196. https://doi.org/10.48550/arXiv.1710.10196
[5]	A. Brock, J. Donahue, K. Simonyan, Large scale GAN training for high fidelity natural image synthesis, arXiv: 1809.11096. https://doi.org/10.48550/arXiv.1809.11096
[6]	H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, et al., Stackgan: text to photo-realistic image synthesis with stacked generative adversarial networks, Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, 5907–5915.
[7]	T. Karras, S. Laine, T. Aila, A style-based generator architecture for generative adversarial networks, Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, 4396–4405. https://doi.org/10.1109/CVPR.2019.00453
[8]	W. Xu, C. Long, R. Wang, G. Wang, Drb-gan: a dynamic resblock generative adversarial network for artistic style transfer, Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, 6383–6392.
[9]	H. Zhen, Y. Shi, J. Yang, J. Vehni, Co-supervised learning paradigm with conditional generative adversarial networks for sample-efficient classification, Appl. Comput. Intell., 3 (2023), 13–26. https://doi.org/10.3934/aci.2023002 doi: 10.3934/aci.2023002
[10]	S. Motamed, P. Rogalla, F. Khalvati, Data augmentation using generative adversarial networks (GANs) for GAN-based detection of Pneumonia and COVID-19 in chest X-ray images, Informatics in Medicine Unlocked, 27 (2021), 100779. https://doi.org/10.1016/j.imu.2021.100779 doi: 10.1016/j.imu.2021.100779
[11]	A. Jadli, M. Hain, A. Chergui, A. Jaize, DCGAN-based data augmentation for document classification, Proceedings of IEEE 2nd International Conference on Electronics, Control, Optimization and Computer Science (ICECOCS), 2020, 1–5. https://doi.org/10.1109/icecocs50124.2020.9314379
[12]	A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, et al., Zero-shot text-to-image generation, Proceedings of the 38th International Conference on Machine Learning, 2021, 8821–8831.
[13]	R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, High-resolution image synthesis with latent diffusion models, Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, 10684–10695. https://doi.org/10.1109/CVPR52688.2022.01042
[14]	B. Xia, Y. Zhang, S. Wang, Y. Wang, X. Wu, Y. Tian, et al., Diffir: efficient diffusion model for image restoration, Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV), 2023, 13049–13059. https://doi.org/10.1109/ICCV51070.2023.01204
[15]	D. Ren, W. Zuo, Q. Hu, P. Zhu, D. Meng, Progressive image deraining networks: a better and simpler baseline, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, 3937–3946. https://doi.org/10.1109/CVPR.2019.00406
[16]	M. Bojarski, D. Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, et al., End to end learning for self-driving cars, arXiv: 1604.07316. https://doi.org/10.48550/arXiv.1604.07316
[17]	H. Zhang, V. M. Patel, Density-aware single image de-raining using a multi-stream dense network, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, 695–704. https://doi.org/10.1109/CVPR.2018.00079
[18]	W. Yang, R. T. Tan, J. Feng, J. Liu, Z. Guo, S. Yan, Deep joint rain detection and removal from a single image, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, 1357–1366. https://doi.org/10.1109/CVPR.2017.183
[19]	X. Fu, J. Huang, X. Ding, Y. Liao, J. Paisley, Clearing the skies: a deep network architecture for single-image rain removal, IEEE Trans. Image Process., 26 (2017), 2944–2956. https://doi.org/10.1109/tip.2017.2691802 doi: 10.1109/tip.2017.2691802
[20]	Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard, et al., Backpropagation applied to handwritten zip code recognition, Neural Comput., 1 (1989), 541–551. https://doi.org/10.1162/neco.1989.1.4.541 doi: 10.1162/neco.1989.1.4.541
[21]	S. Zamir, A. Arora, S. Khan, M. Hayat, F. Khan, M. Yang, Restormer: efficient transformer for high-resolution image restoration, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, 5728–5739. https://doi.org/10.1109/cvpr52688.2022.00564
[22]	Y. Zhang, D. Li, X. Shi, D. He, K. Song, X. Wang, et al., Kbnet: kernel basis network for image restoration, arXiv: 2303.02881. https://doi.org/10.48550/arXiv.2303.02881
[23]	I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, et al., Generative adversarial networks, Commun. ACM, 63 (2020), 139–144. https://doi.org/10.1145/3422622 doi: 10.1145/3422622
[24]	Y. Wei, Z. Zhang, Y. Wang, M. Xu, Y. Yang, S. Yan, et al., Deraincyclegan: rain attentive cyclegan for single image deraining and rainmaking, IEEE Trans. Image Process., 30 (2021), 4788–4801. https://doi.org/10.1109/TIP.2021.3074804 doi: 10.1109/TIP.2021.3074804
[25]	H. Zhang, V. Sindagi, V. M. Patel, Image de-raining using a conditional generative adversarial network, IEEE Trans. Circ. Syst. Vid., 30 (2020), 3943–3956. https://doi.org/10.1109/tcsvt.2019.2920407 doi: 10.1109/tcsvt.2019.2920407
[26]	C. Wang, C. Xu, C. Wang, D. Tao, Perceptual adversarial networks for image-to-image transformation, IEEE Trans. Image Process., 27 (2018), 4066–4079. https://doi.org/10.1109/TIP.2018.2836316 doi: 10.1109/TIP.2018.2836316
[27]	P. Xiang, L. Wang, F. Wu, J. Cheng, M. Zhou, Single-image de-raining with feature-supervised generative adversarial network, IEEE Signal Proc. Let., 26 (2019), 650–654. https://doi.org/10.1109/LSP.2019.2903874 doi: 10.1109/LSP.2019.2903874
[28]	Y. Ren, M. Nie, S. Li, C. Li, Single image de-raining via improved generative adversarial nets, Sensors, 20 (2020), 1591. https://doi.org/10.3390/s20061591 doi: 10.3390/s20061591
[29]	A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, V. Koltun, CARLA: an open urban driving simulator, arXiv: 1711.03938. https://doi.org/10.48550/arXiv.1711.03938
[30]	S. Ioffe, C. Szegedy, Batch normalization: accelerating deep network training by reducing internal covariate shift, Proceedings of the 32nd International Conference on International Conference on Machine Learning, 2015, 448–456.
[31]	J. Springenberg, A. Dosovitskiy, T. Brox, M. A. Riedmiller, Striving for simplicity: the all convolutional net, arXiv: 1412.6806. https://doi.org/10.48550/arXiv.1412.6806
[32]	D. Kingma, J. Ba, Adam: a method for stochastic optimization, arXiv: 1412.6980. https://doi.org/10.48550/arXiv.1412.6980
[33]	J. Ho, A. Jain, P. Abbeel, Denoising diffusion probabilistic models, Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020, 6840–6851.

Reader Comments

Your name:*

Email:*
© 2024 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)