Research article

ODE-RU: a dynamical system view on recurrent neural networks


  • Received: 12 October 2021 Revised: 03 December 2021 Accepted: 20 December 2021 Published: 28 December 2021
  • The core of the demonstration of this paper is to interpret the forward propagation process of machine learning as a parameter estimation problem of nonlinear dynamical systems. This process is to establish a connection between the Recurrent Neural Network and the discrete differential equation, so as to construct a new network structure: ODE-RU. At the same time, under the inspiration of the theory of ordinary differential equations, we propose a new forward propagation mode. In a large number of simulations and experiments, the forward propagation not only shows the trainability of the new architecture, but also achieves a low training error on the basis of main-taining the stability of the network. For the problem requiring long-term memory, we specifically study the obstacle shape reconstruction problem using the backscattering far-field features data set, and demonstrate the effectiveness of the proposed architecture using the data set. The results show that the network can effectively reduce the sensitivity to small changes in the input feature. And the error generated by the ordinary differential equation cyclic unit network in inverting the shape and position of obstacles is less than $ 10^{-2} $.

    Citation: Pinchao Meng, Xinyu Wang, Weishi Yin. ODE-RU: a dynamical system view on recurrent neural networks[J]. Electronic Research Archive, 2022, 30(1): 257-271. doi: 10.3934/era.2022014

    Related Papers:

  • The core of the demonstration of this paper is to interpret the forward propagation process of machine learning as a parameter estimation problem of nonlinear dynamical systems. This process is to establish a connection between the Recurrent Neural Network and the discrete differential equation, so as to construct a new network structure: ODE-RU. At the same time, under the inspiration of the theory of ordinary differential equations, we propose a new forward propagation mode. In a large number of simulations and experiments, the forward propagation not only shows the trainability of the new architecture, but also achieves a low training error on the basis of main-taining the stability of the network. For the problem requiring long-term memory, we specifically study the obstacle shape reconstruction problem using the backscattering far-field features data set, and demonstrate the effectiveness of the proposed architecture using the data set. The results show that the network can effectively reduce the sensitivity to small changes in the input feature. And the error generated by the ordinary differential equation cyclic unit network in inverting the shape and position of obstacles is less than $ 10^{-2} $.



    加载中


    [1] J. Collins, J. Sohl-Dickstein, D. Sussillo, Capacity and trainability in recurrent neural networks, Paper presented at 5th International Conference on Learning Representations, 2017.
    [2] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural Comput., 9 (1997), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735 doi: 10.1162/neco.1997.9.8.1735
    [3] K. Cho, B. V. Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, et al., Learning phrase representations using RNN encoder-decoder for statistical machine translation, Comput. Sci., (2014), 1723–1734. https://doi.org/10.3115/v1/D14-1179 doi: 10.3115/v1/D14-1179
    [4] L. Bottou, F. E. Curtis, J. Nocedal, Optimization methods for large-scale machine learning, SIAM, 60 (2018), 223–231. https://doi.org/10.1137/16M1080173 doi: 10.1137/16M1080173
    [5] B. Chang, M. Chen, E. Haber, E. H. Chi, AntisymmetricRNN: a dynamical system view on recurrent neural networks, Paper presented at 7th International Conference on Learning Representations, 2019.
    [6] E. Haber, K. Lensink, T. Eran, L. Ruthotto, IMEXnet: A forward stable deep neural network, Paper ppresented at: Proceedings of the 36th International Conference on Machine Learning, 2019.
    [7] W. E, A proposal on machine learning via dynamical systems, Commun. Math. Stat., 1 (2017), 1–11. http://doi.org/10.1007/s40304-017-0103-z doi: 10.1007/s40304-017-0103-z
    [8] Y. Lu, A. Zhong, Q. Li, B. Dong, Beyond finite layer neural networks: bridging deep architectures and numerical differential equations, Ppaer pressented at: Proceedings of Machine Learning Research Proceedings of the 35th International Conference on Machine Learning, 2018.
    [9] R. T. Q. Chen, Y. Rubanova, J. Bettencourt, D. Duvenaud, Neural ordinary differential equations, Paper present at : Annual Conference on Neural Information Processing Systems, 2018.
    [10] P. Meng, L. Su, W. Yin, S. Zhang, Solving a kind of inverse scattering problem of acoustic waves based on linear sampling method and neural network, Alexandria Eng. J., 59 (2020), 1451–1462. http://doi.org/10.1016/j.aej.2020.03.047 doi: 10.1016/j.aej.2020.03.047
    [11] W. Yin, W. Yang, H. Liu, A neural network scheme for recovering scattering obstacles with limited phaseless far-field data, J. Comput. Phys., 417 (2020). http://doi.org/10.1016/j.jcp.2020.109594 doi: 10.1016/j.jcp.2020.109594
    [12] Y. Guo, D. Hoemberg, G. Hu, J. Li, H. Liu, A time domain sampling method for inverse acoustic scattering problems, J. Comput. Phys., 314 (2016), 647–660. http://doi.org/10.1016/j.jcp.2016.03.046 doi: 10.1016/j.jcp.2016.03.046
    [13] D. Zhang, F. Sun, L. Lu, Y. Guo, A harmonic polynomial method with a regularization strategy for the boundary value problems of Laplace's equation, Appl. Math. Lett, 79 (2018), 100–104. http://doi.org/10.1016/j.aml.2017.12.003 doi: 10.1016/j.aml.2017.12.003
    [14] H. Liu, M. Petrini, L. Rondi, J. Xiao, Stable determination of sound-hard polyhedral scatterers by a minimal number of scattering measurements, J. Differ. Equations, 262 (2018), 1631–1670. http://doi.org/10.1016/j.jde.2016.10.021 doi: 10.1016/j.jde.2016.10.021
    [15] H. Liu, X. Liu, Recovery of an embedded obstacle and its surrounding medium from formally determined scattering data, Inverse Probl., 33 (2017), 1–20. http://doi.org/10.1088/1361-6420/aa6770 doi: 10.1088/1361-6420/aa6770
    [16] H. Liu, X. Liu, X. Wang, Y. Wang, On a novel inverse scattering scheme using resonant modes with enhanced imaging resolution, Inverse Probl., 35 (2019). http://doi.org/10.1088/1361-6420/ab2932 doi: 10.1088/1361-6420/ab2932
    [17] W. Yin, J. Ge, P. Meng, F. Qu, A neural network method for the inverse scattering problem of impenetrable cavities, Electron. Res. Arch., 28 (2020), 1123–1142. http://doi.org/10.3934/era.2020062 doi: 10.3934/era.2020062
    [18] J. Xie, Y. chen, A numerical analysis method of fixed points and their stability in a complex dynamical systems, CCAMMS, (2019), 59–62.
    [19] W. Stephen, D. S. Mazel, Introduction to applied nonlinear dynamical systems and chaos, Name J., 33 (1991), 81. http://doi.org/10.1063/1.4822950 doi: 10.1063/1.4822950
    [20] G. Wang, Ordinary differential equations, Higher Education Press, (2013), 250–260.
    [21] E. Haber, L. Ruthotto, Stable architectures for deep neural networks, Inverse Probl., 34 (2018). http://doi.org/10.1088/1361-6420/aa9a90 doi: 10.1088/1361-6420/aa9a90
    [22] C. Bo, M. Chen, E. Haber, E. Chi, Antisymmetricrnn: a dynamical system view on recurrent neural networks, Paper present at: 7th International Conference on Learning Representations, 2019.
    [23] U. Ascher, L. Petzold, Computer methods for ordinary differential equations and differential-algebraic equations, in Science press, 2009.
    [24] D. Colton, Inverse acoustic and electromagnetic scattering theory, Inverse Prob., 47 (2003), 67–110. http://doi.org/10.1007/978-3-662-03537-5 doi: 10.1007/978-3-662-03537-5
    [25] D. Colton, J. Coyle, P. Monk, Recent developments in inverse acoustic scattering theory, SIAM, 42 (2000), 369–414. http://doi.org/10.1137/S0036144500367337 doi: 10.1137/S0036144500367337
    [26] A. Kirsch, New characterizations of solutions in inverse scattering theory, Appl. Anal., 76 (2000), 319–350. http://doi.org/10.1080/00036810008840888 doi: 10.1080/00036810008840888
    [27] A. Kirsch, N. Grinberg, The factorization method for inverse problems, Oxford University Press, 2008.
    [28] F. Zeng, P. Suarez, J. Sun, A decomposition method for an interior inverse scattering problem, Inverse Probl. Imaging, 7 (2013), 291–303. http://doi.org/10.3934/ipi.2013.7.291 doi: 10.3934/ipi.2013.7.291
    [29] J. Li, H. Liu, W. Tsui, X. Wang, An inverse scattering approach for geometric body generation: a machine learning perspective, Math. Eng., 1 (2019), 800–823. http://doi.org/10.3934/mine.2019.4.800 doi: 10.3934/mine.2019.4.800
    [30] J. Li, H. Liu, J. Zou, Locating multiple multiscale acoustic scatterers, Multiscale Model. Simul., 12 (2014), 927–952. http://doi.org/10.1137/13093409X doi: 10.1137/13093409X
    [31] H. Liu, J. Zou, Uniqueness in an inverse acoustic obstacle scattering problem for both sound-hard and sound-soft polyhedral scatterers, Inverse Probl., 22 (2016), 515–524. http://doi.org/10.1088/0266-5611/22/2/008 doi: 10.1088/0266-5611/22/2/008
    [32] Q. V. Le, N. Jaitly, G. E. Hinton, A simple way to initialize recurrent networks of rectified linear units, Comput. Sci., preprint, arXiv: 1504.00941.
  • Reader Comments
  • © 2022 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)
通讯作者: 陈斌, bchen63@163.com
  • 1. 

    沈阳化工大学材料科学与工程学院 沈阳 110142

  1. 本站搜索
  2. 百度学术搜索
  3. 万方数据库搜索
  4. CNKI搜索

Metrics

Article views(2650) PDF downloads(294) Cited by(2)

Article outline

Figures and Tables

Figures(5)  /  Tables(4)

Other Articles By Authors

/

DownLoad:  Full-Size Img  PowerPoint
Return
Return

Catalog