Mathematical Biosciences and Engineering, 2019, 16(6): 8025-8059. doi: 10.3934/mbe.2019404 Review Special Issues

Export file:

Format

RIS(for EndNote,Reference Manager,ProCite)
BibTex
Text

Content

Citation Only
Citation and Abstract

Mathematical studies of the dynamics of finite-size binary neural networks: A review of recent progress

Laboratory of Neural Computation, Center for Neuroscience and Cognitive Systems@UniTn, Istituto Italiano di Tecnologia, 38068 Rovereto, Italy

Received: , Accepted: , Published:

Special Issues: Neural Coding 2018

Abstract Full Text(HTML) Figure/Table Related pages

Several mathematical approaches to studying analytically the dynamics of neural networks rely on mean-field approximations, which are rigorously applicable only to networks of infinite size. However, all existing real biological networks have finite size, and many of them, such as microscopic circuits in invertebrates, are composed only of a few tens of neurons. Thus, it is important to be able to extend to small-size networks our ability to study analytically neural dynamics. Analytical solutions of the dynamics of small-size neural networks have remained elusive for many decades, because the powerful methods of statistical analysis, such as the central limit theorem and the law of large numbers, do not apply to small networks. In this article, we critically review recent progress on the study of the dynamics of small networks composed of binary neurons. In particular, we review the mathematical techniques we developed for studying the bifurcations of the network dynamics, the dualism between neural activity and membrane potentials, cross-neuron correlations, and pattern storage in stochastic networks. Then, we compare our results with existing mathematical techniques for studying networks composed of a finite number of neurons. Finally, we highlight key challenges that remain open, future directions for further progress, and possible implications of our results for neuroscience.

Figure/Table

Supplementary

Article Metrics

References

1. W. W. Lytton, From computer to brain:Foundations of computational neuroscience, SpringerVerlag New York, 2002, URL https://doi.org/10.1007/b98859.

2. B. Ermentrout, Neural networks as spatio-temporal pattern-forming systems, Rep. Prog. Phys., 61 (1998), 353-430, URL http://doi.org/10.1088/0034-4885/61/4/002.

3. E. M. Izhikevich, Dynamical systems in neuroscience, MIT Press, 2007, URL https://mitpress.mit.edu/books/dynamical-systems-neuroscience.

4. P. Ashwin, S. Coombes and R. Nicks, Mathematical frameworks for oscillatory network dynamics in neuroscience, J. Math. Neurosci., 6 (2016), 2, URL https://doi.org/10.1186/s13408-015-0033-6.

5. S. I. Amari, Learning patterns and pattern sequences by self-organizing nets of threshold elements, IEEE Trans. Comput., C-21 (1972), 1197-1206, URL https://doi.org/10.1109/T-C.1972.223477.

6. W. A. Little, The existence of persistent states in the brain, Math. Biosci., 19 (1974), 101-120, URL https://doi.org/10.1016/0025-5564(74)90031-5.

7. J. J. Hopfield, Neural networks and physical systems with emergent collective computational abilities, Proc. Natl. Acad. Sci. U.S.A., 79 (1982), 2554-2558, URL http://dx.doi.org/10.1073/pnas.79.8.2554.

8. A. C. C. Coolen, Chapter 14, Statistical mechanics of recurrent neural networks I:Statics, vol. 4 of Handbook of Biological Physics, North-Holland, 2001, 553-618, URL https://doi.org/10.1016/S1383-8121(01)80017-8.

9. A. C. C. Coolen, Chapter 15, Statistical mechanics of recurrent neural networks Ⅱ:Dynamics, vol. 4 of Handbook of Biological Physics, North-Holland, 2001, 619-684, URL https://doi.org/10.1016/S1383-8121(01)80018-X.

10. D. Kernell, The adaptation and the relation between discharge frequency and current strength of cat lumbosacral motoneurones stimulated by long-lasting injected currents, Acta Physiol. Scand., 65 (1965), 65-73, URL https://doi.org/10.1111/j.1748-1716.1965.tb04250.x.

11. D. Kernell, High-frequency repetitive firing of cat lumbosacral motoneurones stimulated by longlasting injected currents, Acta Physiol. Scand., 65 (1965), 74-86, URL https://doi.org/10.1111/j.1748-1716.1965.tb04251.x.

12. P. Peretto, Collective properties of neural networks:A statistical physics approach, Biol. Cybern., 50 (1984), 51-62, URL https://doi.org/10.1007/BF00317939.

13. I. Ginzburg and H. Sompolinsky, Theory of correlations in stochastic neural networks, Phys. Rev. E, 50 (1994), 3171-3191, URL https://doi.org/10.1103/PhysRevE.50.3171.

14. D. Fasoli, A. Cattani and S. Panzeri, Pattern storage, bifurcations and groupwise correlation structure of an exactly solvable asymmetric neural network model, Neural Comput., 30 (2018), 1258-1295, URL https://doi.org/10.1162/NECO_a_01069.

15. D. Fasoli and S. Panzeri, Optimized brute-force algorithms for the bifurcation analysis of a binary neural network model, Phys. Rev. E, 99 (2019), 012316, URL https://doi.org/10.1103/PhysRevE.99.012316.

16. D. Fasoli and S. Panzeri, Stationary-state statistics of a binary neural network model with quenched disorder, Entropy, 21 (2019), 630, URL https://doi.org/10.3390/e21070630.

17. D. J. Amit, Modeling brain function:The world of attractor neural networks, Cambridge University Press, 1989, URL https://doi.org/10.1017/CBO9780511623257.

18. S. I. Amari, Dynamics of pattern formation in lateral-inhibition type neural fields, Biol. Cybern., 27 (1977), 77-87, URL https://doi.org/10.1007/BF00337259.

19. C. van Vreeswijk and H. Sompolinsky, Chaos in neuronal networks with balanced excitatory and inhibitory activity, Science, 274 (1996), 1724-1726, URL https://doi.org/10.1126/science.274.5293.1724.

20. S. I. Amari, Homogeneous nets of neuron-like elements, Biol. Cybern., 17 (1975), 211-220, URL https://doi.org/10.1007/BF00339367.

21. D. Sherrington and S. Kirkpatrick, Solvable model of a spin-glass, Phys. Rev. Lett., 35 (1976), 1792-1796, URL https://doi.org/10.1103/PhysRevLett.35.1792.

22. S. Kirkpatrick and D. Sherrington, Infinite-ranged models of spin-glasses, Phys. Rev. B, 17 (1978), 4384-4403, URL https://doi.org/10.1103/PhysRevB.17.4384.

23. M. Mézard, N. Sourlas, G. Toulouse, et al., Replica symmetry breaking and the nature of the spin glass phase, J. Phys., 45 (1984), 843-854, URL https://doi.org/10.1051/jphys:01984004505084300.

24. M. Mézard, G. Parisi and M. Virasoro, Spin glass theory and beyond:An introduction to the replica method and its applications, World Scientific Singapore, 1986, URL http://www.worldscientific.com/worldscibooks/10.1142/0271.

25. R. J. Glauber, Time dependent statistics of the Ising model, J. Math. Phys., 4 (1963), 294-307, URL https://doi.org/10.1063/1.1703954.

26. A. C. C. Coolen and D. Sherrington, Dynamics of fully connected attractor neural networks near saturation, Phys. Rev. Lett., 71 (1993), 3886-3889, URL https://doi.org/10.1103/PhysRevLett.71.3886.

27. R. W. Williams and K. Herrup, The control of neuron number, Ann. Rev. Neurosci., 11 (1988), 423-453, URL https://doi.org/10.1146/annurev.ne.11.030188.002231.

28. V. B. Mountcastle, The columnar organization of the neocortex, Brain, 120 (1997), 701-722, URL https://doi.org/10.1093/brain/120.4.701.

29. M. Helmstaedter, C. P. de Kock, D. Feldmeyer, et al., Reconstruction of an average cortical column in silico, Brain Res. Rev., 55 (2007), 193-203, URL https://doi.org/10.1016/j.brainresrev.2007.07.011.

30. H. S. Meyer, V. C. Wimmer, M. Oberlaender, et al., Number and laminar distribution of neurons in a thalamocortical projection column of rat vibrissal cortex, Cereb. Cortex, 20 (2010), 2277-2286, URL https://doi.org/10.1093/cercor/bhq067.

31. M. De Luca, C. F. Beckmann, N. De Stefano, et al., fMRI resting state networks define distinct modes of long-distance interactions in the human brain, NeuroImage, 29 (2006), 1359-1367, URL https://doi.org/10.1016/j.neuroimage.2005.08.035.

32. D. Mantini, M. G. Perrucci, C. Del Gratta, et al., Electrophysiological signatures of resting state networks in the human brain, Proc. Natl. Acad. Sci. USA, 104 (2007), 13170-13175, URL https://doi.org/10.1073/pnas.0700668104.

33. R. D. Beer, On the dynamics of small continuous-time recurrent neural networks, Adapt. Behav., 3 (1995), 469-509, URL https://doi.org/10.1177/105971239500300405.

34. F. Pasemann, Complex dynamics and the structure of small neural networks, Network-Comp. Neural, 13 (2002), 195-216, URL http://doi.org/10.1080/net.13.2.195.216.

35. D. Fasoli, A. Cattani and S. Panzeri, The complexity of dynamics in small neural circuits, PLoS Comput. Biol., 12 (2016), e1004992, URL https://doi.org/10.1371/journal.pcbi.1004992.

36. C. van Vreeswijk and H. Sompolinsky, Chaotic balanced state in a model of cortical circuits, Neural Comput., 10 (1998), 1321-1371, URL https://doi.org/10.1162/089976698300017214.

37. B. Cessac, Increase in complexity in random neural networks, J. Phys. I France, 5 (1995), 409-432, URL http://doi.org/10.1051/jp1:1995135.

38. A. Renart, J. De La Rocha, P. Bartho, et al., The asynchronous state in cortical circuits, Science, 327 (2010), 587-590, URL https://doi.org/10.1126/science.1179850.

39. V. Pernice, B. Staude, S. Cardanobile, et al., Recurrent interactions in spiking networks with arbitrary topology, Phys. Rev. E, 85 (2012), 031916, URL https://doi.org/10.1103/PhysRevE.85.031916.

40. J. Trousdale, Y. Hu, E. Shea-Brown, et al., Impact of network structure and cellular response on spike time correlations, PLoS Comput. Biol., 8 (2012), e1002408, URL https://doi.org/10.1371/journal.pcbi.1002408.

41. T. Tetzlaff, M. Helias, G. T. Einevoll, et al., Decorrelation of neural-network activity by inhibitory feedback, PLoS Comput. Biol., 8 (2012), e1002596, URL https://doi.org/10.1371/journal.pcbi.1002596.

42. M. Helias, T. Tetzlaff and M. Diesmann, Echoes in correlated neural systems, New J. Phys., 15 (2013), 023002, URL https://doi.org/10.1088/1367-2630/15/2/023002.

43. T. Schwalger, M. Deger and W. Gerstner, Towards a theory of cortical columns:From spiking neurons to interacting neural populations of finite size, PLoS Comput. Biol., 13 (2017), e1005507, URL https://doi.org/10.1371/journal.pcbi.1005507.

44. D. Dahmen, H. Bos and M. Helias, Correlated fluctuations in strongly coupled binary networks beyond equilibrium, Phys. Rev. X, 6 (2016), 031024, URL https://doi.org/10.1103/PhysRevX.6.031024.

45. Y. A. Kuznetsov, Elements of applied bifurcation theory, vol. 112, Springer-Verlag New York, 1998, URL https://doi.org/10.1007/978-1-4757-3978-7.

46. J. R. L. De Almeida and D. J. Thouless, Stability of the Sherrington-Kirkpatrick solution of a spin glass model, J. Phys. A:Math. Gen., 11 (1978), 983-990, URL https://doi.org/10.1088/0305-4470/11/5/028.

47. R. I. Leine, D. H. Van Campen and B. L. Van De Vrande, Bifurcations in nonlinear discontinuous systems, Nonlinear Dyn., 23 (2000), 105-164, URL https://doi.org/10.1023/A:1008384928636.

48. J. Awrejcewicz and C. H. Lamarque, Bifurcation and chaos in nonsmooth mechanical systems, World Scientific, 2003, URL http://doi.org/10.1142/5342.

49. R. I. Leine and D. H. Van Campen, Bifurcation phenomena in non-smooth dynamical systems, Eur. J. Mech. A-Solid, 25 (2006), 595-616, URL https://doi.org/10.1016/j.euromechsol.2006.04.004.

50. O. Makarenkov and J. S. W. Lamb, Dynamics and bifurcations of nonsmooth systems:A survey, Physica D, 241 (2012), 1826-1844, URL https://doi.org/10.1016/j.physd.2012.08.002.

51. J. Harris and B. Ermentrout, Bifurcations in the Wilson-Cowan equations with nonsmooth firing rate, SIAM J. Appl. Dyn. Syst., 14 (2015), 43-72, URL https://doi.org/10.1137/140977953.

52. S. Parui and S. Banerjee, Border collision bifurcations at the change of state-space dimension, Chaos, 12 (2002), 1054-1069, URL https://doi.org/10.1063/1.1521390.

53. B. Cessac and T. Viéville, On dynamics of integrate-and-fire neural networks with conductance based synapses, Front. Comput. Neurosci., 2 (2008), 2, URL https://doi.org/10.3389/neuro.10.002.2008.

54. V. Avrutin, M. Schanz and S. Banerjee, Multi-parametric bifurcations in a piecewise-linear discontinuous map, Nonlinearity, 19 (2006), 1875-1906, URL https://doi.org/10.1088/0951-7715/19/8/007.

55. R. Kötter, Neuroscience databases:A practical guide, Springer US, 2003, URL http://doi. org/10.1007/978-1-4615-1079-6.

56. T. C. Potjans and M. Diesmann, The cell-type specific cortical microcircuit:Relating structure and activity in a full-scale spiking network model, Cereb. Cortex, 24 (2014), 785-806, URL https://doi.org/10.1093/cercor/bhs358.

57. C. Boucsein, M. Nawrot, P. Schnepel, et al., Beyond the cortical column:Abundance and physiology of horizontal connections imply a strong role for inputs from the surround, Front. Neurosci., 5 (2011), 32, URL https://doi.org/10.3389/fnins.2011.00032.

58. S. E. Palmer, O. Marre, M. J. Berry, et al., Predictive information in a sensory population, Proc. Natl. Acad. Sci. USA, 112 (2015), 6908-6913, URL https://doi.org/10.1073/pnas.1506855112.

59. V. Rostami, P. Porta Mana, S. Grün, et al., Bistability, non-ergodicity, and inhibition in pairwise maximum-entropy models, PLoS Comput. Biol., 13 (2017), e1005762, URL https://doi.org/10.1371/journal.pcbi.1005762.

60. S. I. Amari, Characteristics of random nets of analog neuron-like elements, IEEE Trans. Syst. Man Cybern., SMC-2 (1972), 643-657, URL https://doi.org/10.1109/TSMC.1972.4309193.

61. G. Hermann and J. Touboul, Heterogeneous connections induce oscillations in large-scale networks, Phys. Rev. Lett., 109 (2012), 018702, URL https://doi.org/10.1103/PhysRevLett.109.018702.

62. R. J. Vaughan and W. N. Venables, Permanent expressions for order statistic densities, J. R. Stat. Soc. Ser. B, 34 (1972), 308-310, URL http://www.jstor.org/stable/2985190.

63. R. B. Bapat and M. I. Beg, Order statistics for nonidentically distributed variables and permanents, Sankhyā Ser. A, 51 (1989), 79-93, URL http://www.jstor.org/stable/25050725.

64. R. B. Bapat, Permanents in probability and statistics, Linear Algebra Appl., 127 (1990), 3-25, URL https://doi.org/10.1016/0024-3795(90)90332-7.

65. S. Hande, A note on order statistics for nondentically distributed variables, Sankhyā Ser. A, 56 (1994), 365-368, URL http://www.jstor.org/stable/25050995.

66. C. H. Papadimitriou and K. Steiglitz, Combinatorial optimization:Algorithms and complexity, Prentice Hall, 1982.

67. O. Faugeras, J. Touboul and B. Cessac, A constructive mean-field analysis of multi-population neural networks with random synaptic weights and stochastic inputs, Front. Comput. Neurosci., 3 (2009), 1, URL https://doi.org/10.3389/neuro.10.001.2009.

68. T. Cabana and J. Touboul, Large deviations, dynamics and phase transitions in large stochastic and disordered neural networks, J. Stat. Phys., 153 (2013), 211-269, URL https://doi.org/10.1007/s10955-013-0818-5.

69. K. Balasubramanian, Combinatorics and diagonals of matrices, PhD thesis, Indian Statistical Institute, 1980, URL http://library.isical.ac.in:8080/jspui/handle/10263/3603.

70. E. Bax and J. Franklin, A finite-difference sieve to compute the permanent, Technical Report CalTech-CS-TR-96-04, 1996.

71. E. Bax, Finite-difference algorithms for counting problems, PhD thesis, California Institute of Technology, 1998, URL https://thesis.library.caltech.edu/223/.

72. D. G. Glynn, The permanent of a square matrix, Eur. J. Combin., 31 (2010), 1887-1891, URL https://doi.org/10.1016/j.ejc.2010.01.010.

73. H. Sompolinsky and I. Kanter, Temporal association in asymmetric neural networks, Phys. Rev. Lett., 57 (1986), 2861-2864, URL http://dx.doi.org/10.1103/PhysRevLett.57.2861.

74. D. Kleinfeld, Sequential state generation by model neural networks, Proc. Natl. Acad. Sci. U.S.A., 83 (1986), 9469-9473, URL http://dx.doi.org/10.1073/pnas.83.24.9469.

75. S. Dehaene, J. P. Changeux and J. P. Nadal, Neural networks that learn temporal sequences by selection, Proc. Natl. Acad. Sci. U.S.A., 84 (1987), 2727-2731, URL https://doi.org/10.1073/pnas.84.9.2727.

76. J. Buhmann and K. Schulten, Noise-driven temporal association in neural networks, Europhys Lett., 4 (1987), 1205, URL http://dx.doi.org/10.1209/0295-5075/4/10/021.

77. J. DeFelipe, P. Marco, I. Busturia, et al., Estimation of the number of synapses in the cerebral cortex:Methodological considerations, Cereb. Cortex, 9 (1999), 722, URL http://dx.doi.org/10.1093/cercor/9.7.722.

78. L. Personnaz, I. Guyon and G. Dreyfus, Collective computational properties of neural networks:New learning mechanisms, Phys. Rev. A, 34 (1986), 4217-4228, URL http://dx.doi.org/10.1103/PhysRevA.34.4217.

79. M. Helias, T. Tetzlaff and M. Diesmann, The correlation structure of local neuronal networks intrinsically results from recurrent dynamics, PLoS Comput. Biol., 10 (2014), e1003428, URL https://doi.org/10.1371/journal.pcbi.1003428.

80. G. Dumont, A. Payeur and A. Longtin, A stochastic-field description of finite-size spiking neural networks, PLoS Comput. Biol., 13 (2017), e1005691, URL https://doi.org/10.1371/journal.pcbi.1005691.

81. M. A. Buice and C. C. Chow, Generalized activity equations for spiking neural network dynamics, Front. Comput. Neurosci., 7 (2013), 162, URL https://doi.org/10.3389/fncom.2013.00162.

82. V. Klinshov and I. Franović, Mean-field dynamics of a random neural network with noise, Phys. Rev. E, 92 (2015), 062813, URL https://doi.org/10.1103/PhysRevE.92.062813.

83. S. Hwang, V. Folli, E. Lanza, et al., On the number of limit cycles in asymmetric neural networks, J. Stat. Mech.:Theory Exp., 2019 (2019), 053402, URL https://doi.org/10.1088/1742-5468/ab11e3.

84. O. Faugeras and J. MacLaurin, Asymptotic description of neural networks with correlated synaptic weights, Entropy, 17 (2015), 4701-4743, URL https://doi.org/10.3390/e17074701.

85. D. Martí, N. Brunel and S. Ostojic, Correlations between synapses in pairs of neurons slow down dynamics in randomly connected neural networks, Phys. Rev. E, 97 (2018), 062314, URL https://doi.org/10.1103/PhysRevE.97.062314.

86. S. Panzeri, S. R. Schultz, A. Treves, et al., Correlations and the encoding of information in the nervous system, Proc. Biol. Sci., 266 (1999), 1001-1012, URL https://doi.org/10.1098/rspb.1999.0736.

87. S. Panzeri, J. H. Macke, J. Gross, et al., Neural population coding:Combining insights from microscopic and mass signals, Trends Cogn. Sci., 19 (2015), 162-172, URL https://doi.org/10.1016/j.tics.2015.01.002.

88. Y. Zuo, H. Safaai, G. Notaro, et al., Complementary contributions of spike timing and spike rate to perceptual decisions in rat S1 and S2 cortex, Curr. Biol., 25 (2015), 357-363, URL https://doi.org/10.1016/j.cub.2014.11.065.

89. C. A. Runyan, E. Piasini, S. Panzeri, et al., Distinct timescales of population coding across cortex, Nature, 548 (2017), 92-96, URL https://doi.org/10.1038/nature23020.

90. E. Zohary, M. N. Shadlen and W. T. Newsome, Correlated neuronal discharge rate and its implications for psychophysical performance, Nature, 370 (1994), 140-143, URL https://doi.org/10.1038/370140a0.

91. W. Singer, Neuronal synchrony:A versatile code for the definition of relations?, Neuron, 24 (1999), 49-65, URL https://doi.org/10.1016/S0896-6273(00)80821-1.

92. A. S. Ecker, P. Berens, G. A. Keliris, et al., Decorrelated neuronal firing in cortical microcircuits, Science, 327 (2010), 584-587, URL https://doi.org/10.1126/science.1179867.

93. M. R. Cohen and A. Kohn, Measuring and interpreting neuronal correlations, Nat. Neurosci., 14 (2011), 811-819, URL https://doi.org/10.1038/nn.2842.

94. R. Brette, Computing with neural synchrony, PLoS Comput. Biol., 8 (2012), e1002561, URL https://doi.org/10.1371/journal.pcbi.1002561.

95. R. Moreno-Bote, J. Beck, I. Kanitscheider, et al., Information-limiting correlations, Nat. Neurosci., 17 (2014), 1410-1417, URL https://doi.org/10.1038/nn.3807.

96. W.-C. A. Lee, V. Bonin, M. Reed, et al., Anatomy and function of an excitatory network in the visual cortex, Nature, 532 (2016), 370-374, URL https://doi.org/10.1038/nature17192.

97. J. L. R. Rubenstein and M. M. Merzenich, Model of autism:Increased ratio of excitation/inhibition in key neural systems, Genes Brain Behav., 2 (2003), 255-267, URL https://doi.org/10.1034/j.1601-183X.2003.00037.x.

98. H. Y. Zoghbi, Postnatal neurodevelopmental disorders:Meeting at the synapse?, Science, 302 (2003), 826-830, URL https://doi.org/10.1126/science.1089071.

99. M. Sahin and M. Sur, Genes, circuits, and precision therapies for autism and related neurodevelopmental disorders, Science, 350 (2015), aab3897, URL https://doi.org/10.1126/science.aab3897.

100. J. P. K. Ip, N. Mellios and M. Sur, Rett syndrome:Insights into genetic, molecular and circuit mechanisms, Nat. Rev. Neurosci., 19 (2018), 368-382, URL https://doi.org/10.1038/s41583-018-0006-3.

Download full text in PDF

Export Citation

Article outline

Show full outline