Www-gmm.insa-toulouse.fr

Mean Field Theory for Random Recurrent Spiking Neural Networks
Bruno Cessac†, Olivier Mazet‡, Manuel Samuelides and H´edi Soula † Institut non lin´eaire de Nice, University of Nice Sofia Antipolis, France ‡ Camille Jordan Math. Institute, Lyon, France, Applied Mathematics Department, SUPAERO, Toulouse, France, Artificial Life, Prisma, INSA, Lyon, France Abstract—Recurrent spiking neural networks can
law. These models are called ”Random Recurrent Neu- provide biologically inspired model of robot controller.
ral Networks”(RRNN). In that case, the parameters of We study here the dynamics of large size randomly interest are the order parameters i.e. the statistical pa- connected networks thanks to ”mean field theory”.
rameters. Then the dynamics is amenable because one Mean field theory allows to compute their dynamics can approach it by ”Mean-Field Equations” (MFE) as under the assumption that the dynamics of individ- in Statistical Physics. MFE were introduced for neu- ual neuronsare stochastically independent. We restrict ral networks by Amari [1] and Crisanti and Sompolin- ourselves to the simple case of homogeneous centered sky [12]. We extended their results [4] and used a new gaussian independent synaptic weights. First a theo- approach to prove it in a rigorous way [10]. This ap- retical study allows to derive the mean-field dynamics proach is the ”Large deviation Principle” (LDP)
using a large deviation approach. This dynamics is and comes from the rigorous statistical mechanics [2].
characterized in function of an order parameter which We developped it for analog neuron model. We show is the normalized variance of the coupling. Then vari- here how it can be extended to spiking neural net- ous applications are reviewed which show the applica- Keywords : Mean field theory, recurrent neural
2. Random Recurrent Neural Networks
networks, dynamical systems, spiking neurons.
2.1. The neuron free dynamics
1. Introduction
We consider here discrete time dynamics with finite horizon. The state of an individual neuron i at time t Recurrent neural networks were introduced to im- is described by the membrane potential u prove biological plausibility of artificial neural net- commodity, we shift it by the neuron firing thresh- works as perceptrons since they display internal dy- old θ. So the trajectory of the potential of a single namics. They are useful to implement associative re- neuron is a vector of F = R{0,1,.,T }. First let us call. The first models were endowed with symmetric consider the free dynamics of a neuron. We introduce connexion weights which induced relaxation dynamics and equilibrium states as in [8]. Asymmetric connex- i)(t))t∈{1,.,T } which is a sequence of i.i.d. centered Gaussian variables of variance σ2. This sequence is ion weights were further introduced which enable to called the synaptic noise of neuron i and stands for observe complex dynamics and chaotic attractors. The all the defects of the model; σ is an order parameter role of chaos in cognitive functions was first discussed which is small. We shall consider three types of neu- by W.Freeman and C.Skarda in seminal papers as [11].
ron: binary formal neuron (BF), analog formal neuron The practical importance of such dynamics is due to (AF) and integrate and fire neuron (IF). For BF and the use of on-line hebbian learning to store dynamical AF neuron, the free dynamics is given by the following patterns. More recent advances along that direction are presented in the present conference [7].
The nature of the dynamics depends on the connex- i(t + 1) = wi(t + 1) − θ ion weights. When considering large size neural net- For IF neuron, the free dynamics is given by works, it is impossible to study the dynamics in func- tion of the detailed parameters. One may consider that i(t + 1) = ϕ[ui(t) + θ)] + wi(t + 1) − θ the connexion weights share few values, yet, the effect where γ ∈]0, 1[ is the leak and where ϕ is defined by of the variablility cannot be studied by this approach.
We consider here large random models where the con- nexion weights form a random sample of a probability ϑ is the reset potential and ϑ < 0 < θ. Let P be the Theorem 2 Let QN ∈ P(FN ) be the probability law
distribution of the state trajectory of the neuron under of the network potential trajectory for RRNN. QN is the free dynamics. For a given initial distribution m0, absolutely continuous with respect to the law P ⊗N of it is possible to explicit P for BF and AF neurons: the free dynamics and dQN (u) = exp N Γ(µ the functional Γ is defined on P(F) by P = m0 × N (−θ, σ2)⊗T 2.2. The synaptic potential of RRNN
t+1(η)ξ(t) − 1 dgµ(ξ)}dµ(η) To define the network dynamics, one has to intro- duce the activation variable xi(t) of the neuron at time t. For BF and IF neurons xi(t) = 1 if and only if neu-ron i emits a spike at instant t, otherwise x t+1(η) = η(t + 1) + θ For AF neurons xi(t) ∈ [0, 1] represents the mean firing IF model: Φt+1(η) = η(t + 1) + θ − ϕ[η(t) + θ] i(t) is a non-linear function of ui(t) according to xi(t) = f [ui(t)] where f is the transfer The law of the empirical measure in the free model is function of the neuron equal to the Heaviside function just the law of an i.i.d. N -sample of P . An imme- for BF and IF neurons and to the sigmoid function diate consequence of the theorem is that exp N Γ(.) is for AF neuron). Let us note u = (ui(t)) ∈ FN the the density of the law of the empirical measure in the network trajectory. The spikes are used to transmit RRNN model with respect to the law of the empirical information to other neurons through the synapses.
Let us note J = (Jij) the system of synaptic weights.
The synaptic potential of neuron i of a network of Nneurons at time t+1 is a vector in F which is expressed 3. The mean-field equation
3.1. The basis of LDP approach
Our objective is to compute the limit of the random measure µu when the size N of the network goes to For size N RRNN model with gaussian connexion infinity. By the Sanov theorem we know that in the weights, J is a normal random vector with N ( υ , υ2 ) free dynamics model µu satisfies a Large Deviation Principle (LDP) with the cross-entropy I(µ, P ) as a model properties can be extended to a more general good rate function and thus converges exponentially setting where the weights are non gaussian and depend towards P . So the consideraton of Sanov theorem and on the neuron class in a several population model [5] of theorem 2, leads us to the following statement When u is given, vi(., u) is a gaussian vector in F; its law is defined by its mean and its covari- Large deviation principle Under the law QN of
the RRNN model µu satisfies a LDP principle with only of the empirical distribution on F defined by good rate function H defined by H(µ) = I(µ, P )−Γ(µ) ∈ P(F). They are invariant by any Actually, the rigorous proof is quite technical and permutation of the neuron potential.
some additional hypothesis and approximations are For µ ∈ P(F) let us denote by gµ the normal distri- necessary to follow the approach of [2]. The math- bution on RT with moments mµ and cµ: ematical proof for AF RRNN is detailed in [10]. Thekeypoint is that for all RRNN models, it is possible to f [η(t)]dµ(η) explicit the minimum of the rate function.
cµ(s + 1, t + 1) = υ2 f [η(s)]f [η(t)]dµ(η) Proposition 1 The common probability law of the in-
3.2. The mean-field propagation operator
dividual synaptic potential trajectories vi(., u) is the i are iid according to µ. Then, from the central limit theorem, the law of the v of the network potential trajectory u. in the limit of large networks. So if we feed a trajectory 2.3. The network dynamics
with a random synaptic potential distributed accord-ing to gµ, we obtain a new probability distribution on Then the state of neuron i at time t is updated ac- F which is noted L(µ).
cording to a modification of equation (2) for AF andBF models (resp. (3) for IF models) where the noise Definition 1 Let µ a probability law on F such that
wi(t + 1) is replaced by vi(t + 1) + wi(t + 1) for each the law of the first component is m0. Let u, w, v be t. So gaussian vector computations lead to three independent random vectors with the respective laws µ, N (0, σ2IT ), gµ. Then L(µ) is the probability Remark: The mathematical derivations of the pre-
law on F of the random vector ϑ which is defined by vious results from LDP may be found in [10]. Theyare available for continuous test functions. For spik- ing neurons, the transfer function is not continuous, so ϑ(t + 1) = v(t + 1) + w(t + 1) − θ we have to use a regular approximation of f to apply for the formal neuron models (BF and AF), and by the previous theorems. Though this approximationcannnot be uniform, it is sufficient for the applications.
ϑ(0) = u(0)ϑ(t + 1) = ϕ[u(t) + θ)] + v(t + 1) + w(t + 1) − θ 4. Applications to the dynamical regime of
for the IF neuron model.The operator L on P(F) asdefined above is called the mean-field propagation op- The mean-field equations are used to predict the spontaneous dynamics of RRNN and to implement learning process on the ”edge of chaos”.
Proposition 3 The density of L(µ) over P is
4.1. BF RRNN
For formal neurons, it is clear from (8) that L(µ) is t+1(η)ξ(t + 1) gaussian. Moreover in the case of BF RRNN, the law of L(µ)(t + 1) depends only on xµ(t) = f [η(t)]dµ(η) It is clear from the construction of L that LT (µ) = which is the mean firing rate at time t. Thus, if we set µ0 is a fixed point of L which depends only on the distribution m0 of the inital state. From the previousproposition, we get L(µ)(t + 1) = F Theorem 4 We have I(µ0, P ) = Γ(µ0) and so
υ2xµ(t) + σ2 So it is possible to get from the fixed point of that Provided that µ0 is the only minimum of H, this last recurrence equation a bifurcation map. Three regimes theorem shows that the random sequence (µu)N con- appear: a ”dead one” where there is no firing, an inter- verges exponentially in law to µ0 when N → ∞ mediate one with a stable firing rate and a ”saturated”one with a firing rate equal to 1. The dead regime 3.3. The main results of MFE theory
and the saturated regime are absolute if σ = 0. They The independence of the (ui) has been used to build tend to disappear when the variability of the connex- the mean-field propagation operator but it cannot be ion weights is increasing. Note that this approach is checked exactly since the neuron states are correlated.
supposing the commutation of time limit and size limit The LDP principle allows to prove rigorously the prop- since the mean-field theory was justified for finite-time agation of chaos property. It amounts to the asymp- horizon. Simulations with N = 100 are in a complete totic independence of any finite set of individual tra- agreement with the theoretical predictions and the sta- tionary regime is reached within few time iterations.
Propagation of chaos property Let h1,.hn be
n continuous bounded test functions defined on F, we 4.2. AF RRNN
The results of the theory have been widely extended in [10]. First the hypothesis of gaussian connection E[h1(u1).hn(un)] → hi(η)dµ0(η) can be dropped if it is replaced by an hypothesis of sub-gaussian tails for the distribution of the connex- An important consequence of the exponential conver- ion weights. MFE can be written which describes the gence is the almost sure weak convergence. This result evolution of the empirical distribution of the network allows to use MFE for statements that relies on a single activity along time [4]. Thus, the distribution of the individual activities at a given time does not containenough information about the nature of the dynamics.
Theorem 5 Let h be a continuous bounded test func-
It may be stationary while the neuron states are stable tion defined on F, we have when N → ∞ or while the individual neurons describe synchronous or asynchronous trajectory. We are interested in the dynamic regime of the detailed network in the low- h(η)dµ0(η) a.s. noise limit. A relevant quantity for that purpose is the evolution equation of the distance between two trajec- Acknowledgments
tories along time [6],[3]. Two initial states are selectedindependently and the dynamics are similar with the This work has been supported by French Minister same configuration parameters and independent low of Research through ”Computational and Integrative noise. Then a mean-field theory is developped for the Neuroscience” research contract from 2003 to 2005.
joint law of the two trajectories and allow to study theevolution of the mean-quadratic distance between the References
two trajectories in the low-noise limit. A limit equalto 0 is characteristic of a fixed point, a non-zero limit [1] S. Amari, K. Yosida, and K.I. Kakutani. A math- is the signature of the chaos. These informations may ematical foundation for statistical neurodynam- be recovered by the the study of the asymptotic co- ics. Siam J. Appl. Math., 33(1):95–126, 1977.
variance of the MFE for a single network.
[2] G. Ben Arous and A. Guionnet. Large deviations For instance we applied this approach to predict for langevin spin glass dynamics. Probability The- the behaviour of a 2 population model (excitatory/ ory and Related Fields, 102:455–509, 1995.
inhibitory) in [5]. It was studied in detail with the fol-lowing order parameters (the index label the presynap- [3] B. Cessac. Increase in complexity in random neu- tic and the postsynaptic population: g is quantifying ral networks. Journal de Physique I, 5:409–432, the non-linearity of the transfer function of the AF RRNN, d is quantifying the inhibitory or excitatorycharacter of the two populations. d = 0 amounts to the one-population model. Four asymptotic regimes exist: a stable fixed point regime, a stable periodic cation map and route to chaos in discrete time regim, a chaotic stationary regime and a cyclostation- neural networks. Physica D, 74:24–44, 1994.
ary chaotic regim. Simulation results and theoretical M. Samuelides. Mean field theory and synchro-nization in random recurrent neural networks.
4.3. IF RRNN
Neural Processing Letters, 14:115–126, 2001.
Mean-field theory is generally considered as a good approximation for IF RRNN [9]. Actually, the detailed model and the mean-field dynamics exhibit a transi-tion from a zero mean-firing rate to a non-zero mean- [7] E.Dauc´e, H.Soula, and G.Beslon. Learning meth- firing rate when the standard deviation of the connex- ods for dynamic neural networks. In NOLTA Con- ion weight is increasing. When the lack is growing to one, the critical standard deviation which induces a [8] J. J. Hopfield. Neural networks and physical sys- non-zero firing rate is growing. Still, there is a good tems with emergent collective computational abil- quantitative agreement between simulations and the- ities. Proc. Nat. Acad. Sci., 79:2554–2558, 1982.
oretic MFE predictions. Yet, one is obliged to com-pute MFE predictions to use Monte-Carlo algorithms [9] H.Soula, G.Beslon, and O.Mazet. Spontaneous to simulate MFE dynamics. Another way of using the dynamics of random recurrent spiking neural net- mean-field assumption to predict the firing-rate value works. Neural Computation, accepted for publica- is to model the synaptic potential as a random sum of independent random variables using Wald identities.
It is developped in [9] and allows to predict the the- [10] O.Moynot and M.Samuelides. Large deviations oretical mean-firing rate without using Monte-Carlo and mean-field theory for asymetric random re- current neural networks. Probability Theory andRelated Fields, 123(1):41–75, 2002.
5. Perspectives
[11] C.A. Skarda and W.J. Freeman. Chaos and the new science of the brain, volume 1-2, pages 275– A general framework was proposed to study Ran- 285. In ”Concepts in Neuroscience”, World Sci- dom Recurrent Neural Networks using mean-field the- ory. It allows to predict simulation results for largesize random recurrent neural network dynamics. IF [12] H. Sompolinsky, A. Crisanti, and H.J. Sommers.
RRNN models deserve further investigation. Notably, Chaos in random neural networks. Phys. Rev. the model of random connections is far from biological models and random connectivity has to be tested.

Source: http://www-gmm.insa-toulouse.fr/~omazet/Recherche/ARTICLES/Nolta2005vf.pdf