## Www-gmm.insa-toulouse.fr

**Mean Field Theory for Random Recurrent Spiking Neural Networks**
Bruno Cessac

*†*, Olivier Mazet

*‡*, Manuel Samuelides and H´edi Soula

*† *Institut non lin´eaire de Nice, University of Nice Soﬁa Antipolis, France

*‡ *Camille Jordan Math. Institute, Lyon, France,
Applied Mathematics Department, SUPAERO, Toulouse, France,
Artiﬁcial Life, Prisma, INSA, Lyon, France

**Abstract**—Recurrent spiking neural networks can

law. These models are called

*”Random Recurrent Neu-*
provide biologically inspired model of robot controller.

*ral Networks”*(RRNN). In that case, the parameters of
We study here the dynamics of large size randomly
interest are the

*order parameters *i.e. the statistical pa-
connected networks thanks to ”mean ﬁeld theory”.

rameters. Then the dynamics is amenable because one
Mean ﬁeld theory allows to compute their dynamics
can approach it by

*”Mean-Field Equations” *(MFE) as
under the assumption that the dynamics of individ-
in Statistical Physics. MFE were introduced for neu-
ual neuronsare stochastically independent. We restrict
ral networks by Amari [1] and Crisanti and Sompolin-
ourselves to the simple case of homogeneous centered
sky [12]. We extended their results [4] and used a new
gaussian independent synaptic weights. First a theo-
approach to prove it in a rigorous way [10]. This ap-
retical study allows to derive the mean-ﬁeld dynamics
proach is the

**”Large deviation Principle” **(LDP)

using a large deviation approach. This dynamics is
and comes from the rigorous statistical mechanics [2].

characterized in function of an order parameter which
We developped it for analog neuron model. We show
is the normalized variance of the coupling. Then vari-
here how it can be extended to spiking neural net-
ous applications are reviewed which show the applica-

**Keywords : **Mean ﬁeld theory, recurrent neural

**2. Random Recurrent Neural Networks**
networks, dynamical systems, spiking neurons.

**2.1. The neuron free dynamics**
**1. Introduction**
We consider here discrete time dynamics with ﬁnite
horizon. The state of an individual neuron

*i *at time

*t*
Recurrent neural networks were introduced to im-
is described by the

*membrane potential u*
prove biological plausibility of artiﬁcial neural net-
commodity, we shift it by the neuron ﬁring thresh-
works as perceptrons since they display internal dy-
old

*θ*. So the trajectory of the potential of a single
namics. They are useful to implement associative re-
neuron is a vector of

*F *=

*R{*0

*,*1

*,.,T }*. First let us
call. The ﬁrst models were endowed with symmetric
consider the free dynamics of a neuron. We introduce
connexion weights which induced relaxation dynamics
and equilibrium states as in [8]. Asymmetric connex-

*i*)(

*t*))

*t∈{*1

*,.,T } *which is a sequence of i.i.d. centered
Gaussian variables of variance

*σ*2. This sequence is
ion weights were further introduced which enable to
called the

*synaptic noise *of neuron

*i *and stands for
observe complex dynamics and chaotic attractors. The
all the defects of the model;

*σ *is an order parameter
role of chaos in cognitive functions was ﬁrst discussed
which is small. We shall consider three types of neu-
by W.Freeman and C.Skarda in seminal papers as [11].

ron: binary formal neuron (BF), analog formal neuron
The practical importance of such dynamics is due to
(AF) and integrate and ﬁre neuron (IF). For BF and
the use of on-line hebbian learning to store dynamical
AF neuron, the free dynamics is given by the following
patterns. More recent advances along that direction
are presented in the present conference [7].

The nature of the dynamics depends on the connex-

*i*(

*t *+ 1) =

*wi*(

*t *+ 1)

*− θ*
ion weights. When considering large size neural net-
For IF neuron, the free dynamics is given by
works, it is impossible to study the dynamics in func-
tion of the detailed parameters. One may consider that

*i*(

*t *+ 1) =

*ϕ*[

*ui*(

*t*) +

*θ*)] +

*wi*(

*t *+ 1)

*− θ*
the connexion weights share few values, yet, the eﬀect
where

*γ ∈*]0

*, *1[ is the

*leak *and where

*ϕ *is deﬁned by
of the variablility cannot be studied by this approach.

We consider here large random models where the con-
nexion weights form a random sample of a probability

*ϑ *is the reset potential and

*ϑ < *0

*< θ*. Let

*P *be the

**Theorem 2 ***Let QN ∈ P*(

*FN *)

*be the probability law*
distribution of the state trajectory of the neuron under

*of the network potential trajectory for RRNN. QN is*
the free dynamics. For a given initial distribution

*m*0,

*absolutely continuous with respect to the law P ⊗N of*
it is possible to explicit

*P *for BF and AF neurons:

*the free dynamics and dQN *(

*u*) = exp

*N *Γ(

*µ*
*the functional *Γ

*is deﬁned on P*(

*F*)

*by*
*P *=

*m*0

*× N *(

*−θ, σ*2)

*⊗T*
**2.2. The synaptic potential of RRNN**
*t*+1(

*η*)

*ξ*(

*t*)

*− *1

*dgµ*(

*ξ*)

*}dµ*(

*η*)
To deﬁne the network dynamics, one has to intro-
duce the activation variable

*xi*(

*t*) of the neuron at time

*t*. For BF and IF neurons

*xi*(

*t*) = 1 if and only if neu-ron

*i *emits a spike at instant

*t*, otherwise

*x*
*t*+1(

*η*) =

*η*(

*t *+ 1) +

*θ*
For AF neurons

*xi*(

*t*)

*∈ *[0

*, *1] represents the mean ﬁring

*IF model: *Φ

*t*+1(

*η*) =

*η*(

*t *+ 1) +

*θ − ϕ*[

*η*(

*t*) +

*θ*]

*i*(

*t*) is a non-linear function of

*ui*(

*t*)
according to

*xi*(

*t*) =

*f *[

*ui*(

*t*)] where

*f *is the

*transfer*
The law of the empirical measure in the free model is

*function *of the neuron equal to the Heaviside function
just the law of an i.i.d.

*N *-sample of

*P *. An imme-
for BF and IF neurons and to the sigmoid function
diate consequence of the theorem is that exp

*N *Γ(

*.*) is
for AF neuron). Let us note

*u *= (

*ui*(

*t*))

*∈ FN *the
the density of the law of the empirical measure in the
network trajectory. The spikes are used to transmit
RRNN model with respect to the law of the empirical
information to other neurons through the synapses.

Let us note

*J *= (

*Jij*) the system of

*synaptic weights*.

The

*synaptic potential *of neuron

*i *of a network of

*N*neurons at time

*t*+1 is a vector in

*F *which is expressed

**3. The mean-ﬁeld equation**
**3.1. The basis of LDP approach**
Our objective is to compute the limit of the random
measure

*µu *when the size

*N *of the network goes to
For size

*N *RRNN model with gaussian connexion
inﬁnity. By the Sanov theorem we know that in the
weights,

*J *is a normal random vector with

*N *(

*υ , υ*2 )
free dynamics model

*µu *satisﬁes a Large Deviation
Principle (LDP) with the cross-entropy

*I*(

*µ, P *) as a
model properties can be extended to a more general
good rate function and thus converges exponentially
setting where the weights are non gaussian and depend
towards

*P *. So the consideraton of Sanov theorem and
on the neuron class in a several population model [5]
of theorem 2, leads us to the following statement
When

*u *is given,

*vi*(

*., u*) is a gaussian vector in

*F*; its law is deﬁned by its mean and its covari-

**Large deviation principle ***Under the law QN of*
*the RRNN model µu satisﬁes a LDP principle with*
only of the

*empirical distribution *on

*F *deﬁned by

*good rate function H deﬁned by H*(

*µ*) =

*I*(

*µ, P *)

*−*Γ(

*µ*)

*∈ P*(

*F*). They are invariant by any
Actually, the rigorous proof is quite technical and
permutation of the neuron potential.

some additional hypothesis and approximations are
For

*µ ∈ P*(

*F*) let us denote by

*gµ *the normal distri-
necessary to follow the approach of [2]. The math-
bution on

*RT *with moments

*mµ *and

*cµ*:
ematical proof for AF RRNN is detailed in [10]. Thekeypoint is that for all RRNN models, it is possible to

*f *[

*η*(

*t*)]

*dµ*(

*η*)
explicit the minimum of the rate function.

*cµ*(

*s *+ 1

*, t *+ 1) =

*υ*2

*f *[

*η*(

*s*)]

*f *[

*η*(

*t*)]

*dµ*(

*η*)

**Proposition 1 ***The common probability law of the in-*
**3.2. The mean-ﬁeld propagation operator**
*dividual synaptic potential trajectories vi*(

*., u*)

*is the*
*i *are iid according to

*µ*. Then,
from the central limit theorem, the law of the

*v*
*of the network potential trajectory u.*
in the limit of large networks. So if we feed a trajectory

**2.3. The network dynamics**
with a random synaptic potential distributed accord-ing to

*gµ*, we obtain a new probability distribution on
Then the state of neuron

*i *at time

*t *is updated ac-

*F *which is noted

*L*(

*µ*).

cording to a modiﬁcation of equation (2) for AF andBF models (resp. (3) for IF models) where the noise

**Deﬁnition 1 ***Let µ a probability law on F such that*
*wi*(

*t *+ 1) is replaced by

*vi*(

*t *+ 1) +

*wi*(

*t *+ 1) for each

*the law of the ﬁrst component is m*0

*. Let u, w, v be*
*t*. So gaussian vector computations lead to

*three independent random vectors with the respective*
*laws µ, N *(0

*, σ*2

*IT *)

*, gµ. Then L*(

*µ*)

*is the probability*
**Remark: **The mathematical derivations of the pre-

*law on F of the random vector ϑ which is deﬁned by*
vious results from LDP may be found in [10]. Theyare available for continuous test functions. For spik-
ing neurons, the transfer function is not continuous, so

*ϑ*(

*t *+ 1) =

*v*(

*t *+ 1) +

*w*(

*t *+ 1)

*− θ*
we have to use a regular approximation of

*f *to apply

*for the formal neuron models (BF and AF), and by*
the previous theorems. Though this approximationcannnot be uniform, it is suﬃcient for the applications.

*ϑ*(0) =

*u*(0)

*ϑ*(

*t *+ 1) =

*ϕ*[

*u*(

*t*) +

*θ*)] +

*v*(

*t *+ 1) +

*w*(

*t *+ 1)

*− θ*
**4. Applications to the dynamical regime of**
*for the IF neuron model.The operator L on P*(

*F*)

*asdeﬁned above is called the mean-ﬁeld propagation op-*
The mean-ﬁeld equations are used to predict the
spontaneous dynamics of RRNN and to implement
learning process on the ”edge of chaos”.

**Proposition 3 ***The density of L*(

*µ*)

*over P is*
**4.1. BF RRNN**
For formal neurons, it is clear from (8) that

*L*(

*µ*) is

*t*+1(

*η*)

*ξ*(

*t *+ 1)
gaussian. Moreover in the case of BF RRNN, the law
of

*L*(

*µ*)(

*t *+ 1) depends only on

*xµ*(

*t*) =

*f *[

*η*(

*t*)]

*dµ*(

*η*)
It is clear from the construction of

*L *that

*LT *(

*µ*) =
which is the mean ﬁring rate at time

*t*. Thus, if we set

*µ*0 is a ﬁxed point of

*L *which depends only on the
distribution

*m*0 of the inital state. From the previousproposition, we get

*L*(

*µ*)(

*t *+ 1) =

*F*
**Theorem 4 ***We have I*(

*µ*0

*, P *) = Γ(

*µ*0)

*and so*
*υ*2

*xµ*(

*t*) +

*σ*2
So it is possible to get from the ﬁxed point of that
Provided that

*µ*0 is the

*only *minimum of

*H*, this last
recurrence equation a bifurcation map. Three regimes
theorem shows that the random sequence (

*µu*)

*N *con-
appear: a ”dead one” where there is no ﬁring, an inter-
verges exponentially in law to

*µ*0 when

*N → ∞*
mediate one with a stable ﬁring rate and a ”saturated”one with a ﬁring rate equal to 1. The dead regime

**3.3. The main results of MFE theory**
and the saturated regime are absolute if

*σ *= 0. They
The independence of the (

*ui*) has been used to build
tend to disappear when the variability of the connex-
the mean-ﬁeld propagation operator but it cannot be
ion weights is increasing. Note that this approach is
checked exactly since the neuron states are correlated.

supposing the commutation of time limit and size limit
The LDP principle allows to prove rigorously the

*prop-*
since the mean-ﬁeld theory was justiﬁed for ﬁnite-time

*agation of chaos *property. It amounts to the asymp-
horizon. Simulations with

*N *= 100 are in a complete
totic independence of any

*ﬁnite *set of individual tra-
agreement with the theoretical predictions and the sta-
tionary regime is reached within few time iterations.

**Propagation of chaos property ***Let h*1

*,.hn be*
*n continuous bounded test functions deﬁned on F, we*
**4.2. AF RRNN**
The results of the theory have been widely extended
in [10]. First the hypothesis of gaussian connection
E[

*h*1(

*u*1)

*.hn*(

*un*)]

*→*
*hi*(

*η*)

*dµ*0(

*η*)
can be dropped if it is replaced by an hypothesis of
sub-gaussian tails for the distribution of the connex-
An important consequence of the exponential conver-
ion weights. MFE can be written which describes the
gence is the almost sure weak convergence. This result
evolution of the empirical distribution of the network
allows to use MFE for statements that relies on a single
activity along time [4]. Thus, the distribution of the
individual activities at a given time does not containenough information about the nature of the dynamics.

**Theorem 5 ***Let h be a continuous bounded test func-*
It may be stationary while the neuron states are stable

*tion deﬁned on F, we have when N → ∞*
or while the individual neurons describe synchronous
or asynchronous trajectory. We are interested in the
dynamic regime of the detailed network in the low-

*h*(

*η*)

*dµ*0(

*η*)

*a.s.*
noise limit. A relevant quantity for that purpose is the
evolution equation of the distance between two trajec-

**Acknowledgments**
tories along time [6],[3]. Two initial states are selectedindependently and the dynamics are similar with the
This work has been supported by French Minister
same conﬁguration parameters and independent low
of Research through ”Computational and Integrative
noise. Then a mean-ﬁeld theory is developped for the
Neuroscience” research contract from 2003 to 2005.

joint law of the two trajectories and allow to study theevolution of the mean-quadratic distance between the

**References**
two trajectories in the low-noise limit. A limit equalto 0 is characteristic of a ﬁxed point, a non-zero limit
[1] S. Amari, K. Yosida, and K.I. Kakutani. A math-
is the signature of the chaos. These informations may
ematical foundation for statistical neurodynam-
be recovered by the the study of the asymptotic co-
ics.

*Siam J. Appl. Math.*, 33(1):95–126, 1977.

variance of the MFE for a single network.

[2] G. Ben Arous and A. Guionnet. Large deviations
For instance we applied this approach to predict
for langevin spin glass dynamics.

*Probability The-*
the behaviour of a 2 population model (excitatory/

*ory and Related Fields*, 102:455–509, 1995.

inhibitory) in [5]. It was studied in detail with the fol-lowing order parameters (the index label the presynap-
[3] B. Cessac. Increase in complexity in random neu-
tic and the postsynaptic population:

*g *is quantifying
ral networks.

*Journal de Physique I*, 5:409–432,
the non-linearity of the transfer function of the AF
RRNN,

*d *is quantifying the inhibitory or excitatorycharacter of the two populations.

*d *= 0 amounts to
the one-population model. Four asymptotic regimes
exist: a stable ﬁxed point regime, a stable periodic
cation map and route to chaos in discrete time
regim, a chaotic stationary regime and a cyclostation-
neural networks.

*Physica D*, 74:24–44, 1994.

ary chaotic regim. Simulation results and theoretical
M. Samuelides. Mean ﬁeld theory and synchro-nization in random recurrent neural networks.

**4.3. IF RRNN**
*Neural Processing Letters*, 14:115–126, 2001.

Mean-ﬁeld theory is generally considered as a good
approximation for IF RRNN [9]. Actually, the detailed
model and the mean-ﬁeld dynamics exhibit a transi-tion from a zero mean-ﬁring rate to a non-zero mean-
[7] E.Dauc´e, H.Soula, and G.Beslon. Learning meth-
ﬁring rate when the standard deviation of the connex-
ods for dynamic neural networks. In

*NOLTA Con-*
ion weight is increasing. When the lack is growing to
one, the critical standard deviation which induces a
[8] J. J. Hopﬁeld. Neural networks and physical sys-
non-zero ﬁring rate is growing. Still, there is a good
tems with emergent collective computational abil-
quantitative agreement between simulations and the-
ities.

*Proc. Nat. Acad. Sci.*, 79:2554–2558, 1982.

oretic MFE predictions. Yet, one is obliged to com-pute MFE predictions to use Monte-Carlo algorithms
[9] H.Soula, G.Beslon, and O.Mazet. Spontaneous
to simulate MFE dynamics. Another way of using the
dynamics of random recurrent spiking neural net-
mean-ﬁeld assumption to predict the ﬁring-rate value
works.

*Neural Computation, accepted for publica-*
is to model the synaptic potential as a random sum of
independent random variables using Wald identities.

It is developped in [9] and allows to predict the the-
[10] O.Moynot and M.Samuelides. Large deviations
oretical mean-ﬁring rate without using Monte-Carlo
and mean-ﬁeld theory for asymetric random re-
current neural networks.

*Probability Theory andRelated Fields*, 123(1):41–75, 2002.

**5. Perspectives**
[11] C.A. Skarda and W.J. Freeman.

*Chaos and the*
*new science of the brain*, volume 1-2, pages 275–
A general framework was proposed to study Ran-
285. In ”Concepts in Neuroscience”, World Sci-
dom Recurrent Neural Networks using mean-ﬁeld the-
ory. It allows to predict simulation results for largesize random recurrent neural network dynamics. IF
[12] H. Sompolinsky, A. Crisanti, and H.J. Sommers.

RRNN models deserve further investigation. Notably,
Chaos in random neural networks.

*Phys. Rev.*
the model of random connections is far from biological
models and random connectivity has to be tested.

Source: http://www-gmm.insa-toulouse.fr/~omazet/Recherche/ARTICLES/Nolta2005vf.pdf

EYE INJURIES IN BOXING Jean-Louis Llouquet Introduction The era of blindness as a result of boxing is past. However boxing doctors have become more aware of ocular damage: retinal detachment being the most frequently observed serious injury in professional boxing. Ocular injuries as a result of boxing mainly affects professional boxers The eye is relatively well pr

Clearing the Fog in Nursing Homes - NYTimes.com FEBRUARY 15, 2011, 11:10 AM Clearing the Fog in Nursing Homes By PAULA SPAN The woman, who was in her 90s, had lived for several years at the Ecumen Sunrise nursing home in Two Harbors, Minn., where the staff had grown accustomed to her grimaces and wordless cries. She took a potent cocktail of three psychotropic drugs: Ativan for anxiet