Episode #41 in #TheoreticalNeurosciencePodcast: On functional effects of neuronal heterogeneity – with David Dahmen
theoreticalneuroscience.no/thn41
While network modelers often assume all neurons to be the same, biology says different. Can this hetereogeneity be a good thing?
Do RL agents really need to replan from scratch for every new goal? Why do rodents "preplay" goals they haven't pursued?
We think these questions share a candidate answer.
Zooming out: Compared to fixed-point methods, techniques for characterizing recurrent computations based on transient dynamics are nascent—and potentially vital! Indeed energy landscapes carry structure invisible to equilibrium analysis and neural circuits may exploit this structure for computation.
Transient retrieval also raises a new question: when to read out the memory. Above capacity, optimal readout time is non-monotonic in init. overlap, & peaks at intermediate values, where trajectories can access slow energy-landscape features, making it worthwhile to wait to exploit this structure.
The same effect appears below capacity for initial conditions outside any basin. To summarize these effects, I introduce "transient-recovery curves," a Pareto-style family showing maximum retrieval quality vs. initial overlap, with graceful, noncatastrophic change across the critical capacity.
This phenomenon must have origins in energy-landscape structure. Specifically, slow regions persist near stored patterns as lingering traces of the stable basins that existed below capacity. Trajectories pass through these regions and retrieve the stored pattern before being driven away.
The key phenomenon is that even above capacity, where stable fixed points no longer exist, memories can be transiently recalled, often with high accuracy!
Thus, networks retain substantial memory function beyond their critical capacities, provided one accepts transient rather than persistent recall.
BUT: this interpretation follows from equilibrium analyses, probing only stable fixed points. Studying instead the transient dynamics, a different picture emerges.
To this end, I derive & numerically solve a dynamical mean-field theory (DMFT) for the full, N→∞ dynamics of associative memory models.
Hopfield models (PNAS 1982), and their "dense" generalizations (Krotov & Hopfield, NeurIPS 2016), have a critical capacity above which stable memory states vanish, an effect called the blackout catastrophe (AGS, PRL 1985), typically interpreted as rendering the system unusable as a memory device.
Now in PRE: "Transient dynamics of associative memory models." I argue that the "blackout catastrophe" is not catastrophic when viewed from an out-of-equilibrium, dynamical perspective.
Journal: journals.aps.org/pre/abstract/10.1103/42y2-bsh1
PDF: dclark.io/media/clark-...