Article Figures & Data
Figures
- Figure 1.
Associative memory storage in a recurrent network of inhibitory and excitatory neurons in the presence of errors and noise. A, Error propagation through the network. Inhibitory neurons (red circles) and excitatory neurons (blue triangles) form an all-to-all potentially (structurally) connected network. Red and blue arrows represent actual (functional) connections. Spiking errors (errors contained in
), synaptic noise (
), and intrinsic noise (
) accompany signal transmission (orange lightning signs). Errors in the neurons’ outputs at a given time step become spiking errors in the next time step. B, Fluctuations in PSPs for two associations with target neuron outputs 0 (left) and 1 (right). Large black dots denote PSPs in the absence of errors and noise. Small dots represent PSPs on different trials in the presence of errors and noise. Orange areas to the left of the PSP probability densities (solid lines) represent the probabilities of erroneous spikes (left) and spike failures (right). C, The probability of successful learning by a neuron is a sharply decreasing function of memory load m/N. Solid curves represent the probabilities of successful learning obtained with nonlinear optimization (see Materials and Methods) for neurons receiving N = 200, 400, and 800 homogeneous inputs. The numerical values of βlearn and rin = rout ≡ rlearn are provided in the figure. The values of all other parameters of the model were adapted from Chapeton et al. (2015). At 0.5 success probability, the neuron is said to be loaded to capacity, α. The dashed black line represents the theoretical (critical) capacity, αc, obtained with the replica method in the N → ∞ limit. D, αc as a function of βlearn for different input noise strengths (colored lines). In the case of rin = 0, solution of Equation 1 (blue line) coincides with the solution of the traditional model (Zhang et al., 2019b), which uses a generic robustness parameter (black dots). E, Map of αc for a neuron receiving homogeneous input as a function of rin and rout. F, Same as a function of βlearn and rin = rout ≡ rlearn. The maps in E, F were obtained with the replica method (see Materials and Methods), and the green asterisks correspond to the values of parameters used in C. Dashed isocontours are drawn as a guide to the eye. - Figure 2.
Retrieval of loaded associative memory sequences and the trade-off between capacity and reliability of loaded memories. A, Illustration of memory playout during complete and partial memory retrieval (left). The target memory sequence is shown in black, while the sequences retrieved on different trials are in blue and red. Memory retrieval is incomplete when the retrieved sequence deviates significantly from the target sequence (see text for details). Radii of blue spheres illustrate the root-mean-square Euclidean distances between the retrieved and target states. The fraction of errors as a function of time step during sequence retrieval (right). Successfully retrieved sequences do not deviate from the loaded sequences by more than a threshold amount (dashed line). The parameters of the associative network are provided in the figure. The values of βlearn and rlearn correspond to the green asterisk from Figure 1. B, The probability of successful memory retrieval (green) and the retrieved fraction of loaded sequence length (red) as a function of βlearn. The postsynaptic noise strength βretr = 30 (dashed line) at every step of memory retrieval and rretr was set to 0 at the first step. C, Map of retrieval probability as a function of βlearn and rlearn. Dashed isocontour is drawn as a guide to the eye. The location of the green asterisk is the same as in Figure 1F. D, The trade-off between memory retrieval probability and α. Individual points correspond to all values of βlearn and rlearn considered in C. Higher errors and noise during learning result in lower α and higher retrieval probability regardless of the noise strength during memory retrieval (different colors). The results shown in A–D were obtained with the nonlinear optimization method (see Materials and Methods). For every parameter setting, the results shown in B–D were averaged over 100 networks and 1000 retrievals of the loaded sequence in each network.
- Figure 3.
Postsynaptic noise during learning is required for optimal retrieval of stored information. A, B, Maps of expected retrieved information per memory playout calculated based on completely retrieved sequences (A) and completely and partially retrieved sequence (B) in bits×N2 as functions of βlearn and rlearn. βretr = 30 at every step of memory retrieval, and rretr was set to 0 at the first step. Dashed isocontours are drawn as guides to the eye. The locations of the green asterisks are the same as in Figure 1F. C, The maximum of retrieved information is achieved when βlearn is greater than zero regardless of the value of βretr. The optimal postsynaptic noise strengths were calculated based on the averages of the results from A, blue line, and B, orange line, over the range of rlearn values from A, B. All results were obtained with the nonlinear optimization method (see Materials and Methods) and averaged over 100 networks and 1000 retrievals of the loaded sequence in each network for every parameter setting.
- Figure 4.
Comparison of structural properties of the model and cortical networks. A, Inhibitory and excitatory connection probabilities reported in 87 studies describing 420 local cortical projections. Each dot represents the result of a single study/projection. B, C, Maps of inhibitory and excitatory connection probabilities as functions of βlearn and rlearn. The results are based on the replica method (see Materials and Methods). Dashed isocontours and arrows illustrate the interquartile ranges of the experimentally observed connection probabilities from A. The red contour outlines a region of parameters that is consistent with all structural and dynamical measurements in cortical networks considered in this study. The locations of the green asterisks are the same as in Figure 1F. D–F, Same for the CV of non-zero inhibitory and excitatory connection weights. A, D were adapted from Zhang et al. (2019b).
- Figure 5.
Comparison of dynamical properties of the model and cortical networks. A, The CV of ISI for spontaneous (not learned) activity as a function of βlearn and rlearn. Dashed isocontour and arrows demarcate a region of CV values that is in general agreement with experimental measurements. B, Same for the cross-correlation coefficient of neuron spike trains. C, Same for the anti-correlation coefficient of inhibitory and excitatory postsynaptic inputs to a neuron. The red contour outlines a region of parameters that is consistent with the considered structural and dynamical measurements. The locations of the green asterisk are the same as in Figure 1F. All results were obtained with the nonlinear optimization method (see Materials and Methods) and averaged over 100 networks and 100 runs for each network for every parameter setting.
- Figure 6.
Comparison of solutions obtained with the perceptron-type learning rule, nonlinear optimization, and replica method. A, Output error probability as a function of the number of learning epochs for the perceptron-type learning rule. The black dashed line indicates the target output error probability. Results for three different cases are shown: a not feasible problem (red line), a feasible problem which was not solved with the perceptron-type learning rule (blue line), and a feasible problem which was solved with the perceptron-type learning rule (green line). The parameters of the associative network are provided in the figure. The values of βlearn and rlearn correspond to the green asterisk from Figure 1F. B, Comparisons of connection weights obtained with the perceptron-type learning rule and nonlinear optimization for the three cases shown in A. Straight lines are the best linear fits. C, Comparisons of memory storage capacity, retrieval, structural, and dynamical properties of networks of N = 200, 400, and 800 neurons obtained with the perceptron-type learning rule (red colors) and nonlinear optimization (blue colors). The memory storage capacity and structural properties calculated with the replica method in the N → ∞ limit are shown in black.
- Figure 7.
Properties of connections in associative networks of heterogeneous neurons. A–C, Connection probability (B) and average non-zero connection weight (C) for inhibitory (red) and excitatory (blue) connections in a network of neurons with distributed spiking error probabilities and homogeneous in all other parameters. The spiking error probabilities of inhibitory and excitatory inputs during learning were randomly drawn from the log-normal distribution shown in A. Unreliable inputs have lower probabilities and weights. The parameters of the associative network are shown in A. The values of βlearn and <rlearn> correspond to the green asterisk from Figure 1F. D–F, Same for a network of neurons with heterogeneous postsynaptic noise strengths. The postsynaptic noise strengths of neurons during learning were randomly drawn from the log-normal distribution shown in D. Noisier neurons receive stronger but fewer inhibitory and excitatory inputs.
- Figure 8.
Increasing the noise strength during learning and decreasing it during memory recall lead to more reliable solutions. A, The associative learning problem for a below capacity load in the absence of noise during learning, βlearn = 0. The solution region (blue) is bounded by hyperplanes corresponding to the individual associations (black lines). The learning phase (red arrows) ends as the connection weight vector enters the solution region. The solution shown in A is unreliable because noise during memory retrieval (red cloud of radius βretr) can move it outside the solution region with high probability. B, Adding noise during learning (green cloud of radius βlearn) transforms the association hyperplanes (gray lines) into hypersurfaces (black lines; Eq. 1), reducing the solution region and forcing the connection weight vector further away from the hyperplanes. This increases solution reliability. C, The continued increase of the noise strength improves reliability as the solution region shrinks to zero. At this noise strength, the memory load is at capacity. A further increase in reliability can be achieved by increasing the noise strength during learning (D) or decreasing it during retrieval (E). In the former case, the memory load must be reduced to match the reduction in capacity.
In this issue
