Abstract
Almost all models of visual memory implicitly assume that errors in mnemonic representations are linearly related to distance in stimulus space. Here we show that neither memory nor perception are appropriately scaled in stimulus space; instead, they are based on a transformed similarity representation that is nonlinearly related to stimulus space. This result calls into question a foundational assumption of extant models of visual working memory. Once psychophysical similarity is taken into account, aspects of memory that have been thought to demonstrate a fixed working memory capacity of around three or four items and to require fundamentally different representations—across different stimuli, tasks and types of memory—can be parsimoniously explained with a unitary signal detection framework. These results have substantial implications for the study of visual memory and lead to a substantial reinterpretation of the relationship between perception, working memory and long-term memory.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
All relevant data for this manuscript are available at https://osf.io/j2h65/?view_only=fdd51dd775a945508c7cbbf25b662692.
Code availability
All relevant analysis code for this manuscript is available at https://osf.io/j2h65/?view_only=fdd51dd775a945508c7cbbf25b662692.
Change history
09 October 2020
An amendment to this paper has been published and can be accessed via a link at the top of the paper.
References
Cowan, N. Metatheory of storage capacity limits. Behav. Brain Sci. 24, 154–176 (2001).
Luck, S. J. & Vogel, E. K. Visual working memory capacity: from psychophysics and neurobiology to individual differences. Trends Cogn. Sci. 17, 391–400 (2013).
Baddeley, A. Working memory: looking back and looking forward. Nat. Rev. Neurosci. 4, 829–839 (2003).
Ma, W. J., Husain, M. & Bays, P. M.Changing concepts of working memory. Nat. Neurosci. 17, 347–356 (2014).
Fukuda, K., Vogel, E., Mayr, U. & Awh, E. Quantity, not quality: the relationship between fluid intelligence and working memory capacity. Psychonomic Bull. Rev. 17, 673–679 (2010).
Alloway, T. P. & Alloway, R. G. Investigating the predictive roles of working memory and IQ in academic attainment. J. Exp. Child Psychol. 106, 20–29 (2010).
Zhang, W. & Luck, S. J.Discrete fixed-resolution representations in visual working memory. Nature 453, 233–235 (2008).
Van den Berg, R., Shin, H., Chou, W. C., George, R. & Ma, W. J. Variability in encoding precision accounts for visual short-term memory limitations. Proc. Natl Acad. Sci. USA 109, 8780–8785 (2012).
Bays, P. M. Noise in neural populations accounts for errors in working memory. J. Neurosci. 34, 3632–3645 (2014).
Bays, P. M. Spikes not slots: noise in neural populations limits working memory. Trends Cogn. Sci. 19, 431–438 (2015).
Serences, J. T. Neural mechanisms of information storage in visual short-term memory. Vis. Res. 128, 53–67 (2016).
Bae, G. Y., Olkkonen, M., Allred, S. R., Wilson, C. & Flombaum, J. I. Stimulus-specific variability in color working memory with delayed estimation. J. Vision 14, 7 (2014).
Allred, S. R. & Flombaum, J. I. Relating color working memory and color perception. Trends Cogn. Sci. 18, 562–565 (2014).
Pratte, M. S., Park, Y. E., Rademaker, R. L. & Tong, F.Accounting for stimulus-specific variation in precision reveals a discrete capacity limit in visual working memory. J. Exp. Psychol. Hum. Percept. Perform. 43, 6–17 (2017).
Torgerson, W. S. Theory and Methods of Scaling (Wiley, 1958).
Maloney, L. T. & Yang, J. N. Maximum likelihood difference scaling. J. Vision 3, 5 (2003).
Shepard, R. N. Toward a universal law of generalization for psychological science. Science 237, 1317–1323 (1987).
Sims, C. R. Efficient coding explains the universal law of generalization in human perception. Science 360, 652–656 (2018).
Nosofsky, R. M. Similarity scaling and cognitive process models. Annu. Rev. Psychol. 43, 25–53 (1992).
Tanner, W. P.Jr & Swets, J. A.A decision-making theory of visual detection. Psychol. Rev. 61, 401–409 (1954).
Macmillan, N. A. & Creelman, C. D. Detection Theory: A User’s Guide 2nd edn (Erlbaum, 2005).
Wilken, P. & Ma, W. J. A detection theory account of change detection. J. Vision 4, 11 (2004).
Fougnie, D., Suchow, J. W. & Alvarez, G. A. Variability in the quality of visual working memory. Nat. Commun. 3, 1229 (2012).
Loftus, G. R. & Bamber, D. Weak models, strong models, unidimensional models, and psychological time. J. Exp. Psychol. Learn. Mem. Cogn. 16, 16–19 (1990).
Smith, P. L., Lilburn, S. D., Corbett, E. A., Sewell, D. K. & Kyllingsbæk, S. The attention-weighted sample-size model of visual short-term memory: attention capture predicts resource allocation and memory load. Cogn. Psychol. 89, 71–105 (2016).
Bays, P. M., Catalao, R. F. & Husain, M. The precision of visual working memory is set by allocation of a shared resource. J. Vision 9, 7 (2009).
Roberts, S. & Pashler, H.How persuasive is a good fit? A comment on theory testing. Psychol. Rev. 107, 358–367 (2000).
Kahana, M. J. & Sekuler, R. Recognizing spatial patterns: a noisy exemplar approach. Vis. Res. 42, 2177–2192 (2002).
Gold, J. M., Wilk, C. M., McMahon, R. P., Buchanan, R. W. & Luck, S. J.Working memory for visual features and conjunctions in schizophrenia. J. Abnorm. Psychol. 112, 61–71 (2003).
Haberman, J., Brady, T. F. & Alvarez, G. A.Individual differences in ensemble perception reveal multiple, independent levels of ensemble representation. J. Exp. Psychol. Gen. 144, 432–446 (2015).
Miner, A. E., Schurgin, M. W. & Brady, T. F. Is working memory inherently more ‘precise’ than long-term memory? Extremely high fidelity visual long-term memories for frequently encountered objects. J. Exp. Psychol. Human Percept. Perform. 46, 813 (2020).
Brady, T. F., Konkle, T., Gill, J., Oliva, A. & Alvarez, G. A. Visual long-term memory has the same limit on fidelity as visual working memory. Psychol. Sci. 24, 981–990 (2013).
Asplund, C. L., Fougnie, D., Zughni, S., Martin, J. W. & Marois, R. The attentional blink reveals the probabilistic nature of discrete conscious perception. Psychol. Sci. 25, 824–831 (2014).
Pratte, M. S. Iconic memories die a sudden death. Psychol. Sci. 29, 877–887 (2018).
Richter, F. R., Cooper, R. A., Bays, P. M. & Simons, J. S.Distinct neural mechanisms underlie the success, precision, and vividness of episodic memory. eLife 5, e18260 (2016).
Dunn, J. C. & Kalish, M. L. State–Trace Analysis (Springer, 2018).
Zokaei, N. et al. Visual short-term memory deficits associated with GBA mutation and Parkinson’s disease. Brain 137, 2303–2311 (2014).
Rolinski, M. et al. Visual short-term memory deficits in REM sleep behaviour disorder mirror those in Parkinson’s disease. Brain 139, 47–53 (2015).
Pertzov, Y. et al. Binding deficits in memory following medial temporal lobe damage in patients with voltage-gated potassium channel complex antibody-associated limbic encephalitis. Brain 136, 2474–2485 (2013).
Brady, T. F. & Alvarez, G. A. Hierarchical encoding in visual working memory: ensemble statistics bias memory for individual items. Psychol. Sci. 22, 384–392 (2011).
Brady, T. F. & Alvarez, G. A. Contextual effects in visual working memory reveal hierarchically structured memory representations. J. Vision 15, 6 (2015).
Williams, J., Brady, T. & Störmer, V. S. Guidance of attention by working memory is a matter of representational fidelity, not a privileged status for one or more items. Preprint at PsyArXiv https://doi.org/10.31234/osf.io/c4t92 (2019).
Henson, R. N. A., Rugg, M. D., Shallice, T. & Dolan, R. J. Confidence in recognition memory for words: dissociating right prefrontal roles in episodic retrieval. J. Cogn. Neurosci. 12, 913–923 (2000).
Rutishauser, U. et al. Representation of retrieval confidence by single neurons in the human medial temporal lobe. Nat. Neurosci. 18, 1041–1050 (2015).
Marr, D. Vision: A Computational Investigation Into the Human Representation and Processing of Visual Information (Henry Holt and Company, 1982).
Bays, P. M. Correspondence between population coding and psychophysical scaling models of working memory. Preprint at BioRxiv https://doi.org/10.1101/699884 (2019).
Wei, X. X. & Stocker, A. A.A Bayesian observer model constrained by efficient coding can explain ‘anti-Bayesian’ percepts. Nat. Neurosci. 18, 1509–1517 (2015).
Wei, X. X. & Stocker, A. A. Lawful relation between perceptual bias and discriminability. Proc. Natl Acad. Sci. USA 114, 10244–10249 (2017).
Krauskopf, J. & Gegenfurtner, K. R. Color discrimination and adaptation. Vis. Res. 32, 2165–2175 (1992).
Giesel, M., Hansen, T. & Gegenfurtner, K. R. The discrimination of chromatic textures. J. Vision 9, 11 (2009).
Rademaker, R. L., Tredway, C. H. & Tong, F. Introspective judgments predict the precision and likelihood of successful maintenance of visual working memory. J. Vision 12, 21 (2012).
Wixted, J. T. & Wells, G. L.The relationship between eyewitness confidence and identification accuracy: a new synthesis. Psychol. Sci. Public Interest 18, 10–65 (2017).
Fougnie, D., Brady, T. F. & Alvarez, G. A. If at first you don't retrieve, try, try again: the role of retrieval failures in visual working memory. J. Vis. 14, 851–851 (2014).
Difallah, D., Filatova, E. & Ipeirotis, P. Demographics and dynamics of mechanical Turk workers. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining 135–143 (ACM, 2018).
Brady, T. F. & Tenenbaum, J. B.A probabilistic model of visual working memory: incorporating higher order regularities into working memory capacity estimates. Psychol. Rev. 120, 85–109 (2013).
Nadarajah, S., Afuecheta, E. & Chan, S. On the distribution of maximum of multivariate normal random vectors. Commun. Stat. Theory Methods 48, 2425–2445 (2019).
Fougnie, D., Asplund, C. L. & Marois, R. What are the units of storage in visual working memory? J. Vision 10, 27 (2010).
Suchow, J. W., Brady, T. F., Fougnie, D. & Alvarez, G. A. Modeling visual working memory with the MemToolbox. J. Vision 13, 9 (2013).
Myung, J. I. & Pitt, M. A. in Steven’s Handbook of Experimental Psychology and Cognitive Neuroscience 4th edn, Vol. 5 (eds Wixted, J. & Wagenmakers, E.-J.) 85–118 (John Wiley & Sons, 2018).
Brady, T. F., Konkle, T., Alvarez, G. A. & Oliva, A. Visual long-term memory has a massive storage capacity for object details. Proc. Natl Acad. Sci. USA 105, 14325–14329 (2008).
Wixted, J. T. & Mickes, L.A continuous dual-process model of remember/know judgments. Psychol. Rev. 117, 1025–1054 (2010).
Rohatgi, A. WebPlotDigitizer (2011); https://automeris.io/WebPlotDigitizer/
Thurstone, L. L. A law of comparative judgment. Psychol. Rev. 34, 273–286 (1927).
Rotello, C. M. in Learning and Memory: A Comprehensive Reference 2nd edn, Vol. 2 (eds Byrne, J. H. & Wixted, J. T.) 201–226 (Elsevier, 2017).
Acknowledgements
We thank V. Störmer, R. Rademaker, J. Wolfe, M. Robinson, D. Fougnie and T. Konkle for comments on these ideas and on the manuscript, and Y. H. Chung and B. Hawkins for help with data collection. For funding, we also acknowledge NSF CAREER (BCS-1653457; to T.F.B.). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.
Author information
Authors and Affiliations
Contributions
M.W.S. jointly conceived of the model with J.T.W. and T.F.B. M.W.S. and T.F.B. designed the experiments. T.F.B. wrote the code, ran the model and analysed the output data. M.W.S., J.T.W. and T.F.B. wrote the manuscript.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Peer review information Primary Handling Editor: Marike Schiffer.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data
Extended Data Fig. 1 Similarity as a function of distance in color space.
a, Data from all distances in the fixed distance triad task (Fig. 1c). On each trial, there was a target color, here always at 0°, and participants’ task was to choose which of two other colors was closer to the target color in color space. The two choice colors always differed by 30°. The x-axis shows the closer color of the two choice colors. That is, the 150° label on the x-axis reflects performance on a condition where the two choices were 150° and 180° away from the target color. As shown with a subset of this data in Fig. 1c, increasing distance from the target results in a decreased ability to tell which of two colors is closer to the target in color space. This shows the non-linearity of color space with respect to judgments of color similarity. Note that this function does not depict the actual psychophysical similarity function: Roughly speaking, the d’ estimate in this graph is the estimate of instantaneous slope (over a 30 deg. range) in the similarity function in Fig. 1f. b, Despite being conceived of as a color wheel in many memory experiments, in reality, participants internal representation of color–and thus the confusability between colors–ought to be a function of their linear distance in an approximately 3D color space, not their angular distance along the circumference of an artificially imposed wheel. Since the colors are equal luminance, we can conceive of this on a 2D plane. Thus, on this plane the confusability of a color “180 degrees away” on the wheel is only slightly lower than one “150 degrees away” on the wheel, since in 2D color space it is only slightly further away. This simple non-linearity from ignoring the global structure of the color ‘wheel’ partially explains the long tails observed in typical color report experiments, although it does not explain the full degree of this non-linearity, which is additionally attributable to psychophysical similarity being a non-linear function even of distance across 2D color space. c, The similarity function remains non-linear even in 2D color space. Distances here are scaled relative to the color wheel rather than in absolute CIELa*b* values., for example, an item 180 degrees opposite on the color wheel is “120” in real distance since if the distance along the circumference is 180, 120 is the distance across the color wheel. d, Plotted on a log axis, the similarity falls off approximately linearly, indicating that similarity falls of roughly exponentially with the exception of colors nearby the target. The non-exponential fall-off near the 0 point reflects perceptual noise/lack of perceptual discriminability between nearby colors. As shown in Fig. 1, when you convolve measured perceptual noise with an exponential function, this provides a very good fit to the similarity function, consistent with a wide-variety of evidence about the structure of similarity and generalization19.
Extended Data Fig. 2 Simulations of uncorrelated vs. correlated noise versions of TCC.
In the main text, we report d’ from a version of TCC where noise in similar color channels is correlated, based on measured perceptual confusions. However, this decision to correlate the noise of nearby colors is not critical, as shown in this simulation of uncorrelated vs. correlated noise versions of TCC. Only the correlated-noise TCC produces true d’ values–those that are interchangeable with d’ you’d estimate from a same/diff task with the same stimuli. However, the simpler uncorrelated noise TCC predicts the exact same distributions of errors in continuous report, and the d’ values between the correlated and uncorrelated noise models are linearly related by a factor of ~0.65. Thus, in many cases it may be useful to fit the uncorrelated TCC to data and then adjust the d’ rather than fitting correlated noise TCC. This also means that for color, similarity alone without perceptual confusion data can be used to make linear (but not exact) predictions about confusability in n-AFC tasks outside the range of perceptual confusion (approx. 15 deg).
Extended Data Fig. 3 Simulations comparing the measured psychological similarity function to a linear similarity function.
Simulations show data sampled from TCC, using either the measured psychological similarity function or a linear similarity function. Given a linear similarity function, it is clear TCC does not predict response distributions similar to human performance – accurate memory fits are critically dependent on the well-known exponential-like shape of similarity functions. Notice also how the max rule from the signal detection decision process plays a major role in the shape of the distributions. Since people pick the strongest signal, the distribution of max signals is peakier than the underlying signals themselves (which always follows the similarity function).
Extended Data Fig. 4 Comparisons of fit to memory data for different measured similarity functions.
Comparison of fit to memory data for similarity functions reported in main text. In the current data for color, both the model-based triad psychophysical scaling data and the Likert similarity rating produce extremely similar data (see Fig. 1). Thus, they all produce similar fits to the memory data (shown here are the set size data). It is important to note that depending on the number of trials, a large number of data points (that is subjects) may be necessary in order to obtain reliable estimates of a given stimulus space in the triad and quad scaling tasks (we use the quad task for face similarity). The Likert task requires considerably less data to estimate, and it was in agreement with the results of the triad task for colors, so we rely on it as our primary measure of similarity in the current fits. However, depending on the stimulus space, observers may utilize different strategies in such subjective similarity tasks (particularly for spaces, like orientation, where it is obviously a linear physical manipulation), and ultimately an objective task like the quad task may be best to understand the psychophysical similarity function. This is why for the face space task we used the quad similarity task. The task used to estimate similarity is important in that it is important that participants provide judgments of the absolute interval between stimuli and not rely on categories or verbal labels, or, in the triad task, that participants not rely on a relational or relative encoding of the two choice items rather than their absolute distance to the target item. How best to ensure that participants rely on absolute intervals is represented in a large literature dating to Thurstone63 and Torgerson15.
Extended Data Fig. 5 Non-uniformities across color space in memory and similarity.
Non-uniformities in memory and similarity for set size data reported in the main text. Many stimulus spaces contain non-uniformities, which may affect subsequent working memory performance. Indeed, Bae et al.12 discovered non-uniformities in working memory for color, where responses for targets tend to be more precise for some colors than others and can be biased towards nearby categorical anchors (that is red, blue, yellow, etc). While many assume randomizing target colors in working memory should account for potential biases arising from a non-uniform feature space, others have suggested these differences may have broader consequences than previously considered13,14. A key advantage of TCC is that by taking into account the psychophysical similarity function, non-uniformities within whatever feature space being probed can be automatically captured if psychophysical similarity data is measured separately from each relevant starting point in the feature space (for example, Fig. 1d). In the current work, we mostly use only a single psychophysical similarity estimate averaged across possible starting points and fit memory data averaged across starting points. However, this is not necessary to the TCC framework, and is only a simplification–if we wish to fit memory data averaged across all targets, we should use similarity averaged across all targets (or use the particular similarity function relevant to each item on each trial). Here we show that rather than using a psychophysical similarity function that averages over all targets, one can also use similarity specific to each possible target, which differ and have predictable consequences for memory in our set size experiment. For example, the propensity of errors (at set size 1, 3, 6 and 8) in the clockwise vs. counterclockwise direction for a given target color is directly predicted by the similarity function–even when very similar colors have more similar colors in opposite directions (top row), and this is true across all color bins (bottom right). Thus, using target-specific similarity functions naturally captures potential non-uniformities or biases within a feature space with no change in the TCC framework.
Extended Data Fig. 6 d’ as a function of set size.
Data from the set size experiment reported in the main text. While memory strength varies according to a variety of different factors, many researchers have been particularly interested in the influence of set size. TCC shows that at a fixed encoding time and with a fixed delay, memory strength (d’) decreases according to a power law as set size changes, broadly consistent with fixed resource theories of memory25,26. However, capacity cannot be fixed globally, as the total “capacity” appears to smoothly change with encoding time and delay and differs for different stimuli.
Extended Data Fig. 7 Variation in representational fidelity with the same d’ by separating on strength of strongest memory signal.
Simulation from TCC illustrating how signal detection can predict variance in representational fidelity as a function of confidence even with a fixed d’ (see also42). Some studies used to support variability of information across individual items or trials have done so by using a confidence metric51. While variability and confidence are distinct from one another, in a large amount of research they are inextricably linked. An interesting advantage and implication of signal detection-based models is that they naturally predict confidence data64. In particular, the strength of the winning memory match signal is used as the measure of memory strength–and confidence–in signal detection models of memory. Thus, even with a fixed d’ value for all items, TCC naturally predicts varying distributions relative to confidence. This likely explains some of the evidence previously observed in the literature that when distinguishing responses according to confidence, researchers found support for variability in precision among items / trials. Note that this occurs in TCC even though d’ is fixed in this simulation–that is, all trials are generated from a process with the same signal-to-noise ratio. Thus, variability in responses as a function of confidence (or related effects, like improved performance when participants choose their own favorite item to report23) are not evidence for variability in d’ in TCC, but simply a natural prediction of the underlying signal detection process. Of course, it is possible d’ may also vary between items, which remains an open question.
Extended Data Fig. 8 Simulation of expected confidence as a function of set size in TCC.
Participants in a set size 8 working memory experiment often feel like they do not remember an item and are “guessing”, leading to a wide variety of models that predict people know nothing about many items at high set sizes and truly are objectively guessing. However, as noted in Extended Data Fig. 7, signal detection naturally accounts for varying confidence, and so can easily account for this subjective feeling of guessing even though in fact, models like TCC predict that people are almost never responding based on no information at all about the item they just saw. In particular, confidence in signal detection is based on the strength of the winning memory signal. Imagine that the subjective feeling of guessing occurs whenever your memory match signal is below some threshold (here, arbitrarily set to 2.75). This would lead to people never feeling like they are guessing at set size 1, and nearly always feeling like they are guessing if they objectively closed their eyes and saw nothing. However, this would also make people feel like they are guessing a large part of the time at set size 6 and 8, even though this data is simulated from TCC–and the generative process always contains information about all items. This is the key distinction in signal detection models between the subjective feeling of guessing and the claim that people are objectively guessing.
Supplementary information
Supplementary Information
Supplementary discussion, Supplementary Figs. 1–6 and Supplementary Tables 1–4.
Rights and permissions
About this article
Cite this article
Schurgin, M.W., Wixted, J.T. & Brady, T.F. Psychophysical scaling reveals a unified theory of visual memory strength. Nat Hum Behav 4, 1156–1172 (2020). https://doi.org/10.1038/s41562-020-00938-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41562-020-00938-0
This article is cited by
-
Scaling models of visual working memory to natural images
Communications Psychology (2024)
-
Noisy and hierarchical visual memory across timescales
Nature Reviews Psychology (2024)
-
Meaningful objects avoid attribute amnesia due to incidental long-term memories
Scientific Reports (2023)
-
The roles of attention, executive function and knowledge in cognitive ageing of working memory
Nature Reviews Psychology (2023)
-
Feature-based attention warps the perception of visual features
Scientific Reports (2023)