Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Geometric models reveal behavioural and neural signatures of transforming experiences into memories

Nature Human Behaviour volume 5, pages 905–919 (2021)Cite this article

Subjects

Abstract

How do we preserve and distort our ongoing experiences when encoding them into episodic memories? The mental contexts in which we interpret experiences are often person-specific, even when the experiences themselves are shared. Here we develop a geometric framework for mathematically characterizing the subjective conceptual content of dynamic naturalistic experiences. We model experiences and memories as trajectories through word-embedding spaces whose coordinates reflect the universe of thoughts under consideration. Memory encoding can then be modelled as geometrically preserving or distorting the ‘shape’ of the original experience. We applied our approach to data collected as participants watched and verbally recounted a television episode while undergoing functional neuroimaging. Participants’ recountings preserved coarse spatial properties (essential narrative elements) but not fine spatial scale (low-level) details of the episode’s trajectory. We also identified networks of brain structures sensitive to these trajectory shapes.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Topic weights in episode and recall content.
Fig. 2: Modelling naturalistic stimuli and recalls.
Fig. 3: Naturalistic extensions of classic list-learning memory analyses.
Fig. 4: Novel content-based metrics of naturalistic memory: precision and distinctiveness.
Fig. 5: Precision reflects the completeness of recall, whereas distinctiveness reflects recall specificity.
Fig. 6: Trajectories through topic space capture the dynamic content of the episode and recalls.
Fig. 7: Language used in the most and least precisely remembered events.
Fig. 8: Brain structures that underlie the transformation of experience into memory.

Similar content being viewed by others

Data availability

The fMRI data we analysed are available online at https://dataspace.princeton.edu/jspui/handle/88435/dsp01nz8062179. The behavioural data are available at https://github.com/ContextLab/sherlock-topic-model-paper/tree/master/data/raw.

Code availability

All of our analysis code can be downloaded from https://github.com/ContextLab/sherlock-topic-model-paper.

References

  1. Murdock, B. B. The serial position effect of free recall. J. Exp. Psychol. 64, 482–488 (1962).

    Article  Google Scholar 

  2. Kahana, M. J. Associative retrieval processes in free recall. Mem. Cogn. 24, 103–109 (1996).

    Article  CAS  Google Scholar 

  3. Yonelinas, A. P. The nature of recollection and familiarity: a review of 30 years of research. J. Mem. Lang. 46, 441–517 (2002).

    Article  Google Scholar 

  4. Kahana, M. J. Foundations of Human Memory (Oxford Univ. Press, 2012).

  5. Koriat, A. & Goldsmith, M. Memory in naturalistic and laboratory contexts: distinguishing accuracy-oriented and quantity-oriented approaches to memory assessment. J. Exp. Psychol. 123, 297–315 (1994).

    Article  CAS  Google Scholar 

  6. Huk, A., Bonnen, K. & He, B. J. Beyond trial-based paradigms: continuous behavior, ongoing neural activity, and naturalistic stimuli. J. Neurosci. 38, 7551–7558 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  7. Lerner, Y., Honey, C. J., Silbert, L. J. & Hasson, U. Topographic mapping of a hierarchy of temporal receptive windows using a narrated story. J. Neurosci. 31, 2906–2915 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. Manning, J. R. Episodic memory: mental time travel or a quantum ‘memory wave’ function? Preprint at OSF https://doi.org/10.31234/osf.io/6zjwb (2019).

  9. Manning, J. R. in Handbook of Human Memory (eds Kahana, M. J. & Wagner, A. D.) (Oxford Univ. Press, in the press).

  10. Howard, M. W. & Kahana, M. J. A distributed representation of temporal context. J. Math. Psychol. 46, 269–299 (2002).

    Article  Google Scholar 

  11. Howard, M. W. et al. A unified mathematical framework for coding time, space, and sequences in the medial temporal lobe. J. Neurosci. 34, 4692–4707 (2014).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  12. Manning, J. R., Norman, K. A. & Kahana, M. J. in The Cognitive Neurosciences 5th edition (ed. Gazzaniga, M.) 557–566 (MIT Press, 2015).

  13. Ranganath, C. & Ritchey, M. Two cortical systems for memory-guided behavior. Nat. Rev. Neurosci. 13, 713–726 (2012).

    Article  CAS  PubMed  Google Scholar 

  14. Zacks, J. M., Speer, N. K., Swallow, K. M., Braver, T. S. & Reynolds, J. R. Event perception: a mind–brain perspective. Psychol. Bull. 133, 273–293 (2007).

    Article  PubMed  PubMed Central  Google Scholar 

  15. Zwaan, R. A. & Radvansky, G. A. Situation models in language comprehension and memory. Psychol. Bull. 123, 162 – 185 (1998).

    Article  Google Scholar 

  16. Radvansky, G. A. & Zacks, J. M. Event boundaries in memory and cognition. Curr. Opin. Behav. Sci. 17, 133–140 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  17. Brunec, I. K., Moscovitch, M. M. & Barense, M. D. Boundaries shape cognitive representations of spaces and events. Trends Cogn. Sci. 22, 637–650 (2018).

    Article  PubMed  Google Scholar 

  18. Heusser, A. C., Ezzyat, Y., Shiff, I. & Davachi, L. Perceptual boundaries cause mnemonic trade-offs between local boundary processing and across-trial associative binding. J. Exp. Psychol. Learn. Mem. Cogn. 44, 1075–1090 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  19. Clewett, D. & Davachi, L. The ebb and flow of experience determines the temporal structure of memory. Curr. Opin. Behav. Sci. 17, 186–193 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  20. Ezzyat, Y. & Davachi, L. What constitutes an episode in episodic memory? Psychol. Sci. 22, 243–252 (2011).

    Article  PubMed  Google Scholar 

  21. DuBrow, S. & Davachi, L. The influence of contextual boundaries on memory for the sequential order of events. J. Exp. Psychol. 142, 1277–1286 (2013).

    Article  Google Scholar 

  22. Tompary, A. & Davachi, L. Consolidation promotes the emergence of representational overlap in the hippocampus and medial prefrontal cortex. Neuron 96, 228–241 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  23. Chen, J. et al. Shared memories reveal shared structure in neural activity across individuals. Nat. Neurosci. 20, 115 (2017).

    Article  CAS  PubMed  Google Scholar 

  24. Blei, D. M., Ng, A. Y. & Jordan, M. I. Latent Dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003).

    Google Scholar 

  25. Rabiner, L. A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77, 257–286 (1989).

    Article  Google Scholar 

  26. Baldassano, C. et al. Discovering event structure in continuous narrative perception and memory. Neuron 95, 709–721 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  27. Blei, D. M. & Lafferty, J. D. Dynamic topic models. In Proc. 23rd International Conference on Machine Learning, ICML ’06 113–120 (ACM, 2006).

  28. Manning, J. R., Polyn, S. M., Baltuch, G., Litt, B. & Kahana, M. J. Oscillatory patterns in temporal lobe reveal context reinstatement during memory search. Proc. Natl Acad. Sci. USA 108, 12893–12897 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  29. Howard, M. W., Viskontas, I. V., Shankar, K. H. & Fried, I. Ensembles of human MTL neurons ‘jump back in time’ in response to a repeated stimulus. Hippocampus 22, 1833–1847 (2012).

    Article  PubMed  PubMed Central  Google Scholar 

  30. Atkinson, R. C. & Shiffrin, R. M. Human memory: a proposed system and its control processes. J. Learn. Motiv. 2, 89–105 (1968).

    Google Scholar 

  31. Postman, L. & Phillips, L. W. Short-term temporal changes in free recall. Q. J. Exp. Psychol. 17, 132–138 (1965).

    Article  Google Scholar 

  32. Welch, G. B. & Burnett, C. T. Is primacy a factor in association-formation. Am. J. Psychol. 35, 396–401 (1924).

    Article  Google Scholar 

  33. Polyn, S. M., Norman, K. A. & Kahana, M. J. A context maintenance and retrieval model of organizational processes in free recall. Psychol. Rev. 116, 129–156 (2009).

    Article  PubMed  PubMed Central  Google Scholar 

  34. Manning, J. R. & Kahana, M. J. Interpreting semantic clustering effects in free recall. Memory 20, 511–517 (2012).

    Article  PubMed  PubMed Central  Google Scholar 

  35. Heusser, A. C., Ziman, K., Owen, L. L. W. & Manning, J. R. HyperTools: a Python toolbox for gaining geometric insights into high-dimensional data. J. Mach. Learn. Res. 18, 1–6 (2018).

    Google Scholar 

  36. McInnes, L., Healy, J. & Melville, J. UMAP: uniform manifold approximation and projection for dimension reduction. Preprint at arXiv https://arxiv.org/abs/1802.03426 (2018).

  37. Paller, K. A. & Wagner, A. D. Observing the transformation of experience into memory. Trends Cogn. Sci. 6, 93–102 (2002).

    Article  PubMed  Google Scholar 

  38. Yarkoni, T., Poldrack, R. A., Nichols, T. E., Van Essen, D. C. & Wager, T. D. Large-scale automated synthesis of human functional neuroimaging data. Nat. Methods 8, 665 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  39. Bellmund, J. L. S., Gärdenfors, P., Moser, E. I. & Doeller, C. F. Navigating cognition: spatial codes for human thinking. Science 362, eaat6766 (2018).

    Article  PubMed  CAS  Google Scholar 

  40. Bellmund, J. L. S. et al. Deforming the metric of cognitive maps distorts memory. Nat. Hum. Behav. 4, 177–188 (2020).

    Article  PubMed  Google Scholar 

  41. Constantinescu, A. O., O’Reilly, J. X. & Behrens, T. E. J. Organizing conceptual knowledge in humans with a gridlike code. Science 352, 1464–1468 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  42. Gilboa, A. & Marlatte, H. Neurobiology of schemas and schema-mediated memory. Trends Cogn. Sci. 21, 618–631 (2017).

    Article  PubMed  Google Scholar 

  43. Baldassano, C., Hasson, U. & Norman, K. A. Representation of real-world event schemas during narrative perception. J. Neurosci. 38, 9689–9699 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  44. Huth, A. G., Nisimoto, S., Vu, A. T. & Gallant, J. L. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron 76, 1210–1224 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  45. Huth, A. G., de Heer, W. A., Griffiths, T. L., Theunissen, F. E. & Gallant, J. L. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature 532, 453–458 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  46. Gagnepain, P. et al. Collective memory shapes the organization of individual memories in the medial prefrontal cortex. Nat. Hum. Behav. 4, 189–200 (2020).

    Article  PubMed  Google Scholar 

  47. Simony, E., Honey, C. J., Chen, J. & Hasson, U. Dynamic reconfiguration of the default mode network during narrative comprehension. Nat. Commun. 7, 1–13 (2016).

    Article  CAS  Google Scholar 

  48. Zadbood, A., Chen, J., Leong, Y. C., Norman, K. A. & Hasson, U. How we transmit memories to other brains: Constructing shared neural representations via communication. Cereb. Cortex 27, 4988–5000 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  49. Simony, E. & Chang, C. Analysis of stimulus-induced brain dynamics during naturalistic paradigms. NeuroImage 216, 116461 (2020).

    Article  PubMed  Google Scholar 

  50. Landauer, T. K. & Dumais, S. T. A solution to Plato’s problem: the latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychol. Rev. 104, 211–240 (1997).

    Article  Google Scholar 

  51. Mikolov, T., Chen, K., Corrado, G. & Dean, J. Efficient estimation of word representations in vector space. Preprint at arXiv https://arxiv.org/abs/1301.3781 (2013).

  52. Cer, D. et al. Universal sentence encoder. Preprint at arXiv https://arxiv.org/abs/1803.11175 (2018).

  53. Radford, A. et al. Language models are unsupervised multitask learners. OpenAI Blog 1, 9 (2019).

    Google Scholar 

  54. Brown, T. B. et al. Language models are few-shot learners. Preprint at arXiv https://arxiv.org/abs/2005.14165 (2020).

  55. Pedregosa, F. et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011).

    Google Scholar 

  56. Hasson, U., Yang, E., Vallines, I., Heeger, D. J. & Rubin, N. A hierarchy of temporal receptive windows in human cortex. J. Neurosci. 28, 2539–2550 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  57. Hasson, U., Chen, J. & Honey, C. J. Hierarchical process memory: memory as an integral component of information processing. Trends Cogn. Sci. 19, 304–315 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  58. Dobrushin, R. L. Prescribing a system of random variables by conditional distributions. Theory Probab. Appl. 15, 458–486 (1970).

    Article  Google Scholar 

  59. Ramdas, A., Trillos, N. & Cuturi, M. On Wasserstein two-sample testing and related families of nonparametric tests. Entropy 19, 47 (2017).

    Article  Google Scholar 

  60. Heusser, A. C., Fitzpatrick, P. C., Field, C. E., Ziman, K. & Manning, J. R. Quail: a Python toolbox for analyzing and plotting free recall data. J. Open Source Softw. 2, 424 (2017).

    Article  Google Scholar 

  61. Fisher, R. A.Statistical Methods for Research Workers (Oliver and Boyd, 1925).

  62. Berndt, D. J. & Clifford, J. Using dynamic time warping to find patterns in time series. In AAAIWS ’94: Proc. 3rd International Conference on Knowledge Discovery and Data Mining 359–370 (1994).

  63. Freedman, D., Riesenhuber, M., Poggio, T. & Miller, E. Categorical representation of visual stimuli in the primate prefrontal cortex. Science 291, 312–316 (2001).

    Article  CAS  PubMed  Google Scholar 

  64. Sigman, M. & Dehaene, S. Brain mechanisms of serial and parallel processing during dual-task performance. J. Neurosci. 28, 7585–7589 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  65. Charron, S. & Koechlin, E. Divided representations of current goals in the human frontal lobes. Science 328, 360–363 (2010).

    Article  CAS  PubMed  Google Scholar 

  66. Rishel, C. A., Huang, G. & Freedman, D. J. Independent category and spatial encoding in parietal cortex. Neuron 77, 969–979 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  67. Kriegeskorte, N., Mur, M. & Bandettini, P. Representational similarity analysis—connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2, 1 – 28 (2008).

    Google Scholar 

Download references

Acknowledgements

We thank L. Chang, J. Chen, C. Honey, C. Lee, L. Owen, E. Whitaker, X. Xu and K. Ziman for feedback and scientific discussions, and we thank J. Chen, Y. C. Leong, C. Honey, C. Yong, K. Norman and U. Hasson for sharing the data used in our study. Our work was supported in part by National Science Foundation Established Program to Stimulate Competitive Research Award number 1632738. The content is solely the responsibility of the authors and does not necessarily represent the official views of our supporting organizations. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Author notes

  1. These authors contributed equally: Andrew C. Heusser, Paxton C. Fitzpatrick.

Authors and Affiliations

  1. Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA

    Andrew C. Heusser, Paxton C. Fitzpatrick & Jeremy R. Manning

  2. Akili Interactive Labs, Boston, MA, USA

    Andrew C. Heusser

Authors
  1. Andrew C. Heusser
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paxton C. Fitzpatrick
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jeremy R. Manning
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization: A.C.H. and J.R.M.; methodology: A.C.H., P.C.F. and J.R.M.; software: A.C.H., P.C.F. and J.R.M.; analysis: A.C.H., P.C.F. and J.R.M.; writing, reviewing and editing: A.C.H., P.C.F. and J.R.M.; and supervision: J.R.M.

Corresponding author

Correspondence to Jeremy R. Manning.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Primary Handling Editor: Marike Schiffer.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Methods detail for recall trajectory analysis displayed in Figure 6B.

A. This panel replicates Figure 6B, but with two additions. First, individual participants’ recall trajectories are displayed (faintly) as light gray lines. Second, three locations on the trajectory have been highlighted (blue, yellow, and red circles). B. These zoomed-in views of the locations highlighted in Panel A show the average trajectory (black) and individual participants’ trajectories (gray lines) that intersect the given region of topic space. C. For each circular region of topic space tiling the 2D embedding plane displayed in Panel A, we compute the distribution of angles formed between each participant’s trajectory segment (that is, the point at which the trajectory enters and exists the region of topic space) and the x-axis. The distributions of angles for these three example regions are displayed in the colored rose plots. We use Rayleigh tests to assign an arrow direction, length, and color for that region of topic space. Arrows displayed in color are significant at the p < 0.05 level (corrected). The arrow directions are oriented according to the circular means of each distribution, and the arrow lengths are proportional to the lengths of those mean vectors. The example regions have been oriented from left to right in decreasing order of consistency across participants.

Extended Data Fig. 2 Recall temporal correlation matrices and event segmentation fits.

Each panel is in the same format as Figure 2E. The yellow boxes indicate HMM-identified event boundaries.

Extended Data Fig. 3 Episode-recall event correlation matrices.

Each panel is in the same format as Figure 2G. The yellow boxes mark the matched episode event for each recall event (that is, the maximum correlation in each column).

Extended Data Fig. 4 Episode and recall topic proportions matrix K-optimization functions.

We selected the optimal K-value for the episode and each recall topic proportions matrix using the formula described in Methods. This computation resulted in a curve for each matrix, describing the Wasserstein distance between the distributions of within-event and across-event topic vector correlations, as a function of K.

Supplementary information

Supplementary Information

Supplementary Methods, Supplementary Figs. 1–3 and Supplementary References.

Reporting Summary

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Heusser, A.C., Fitzpatrick, P.C. & Manning, J.R. Geometric models reveal behavioural and neural signatures of transforming experiences into memories. Nat Hum Behav 5, 905–919 (2021). https://doi.org/10.1038/s41562-021-01051-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41562-021-01051-6

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing