1932

Abstract

In addition to the role that our visual system plays in determining what we are seeing right now, visual computations contribute in important ways to predicting what we will see next. While the role of memory in creating future predictions is often overlooked, efficient predictive computation requires the use of information about the past to estimate future events. In this article, we introduce a framework for understanding the relationship between memory and visual prediction and review the two classes of mechanisms that the visual system relies on to create future predictions. We also discuss the principles that define the mapping from predictive computations to predictive mechanisms and how downstream brain areas interpret the predictive signals computed by the visual system.

Keyword(s): memorypredictionvision
Loading

Article metrics loading...

/content/journals/10.1146/annurev-vision-093019-112249
2021-09-15
2024-03-28
Loading full text...

Full text loading...

/deliver/fulltext/vision/7/1/annurev-vision-093019-112249.html?itemId=/content/journals/10.1146/annurev-vision-093019-112249&mimeType=html&fmt=ahah

Literature Cited

  1. Aly M, Turk-Browne NB 2016. Attention promotes episodic encoding by stabilizing hippocampal representations. PNAS 113:E420–29
    [Google Scholar]
  2. Aly M, Turk-Browne NB 2017. How hippocampal memory shapes, and is shaped by, attention. The Hippocampus from Cells to Systems: Structure, Connectivity, and Functional Contributions to Memory and Flexible Cognition DE Hannula, MC Duff 369–403 Berlin: Springer
    [Google Scholar]
  3. Anstis S. 2007. The flash-lag effect during illusory chopstick rotation. Perception 36:1043–48
    [Google Scholar]
  4. Bastos AM, Usrey WM, Adams RA, Mangun GR, Fries P, Friston KJ. 2012. Canonical microcircuits for predictive coding. Neuron 76:695–711
    [Google Scholar]
  5. Bellemare MG, Srinivasan S, Ostrovski G, Schaul T, Saxton D, Munos R 2016. Unifying count-based exploration and intrinsic motivation. Advances in Neural Information Processing Systems 29 (NIPS 2016) D Lee, M Sugiyama, U Luxburg, I Guyon, R Garnett 1471–79 N.p.: NeurIPS
    [Google Scholar]
  6. Berry MJ II, Brivanlou IH, Jordan TA, Meister M. 1999. Anticipation of moving stimuli by the retina. Nature 398:334–38
    [Google Scholar]
  7. Berry MJ II, Schwartz G 2011. The retina as embodying predictions about the visual world. Predictions in the Brain: Using Our Past to Generate a Future M Bar 295–310 Oxford, UK: Oxford Univ. Press
    [Google Scholar]
  8. Bialek W, Nemenman I, Tishby N. 2001. Predictability, complexity, and learning. Neural Comput 13:2409–63
    [Google Scholar]
  9. Bogacz R, Brown MW. 2003. Comparison of computational models of familiarity discrimination in the perirhinal cortex. Hippocampus 13:494–524
    [Google Scholar]
  10. Caplar N, Tacchella S, Birrer S. 2017. Quantitative evaluation of gender bias in astronomical publications from citation counts. Nat. Astron. 1:0141
    [Google Scholar]
  11. Chalk M, Marre O, Tkačik G 2018. Toward a unified theory of efficient, predictive, and sparse coding. PNAS 115:186–91
    [Google Scholar]
  12. Chávez AE, Grimes WN, Diamond JS. 2010. Mechanisms underlying lateral GABAergic feedback onto rod bipolar cells in rat retina. J. Neurosci. 30:2330–39
    [Google Scholar]
  13. Chen EY, Chou J, Park J, Schwartz G, Berry MJ. 2014. The neural circuit mechanisms underlying the retinal response to motion reversal. J. Neurosci. 34:15557–75
    [Google Scholar]
  14. Cohen NJ, Eichenbaum H. 1993. Memory, Amnesia, and the Hippocampal System Cambridge, MA: MIT Press
  15. de Lange FP, Heilbron M, Kok P. 2018. How do expectations shape perception?. Trends Cogn. Sci. 22:764–79
    [Google Scholar]
  16. Desimone R. 1996. Neural mechanisms for visual memory and their role in attention. PNAS 93:13494–99
    [Google Scholar]
  17. Dong C-J, Werblin FS. 1998. Temporal contrast enhancement via GABAC feedback at bipolar terminals in the tiger salamander retina. J. Neurophysiol. 79:2171–80
    [Google Scholar]
  18. Drinnenberg A, Franke F, Morikawa RK, Jüttner J, Hillier D et al. 2018. How diverse retinal functions arise from feedback at the first visual synapse. Neuron 99:117–34.e11
    [Google Scholar]
  19. Dworkin JD, Linn KA, Teich EG, Zurn P, Shinohara RT, Bassett DS. 2020. The extent and drivers of gender imbalance in neuroscience reference lists. Nat. Neurosci. 23:918–26
    [Google Scholar]
  20. Eagleman DM, Sejnowski TJ. 2000. Motion integration and postdiction in visual awareness. Science 287:2036–38
    [Google Scholar]
  21. Friston K. 2005. A theory of cortical responses. Philos. Trans. R. Soc. Lond. B 360:815–36
    [Google Scholar]
  22. Gold JI, Stocker AA. 2017. Visual decision-making in an uncertain and dynamic world. Annu. Rev. Vis. Sci. 3:227–50
    [Google Scholar]
  23. Gollisch T, Meister M. 2010. Eye smarter than scientists believed: neural computations in circuits of the retina. Neuron 65:150–64
    [Google Scholar]
  24. Gottlieb J, Oudeyer PY. 2018. Towards a neuroscience of active sampling and curiosity. Nat. Rev. Neurosci. 19:758–70
    [Google Scholar]
  25. Grotheer M, Kovacs G. 2014. Repetition probability effects depend on prior experiences. J. Neurosci. 34:6640–46
    [Google Scholar]
  26. Hawkins J, Blakeslee S. 2004. On Intelligence New York: Times Books
  27. Hazelhoff FF, Wiersma H. 1924. Die Wahrnehmungszeit. [The sensation time. .] Z. Psychol. 96:171–88
    [Google Scholar]
  28. Heeger DJ. 2017. Theory of cortical function. PNAS 114:1773–82
    [Google Scholar]
  29. Hindy NC, Ng FY, Turk-Browne NB. 2016. Linking pattern completion in the hippocampus to predictive coding in visual cortex. Nat. Neurosci. 19:665–67
    [Google Scholar]
  30. Hosoya T, Baccus SA, Meister M. 2005. Dynamic predictive coding by the retina. Nature 436:71–77
    [Google Scholar]
  31. Houthooft R, Chen X, Duan Y, Schulman J, De Turck F, Abbeel P 2016. VIME: variational information maximizing exploration. Advances in Neural Information Processing Systems 29 (NIPS 2016) D Lee, M Sugiyama, U Luxburg, I Guyon, R Garnett 1109–17 N.p.: NeurIPS
    [Google Scholar]
  32. Jackman SL, Babai N, Chambers JJ, Thoreson WB, Kramer RH. 2011. A positive feedback synapse from retinal horizontal cells to cone photoreceptors. PLOS Biol 9:e1001057
    [Google Scholar]
  33. Jehee JF, Ballard DH. 2009. Predictive feedback can account for biphasic responses in the lateral geniculate nucleus. PLOS Comput. Biol. 5:e1000373
    [Google Scholar]
  34. Jutras MJ, Buffalo EA 2010. Recognition memory signals in the macaque hippocampus. PNAS 107:401–6
    [Google Scholar]
  35. Kaliukhovich DA, Vogels R. 2011. Stimulus repetition probability does not affect repetition suppression in macaque inferior temporal cortex. Cereb. Cortex 21:1547–58
    [Google Scholar]
  36. Kanai R, Sheth BR, Shimojo S. 2004. Stopping the motion and sleuthing the flash-lag effect: spatial uncertainty is the key to perceptual mislocalization. Vis. Res. 44:2605–19
    [Google Scholar]
  37. Kastner DB, Baccus SA. 2013. Spatial segregation of adaptation and predictive sensitization in retinal ganglion cells. Neuron 79:541–54
    [Google Scholar]
  38. Kastner DB, Baccus SA. 2014. Insights from the retina into the diverse and general computations of adaptation, detection, and prediction. Curr. Opin. Neurobiol. 25:63–69
    [Google Scholar]
  39. Keller GB, Mrsic-Flogel TD. 2018. Predictive processing: a canonical cortical computation. Neuron 100:424–35
    [Google Scholar]
  40. Khoei MA, Masson GS, Perrinet LU. 2017. The flash-lag effect as a motion-based predictive shift. PLOS Comput. Biol. 13:e1005068
    [Google Scholar]
  41. Kiebel SJ, Daunizeau J, Friston KJ. 2008. A hierarchy of time-scales and the brain. PLOS Comput. Biol. 4:e1000209
    [Google Scholar]
  42. Kim JG, Gregory E, Landau B, McCloskey M, Turk-Browne NB, Kastner S 2020. Functions of ventral visual cortex after bilateral medial temporal lobe damage. Prog. Neurobiol. 191:101819
    [Google Scholar]
  43. Kohn A. 2007. Visual adaptation: physiology, mechanisms, and functional benefits. J. Neurophysiol. 97:3155–64
    [Google Scholar]
  44. Koster-Hale J, Saxe R 2013. Theory of mind: a neural prediction problem. Neuron 79:836–48
    [Google Scholar]
  45. Kovacs G, Kaiser D, Kaliukhovich DA, Vidnyanszky Z, Vogels R. 2013. Repetition probability does not affect fMRI repetition suppression for objects. J. Neurosci. 33:9805–12
    [Google Scholar]
  46. Lee TS, Mumford D. 2003. Hierarchical Bayesian inference in the visual cortex. J. Opt. Soc. Am. A 20:1434–48
    [Google Scholar]
  47. Li L, Miller EK, Desimone R. 1993. The representation of stimulus familiarity in anterior inferior temporal cortex. J. Neurophysiol. 69:1918–29
    [Google Scholar]
  48. Lopez-Moliner J, Linares D. 2006. The flash-lag effect is reduced when the flash is perceived as a sensory consequence of our action. Vis. Res. 46:2122–29
    [Google Scholar]
  49. Lotter W, Kreiman G, Cox DD. 2017. Deep predictive coding networks for video prediction and unsupervised learning Paper presented at 5th International Conference on Learning Representations (ICLR) Toulon, France: Apr. 24–26
  50. Maliniak D, Powers R, Walter BF. 2013. The gender citation gap in international relations. Int. Organ. 67:889–922
    [Google Scholar]
  51. Marr D. 1971. Simple memory: a theory for archicortex. Philos. Trans. R. Soc. Lond. B 262:23–81
    [Google Scholar]
  52. Marzen SE, DeDeo S. 2017. The evolution of lossy compression. J. R. Soc. Interface 14:0166
    [Google Scholar]
  53. Meyer T, Olson CR 2011. Statistical learning of visual transitions in monkey inferotemporal cortex. PNAS 108:19401–6
    [Google Scholar]
  54. Meyer T, Ramachandran S, Olson CR. 2014. Statistical learning of serial visual transitions by neurons in monkey inferotemporal cortex. J. Neurosci. 34:9332–37
    [Google Scholar]
  55. Meyer T, Rust NC 2018. Single-exposure visual memory judgments are reflected in inferotemporal cortex. eLife 7:e32259
    [Google Scholar]
  56. Miyashita Y. 1988. Neuronal correlate of visual associative long-term memory in the primate temporal cortex. Nature 335:817–20
    [Google Scholar]
  57. Mlynarski WF, Hermundstad AM 2018. Adaptive coding for dynamic sensory inference. eLife 7:e32055
    [Google Scholar]
  58. Nieman D, Nijhawan R, Khurana B, Shimojo S. 2006. Cyclopean flash-lag illusion. Vis. Res. 46:3909–14
    [Google Scholar]
  59. Nijhawan R. 2002. Neural delays, visual motion and the flash-lag effect. Trends Cogn. Sci. 6:387–93
    [Google Scholar]
  60. Palmer SE, Marre O, Berry MJ II, Bialek W. 2015. Predictive information in a sensory population. PNAS 112:6908–13
    [Google Scholar]
  61. Patel SS, Ogmen H, Bedell HE, Sampath V. 2000. Flash-lag effect: differential latency, not postdiction. Science 290:1051
    [Google Scholar]
  62. Pathak D, Agrawal P, Efros AA, Darrell T. 2017. Curiosity-driven exploration by self-supervised prediction. ICML '17: Proceedings of the 34th International Conference on Machine Learning2778–87 New York: Assoc. Comput. Mach.
    [Google Scholar]
  63. Ranganath C, Rainer G 2003. Neural mechanisms for detecting and remembering novel events. Nat. Rev. Neurosci. 4:193–202
    [Google Scholar]
  64. Rao RP, Ballard DH. 1997. Dynamic model of visual recognition predicts neural response properties in the visual cortex. Neural Comput 9:721–63
    [Google Scholar]
  65. Rao RP, Ballard DH. 1999. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat. Neurosci. 2:79–87
    [Google Scholar]
  66. Sakon JJ, Suzuki WA 2019. A neural signature of pattern separation in the monkey hippocampus. PNAS 116:9634–43
    [Google Scholar]
  67. Salisbury JM, Palmer SE. 2016. Optimal prediction in the retina and natural motion statistics. J. Stat. Phys. 162:1309–23
    [Google Scholar]
  68. Sawamura H, Orban GA, Vogels R. 2006. Selectivity of neuronal adaptation does not match response selectivity: a single-cell study of the fMRI adaptation paradigm. Neuron 49:307–18
    [Google Scholar]
  69. Schacter DL, Addis DR, Buckner RL. 2007. Remembering the past to imagine the future: the prospective brain. Nat. Rev. Neurosci. 8:657–61
    [Google Scholar]
  70. Schapiro AC, Gregory E, Landau B, McCloskey M, Turk-Browne NB. 2014. The necessity of the medial temporal lobe for statistical learning. J. Cogn. Neurosci. 26:1736–47
    [Google Scholar]
  71. Schapiro AC, Kustner LV, Turk-Browne NB. 2012. Shaping of object representations in the human medial temporal lobe based on temporal regularities. Curr. Biol. 22:1622–27
    [Google Scholar]
  72. Schwartz G, Taylor S, Fisher C, Harris R, Berry MJ II 2007. Synchronized firing among retinal ganglion cells signals motion reversal. Neuron 55:958–69
    [Google Scholar]
  73. Schwiedrzik CM, Freiwald WA. 2017. High-level prediction signals in a low-level area of the macaque face-processing hierarchy. Neuron 96:89–97.e4
    [Google Scholar]
  74. Sheth BR, Nijhawan R, Shimojo S. 2000. Changing objects lead briefly flashed ones. Nat. Neurosci. 3:489–95
    [Google Scholar]
  75. Shioiri S, Yamamoto K, Oshida H, Matsubara K, Yaguchi H. 2010. Measuring attention using flash-lag effect. J. Vis. 10:10
    [Google Scholar]
  76. Srinivasan MV, Laughlin SB, Dubs A. 1982. Predictive coding: a fresh view of inhibition in the retina. Proc. R. Soc. Lond. B 216:427–59
    [Google Scholar]
  77. Stadie BC, Levine S, Abbeel P. 2015. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv:1507.00814 [cs.AI]
  78. Still S. 2014. Information bottleneck approach to predictive inference. Entropy 16:968–89
    [Google Scholar]
  79. Subramaniyan M, Ecker AS, Patel SS, Cotton RJ, Bethge M et al. 2018. Faster processing of moving compared with flashed bars in awake macaque V1 provides a neural correlate of the flash lag illusion. J. Neurophysiol. 120:2430–52
    [Google Scholar]
  80. Summerfield C, Trittschuh EH, Monti JM, Mesulam MM, Egner T. 2008. Neural repetition suppression reflects fulfilled perceptual expectations. Nat. Neurosci. 11:1004–6
    [Google Scholar]
  81. Thoreson WB, Babai N, Bartoletti TM. 2008. Feedback from horizontal cells to rod photoreceptors in vertebrate retina. J. Neurosci. 28:5691–95
    [Google Scholar]
  82. Tishby N, Pereira FC, Bialek W. 2000. The information bottleneck method. arXiv:physics/0004057 [physics.data-an]
  83. Trenholm S, Schwab DJ, Balasubramanian V, Awatramani GB. 2013. Lag normalization in an electrically coupled neural network. Nat. Neurosci. 16:154–56
    [Google Scholar]
  84. Vinken K, Op de Beeck HP, Vogels R 2018. Face repetition probability does not affect repetition suppression in macaque inferotemporal cortex. J. Neurosci. 38:7492–504
    [Google Scholar]
  85. Vogels R. 2016. Sources of adaptation of inferior temporal cortical responses. Cortex 80:185–95
    [Google Scholar]
  86. Wacongne C, Changeux J-P, Dehaene S. 2012. A neuronal model of predictive coding accounting for the mismatch negativity. J. Neurosci. 32:3665–78
    [Google Scholar]
  87. Wark B, Fairhall A, Rieke F. 2009. Timescales of inference in visual adaptation. Neuron 61:750–61
    [Google Scholar]
  88. Weber AI, Fairhall AL. 2019. The role of adaptation in neural coding. Curr. Opin. Neurobiol. 58:135–40
    [Google Scholar]
  89. Weber AI, Krishnamurthy K, Fairhall AL. 2019. Coding principles in adaptation. Annu. Rev. Vis. Sci. 5:427–49
    [Google Scholar]
  90. Xiang JZ, Brown MW. 1998. Differential neuronal encoding of novelty, familiarity and recency in regions of the anterior temporal lobe. Neuropharmacology 37:657–76
    [Google Scholar]
  91. Zhou D, Cornblath EJ, Stiso J, Teich EG, Dworkin JD. 2020. Gender diversity statement and code notebook v1.0. Software https://zenodo.org/record/3672110#.YAHAV-hKhjV
    [Google Scholar]
  92. Zhou J, Benson NC, Kay KN, Winawer J. 2018. Compressive temporal summation in human visual cortex. J. Neurosci. 38:691–709
    [Google Scholar]
  93. Zurn P, Bassett DS, Rust NC. 2020. The Citation Diversity Statement: a practice of transparency, a way of life. Trends Cogn. Sci. 24:669–72
    [Google Scholar]
/content/journals/10.1146/annurev-vision-093019-112249
Loading
/content/journals/10.1146/annurev-vision-093019-112249
Loading

Data & Media loading...

  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error