Abstract
Recent neurophysiological and neuroanatomical studies suggest a close interaction between sensory and motor processes across the neocortex. Here, I propose that the neocortex implements active predictive coding (APC): each cortical area estimates both latent sensory states and actions (including potentially abstract actions internal to the cortex), and the cortex as a whole predicts the consequences of actions at multiple hierarchical levels. Feedback from higher areas modulates the dynamics of state and action networks in lower areas. I show how the same APC architecture can explain (1) how we recognize an object and its parts using eye movements, (2) why perception seems stable despite eye movements, (3) how we learn compositional representations, for example, part–whole hierarchies, (4) how complex actions can be planned using simpler actions, and (5) how we form episodic memories of sensory–motor experiences and learn abstract concepts such as a family tree. I postulate a mapping of the APC model to the laminar architecture of the cortex and suggest possible roles for cortico–cortical and cortico–subcortical pathways.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Rao, R. P. N. & Ballard, D. H. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat. Neurosci. 2, 79–87 (1999).
Keller, G. B. & Mrsic-Flogel, T. D. Predictive processing: a canonical cortical computation. Neuron 100, 424–435 (2018).
Friston, K. & Kiebel, S. Predictive coding under the free-energy principle. Philos. Trans. R. Soc. London B Biol. Sci. 364, 1211–1221 (2009).
Jiang, L. P. & Rao, R. P. N. Predictive coding theories of cortical function. Oxford Research Encyclopedia of Neuroscience https://doi.org/10.1093/acrefore/9780190264086.013.328 (Oxford Univ. Press, 2022).
Halpern, B. P. Tasting and smelling as active, exploratory sensory processes. Am. J. Otolaryngol. 4, 246–249 (1983).
Lederman, S. J. & Klatzky, R. L. Hand movements: a window into haptic object recognition. Cogn. Psychol. 19, 342–368 (1987).
Ahissar, E. & Assa, E. Perception as a closed-loop convergence process. eLife 5, e12830 (2016).
Friston, K. The free-energy principle: a unified brain theory? Nat. Rev. Neurosci. 11, 127–138 (2010).
Zatka-Haas, P., Steinmetz, N. A., Carandini, M. & Harris, K. D. Sensory coding and the causal impact of mouse cortex in a visual decision. eLife 10, e63163 (2021).
Steinmetz, N. A., Zatka-Haas, P., Carandini, M. & Harris, K. D. Distributed coding of choice, action and engagement across the mouse brain. Nature 576, 266–273 (2019).
Stringer, C. et al. Spontaneous behaviors drive multidimensional, brainwide activity. Science 364, eaav7893 (2019).
Talluri, B. C. et al. Activity in primate visual cortex is minimally driven by spontaneous movements. Nat. Neurosci. 26, 1953–1959 (2023).
Jordan, R. & Keller, G. B. Opposing influence of top–down and bottom–up input on excitatory layer 2/3 neurons in mouse primary visual cortex. Neuron 108, 1194–1206 (2020).
Nakamura, K. & Colby, C. L. Updating of the visual representation in monkey striate and extrastriate cortex during saccades. Proc. Natl Acad. Sci. USA 99, 4026–4031 (2002).
Duhamel, J. R., Colby, C. L. & Goldberg, M. E. The updating of the representation of visual space in parietal cortex by intended eye movements. Science 255, 90–92 (1992).
Umeno, M. M. & Goldberg, M. E. Spatial processing in the monkey frontal eye field. I. Predictive visual responses. J. Neurophysiol. 78, 1373–1383 (1997).
Wurtz, R. H., McAlonan, K., Cavanaugh, J. & Berman, R. A. Thalamic pathways for active vision. Trends Cogn. Sci. 15, 177–184 (2011).
Sherman, S. M. & Guillery, R. W. Functional Connections of Cortical Areas: A New View from the Thalamus (MIT, 2013).
Prasad, J., Carroll, B. & Sherman, S. Layer 5 corticofugal projections from diverse cortical areas: variations on a pattern of thalamic and extrathalamic targets. J. Neurosci. 40, 5785–5796 (2020).
Mountcastle, V. in the Mindful Brain (eds Edelman, G. & Mountcastle, V.) 7–50 (MIT, 1978).
Creutzfeldt, O. D. Generality of the functional structure of the neocortex. Naturwissenschaften 64, 507–517 (1977).
Mumford, D. On the computational architecture of the neocortex. II. The role of cortico–cortical loops. Biol. Cybern. 66, 241–251 (1992).
Hawkins, J. A Thousand Brains: A New Theory of Intelligence (Basic Books, 2021).
Douglas, R. J. & Martin, K. A. Neuronal circuits of the neocortex. Annu. Rev. Neurosci. 27, 419–451 (2004).
Harris, K. D. & Shepherd, G. M. The neocortical circuit: themes and variations. Nat. Neurosci. 18, 170–181 (2015).
Roe, A. W., Pallas, S. L., Kwon, Y. H. & Sur, M. Visual projections routed to the auditory pathway in ferrets: receptive fields of visual neurons in primary auditory cortex. J. Neurosci. 12, 3651–3664 (1992).
Felleman, D. & Essen, D. V. Distributed hierarchical processing in the primate cerebral cortex. Cereb. Cortex 1, 1–47 (1991).
Murray, J. D. et al. A hierarchy of intrinsic timescales across primate cortex. Nat. Neurosci. 17, 1661–1663 (2014).
Siegle, J. H. et al. Survey of spiking in the mouse visual system reveals functional hierarchy. Nature 592, 86–92 (2021).
Grafton, S. T. & de C. Hamilton, A. F. Evidence for a distributed hierarchy of action representation in the brain. Hum. Mov. Sci. 26, 590–616 (2007).
Gklezakos, D. C. & Rao, R. P. N. Active predictive coding networks: a neural solution to the problem of learning reference frames and part–whole hierarchies. Preprint at arxiv.org/abs/2201.08813 (2022).
Rao, R. P. N., Gklezakos, D. C. & Sathish, V. Active predictive coding: a unifying neural model for active perception, compositional learning, and hierarchical planning. Neural Comput. 36, 1–32 (2024).
Fisher, A. & Rao, R. P. N. Recursive neural programs: a differentiable framework for learning compositional part–whole hierarchies and image grammars. PNAS Nexus 2, pgad337 (2023).
Kasper, E., Larkman, A., Lübke, J. & Blakemore, C. Pyramidal neurons in layer 5 of the rat visual cortex. I. Correlation among cell morphology, intrinsic electrophysiological properties, and axon targets. J. Comp. Neurol. 339, 459–474 (1994).
Stebbings, K., Lesicko, A. & Llano, D. The auditory corticocollicular system: molecular and circuit-level considerations. Hear. Res. 314, 51–59 (2014).
Xiong, X. et al. Auditory cortex controls sound-driven innate defense behaviour through corticofugal projections to inferior colliculus. Nat. Commun. 6, 7224 (2015).
Frezel, N. et al. In-depth characterization of layer 5 output neurons of the primary somatosensory cortex innervating the mouse dorsal spinal cord. Cereb. Cortex Commun. 1, tgaa052 (2020).
Rathelot, J. A. & Strick, P. L. Subdivisions of primary motor cortex based on cortico–motoneuronal cells. Proc. Natl Acad. Sci. USA 106, 918–923 (2009).
Mao, T. et al. Long-range neuronal circuits underlying the interaction between sensory and motor cortex. Neuron 72, 111–123 (2011).
Hooks, B. M. et al. Organization of cortical and thalamic input to pyramidal neurons in mouse motor cortex. J. Neurosci. 33, 748–760 (2013).
Heindorf, M., Arber, S. & Keller, G. B. Mouse motor cortex coordinates the behavioral response to unpredicted sensory feedback. Neuron 99, 1040–1054 (2018).
Holey, B. E. & Schneider, D. M. Sensation and expectation are embedded in mouse motor cortical activity. Preprint at bioRxiv https://doi.org/10.1101/2023.09.13.557633 (2023).
Kim, E., Juavinett, A., Kyubwa, E., Jacobs, M. & Callaway, E. Three types of cortical layer 5 neurons that differ in brain-wide connectivity and function. Neuron 88, 1253–1267 (2015).
Sutton, R. S. & Barto, A. G. Reinforcement Learning: An Introduction 2nd edn http://incompleteideas.net/book/the-book-2nd.html (MIT Press, 2018).
Kaelbling, L. P., Littman, M. L. & Cassandra, A. R. Planning and acting in partially observable stochastic domains. Artif. Intell. 101, 99–134 (1998).
Rao, R. P. N. Decision making under uncertainty: a neural model based on partially observable Markov decision processes. Front. Comput. Neurosci. 4, 146 (2010).
von Helmholtz, H. Handbuch der Physiologischen Optik Vol. 3 (Voss, 1867).
Friston, K., Adams, R. A., Perrinet, L. & Breakspear, M. Perceptions as hypotheses: saccades as experiments. Front. Psychol. 3, 151 (2012).
Bastos, A. M. et al. Canonical microcircuits for predictive coding. Neuron 76, 695–711 (2012).
Shipp, S. Neural elements for predictive coding. Front. Psychol. 7, 1792 (2016).
Miura, S. & Scanziani, M. Distinguishing externally from saccade-induced motion in visual cortex. Nature 610, 135–142 (2022).
Keller, G. B., Bonhoeffer, T. & Hübener, M. Sensorimotor mismatch signals in primary visual cortex of the behaving mouse. Neuron 74, 809–815 (2012).
Bastos, A. M., Lundqvist, M., Waite, A. S., Kopell, N. & Miller, E. K. Layer and rhythm specificity for predictive routing. Proc. Natl Acad. Sci. USA 117, 31459–31469 (2020).
Leinweber, M., Ward, D. R., Sobczak, J. M., Attinger, A. & Keller, G. B. Sensorimotor circuit in mouse cortex for visual flow predictions. Neuron 95, 1420–1432 (2017).
Schneider, D. M., Sundararajan, J. & Mooney, R. A cortical filter that learns to suppress the acoustic consequences of movement. Nature 561, 391–395 (2018).
Markov, N. T. & Kennedy, H. The importance of being hierarchical. Curr. Opin. Neurobiol. 23, 187–194 (2013).
Lake, B. M., Ullman, T. D., Tenenbaum, J. B. & Gershman, S. J. Building machines that learn and think like people. Behav. Brain Sci. 40, e253 (2017).
Smolensky, P., McCoy, R. T., Fernandez, R., Goldrick, M. & Gao, J. Neurocompositional computing: from the central paradox of cognition to a new generation of AI systems. AI Mag. 43, 308–322 (2022).
Lewis, M., Purdy, S., Ahmad, S. & Hawkins, J. Locations in the neocortex: a theory of sensorimotor object recognition using cortical grid cells. Front. Neural Circuits 13, 22 (2019).
Olshausen, B. A. & Field, D. J. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381, 607–609 (1996).
Jiang, L. P. & Rao, R. P. N. Dynamic predictive coding: a model of hierarchical sequence learning and prediction in the neocortex. PLoS Comput. Biol. 20, e1011801 (2024).
Whittington, J. C. R. & Bogacz, R. An approximation of the error backpropagation algorithm in a predictive coding network with local Hebbian synaptic plasticity. Neural Comput. 29, 1229–1262 (2017).
Attias, H. Planning by probabilistic inference. In Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics (AISTATS 2003) (eds Bishop, C. M. & Frey, B. J.) 9–16 (PMLR, 2003).
Verma, D. & Rao, R. P. N. Planning and acting in uncertain environments using probabilistic inference. In 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems 2382–2387 (IEEE, 2006).
Botvinick, M. & Toussaint, M. Planning as inference. Trends Cogn. Sci. 16, 485–488 (2012).
Richards, A. Robust Constrained Model Predictive Control. PhD thesis, MIT (2004).
Botvinick, M. M., Niv, Y. & Barto, A. G. Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective. Cognition 113, 262–280 (2009).
Kahneman, D. Thinking, Fast and Slow (Farrar, Straus and Giroux, 2011).
Friston, K., Parr, T. & de Vries, B. The graphical brain: belief propagation and active inference. Netw. Neurosci. 1, 381–414 (2017).
O’Reilly, R. C., Russin, J. L., Zolfaghar, M. & Rohrlich, J. Deep predictive learning in neocortex and pulvinar. J. Cogn. Neurosci. 33, 1158–1196 (2021).
Willett, F. R. et al. Hand knob area of premotor cortex represents the whole body in a compositional way. Cell 181, 396–409 (2020).
Yang, G. R., Joglekar, M. R., Song, H. F., Newsome, W. T. & Wang, X.-J. Task representations in neural networks trained to perform many cognitive tasks. Nat. Neurosci. 22, 297–306 (2019).
Kurth-Nelson, Z. et al. Replay and compositional computation. Neuron 111, 454–469 (2023).
Mnih, V., Heess, N., Graves, A. & Kavukcuoglu, K. Recurrent models of visual attention. In Advances in Neural Information Processing Systems 27 (eds Ghahramani, Z. et al.) 2204–2212 (Curran Associates, 2014).
Makino, H. Arithmetic value representation for hierarchical behavior composition. Nat. Neurosci. 26, 140–149 (2023).
Hogendoorn, H. Perception in real-time: predicting the present, reconstructing the past. Trends Cogn. Sci. 26, 128–141 (2022).
Whittington, J. C. R. et al. The Tolman–Eichenbaum machine: unifying space and relational memory through generalization in the hippocampal formation. Cell 183, 1249–1263 (2020).
Hafner, D., Lee, K.-H., Fischer, I. & Abbeel, P. Deep hierarchical planning from pixels. In Advances in Neural Information Processing Systems 35 (eds Koyejo, S. et al.) 26091–26104 (Curran Associates, 2022).
Lee, T. S. & Mumford, D. Hierarchical Bayesian inference in the visual cortex. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 20, 1434–1448 (2003).
George, D. & Hawkins, J. Towards a mathematical theory of cortical micro-circuits. PLoS Comput. Biol. 5, e1000532 (2009).
Wolpert, D. M. & Miall, R. C. Forward models for physiological motor control. Neural Netw. 9, 1265–1279 (1996).
Mehta, M. R. Neuronal dynamics of predictive coding. Neuroscientist 7, 490–495 (2001).
Heeger, D. J. Theory of cortical function. Proc. Natl Acad. Sci. USA 114, 1773–1782 (2017).
Mackay, D. in Automata Studies (eds Shannon, C. E. & McCarthy, J.) 235–251 (Princeton Univ., 1956).
Albus, J. S. Brains, Behavior and Robotics (BYTE, 1981).
Scott, S. Optimal feedback control and the neural basis of volitional motor control. Nat. Rev. Neurosci. 5, 532–546 (2004).
Jordan, M. I. & Rumelhart, D. E. Forward models: supervised learning with a distal teacher. Cogn. Sci. 16, 307–354 (1992).
Kawato, M. Internal models for motor control and trajectory planning. Curr. Opin. Neurobiol. 9, 718–727 (1999).
Fetz, E. E. in Textbook of Physiology (eds Patton, H. D. et al.) 608–631 (Saunders, 1989).
Jones, E. G., Coulter, J. D. & Hendry, S. H. C. Intracortical connectivity of architectonic fields in the somatic sensory, motor and parietal cortex of monkeys. J. Comp. Neurol. 181, 291–347 (1978).
Adams, R., Shipp, S. & Friston, K. Predictions not commands: active inference in the motor system. Brain Struct. Funct. 218, 611–643 (2013).
Falchier, A., Clavagnier, S., Barone, P. & Kennedy, H. Anatomical evidence of multimodal integration in primate striate cortex. J. Neurosci. 22, 5749–5759 (2002).
Audette, N. J., Zhou, W., La Chioma, A. & Schneider, D. M. Precise movement-based predictions in the mouse auditory cortex. Curr. Biol. 32, 4925–4940 (2022).
Craik, K. J. W. The Nature of Explanation (Macmillan, 1943).
Churchland, M. M. et al. Neural population dynamics during reaching. Nature 487, 51–56 (2012).
Sussillo, D., Churchland, M. M., Kaufman, M. T. & Shenoy, K. V. A neural network that finds a naturalistic solution for the production of muscle activity. Nat. Neurosci. 18, 1025–1033 (2015).
Sauerbrei, B. A. et al. Cortical pattern generation during dexterous movement is input-driven. Nature 577, 386–391 (2020).
Chaudhuri, R., Knoblauch, K., Gariel, M.-A., Kennedy, H. & Wang, X.-J. A large-scale circuit mechanism for hierarchical dynamical processing in the primate cortex. Neuron 88, 419–431 (2015).
Salinas, E. & Sejnowski, T. J. Gain modulation in the central nervous system: where behavior, neurophysiology, and computation meet. Neuroscientist 7, 430–440 (2001).
Larkum, M. E., Senn, W. & Lüscher, H.-R. Top–down dendritic input increases the gain of layer 5 pyramidal neurons. Cereb. Cortex 14, 1059–1070 (2004).
Ferguson, K. A. & Cardin, J. A. Mechanisms underlying gain modulation in the cortex. Nat. Rev. Neurosci. 21, 80–92 (2020).
Stroud, J. P., Porter, M. A., Hennequin, G. & Vogels, T. P. Motor primitives in space and time via targeted gain modulation in cortical networks. Nat. Neurosci. 21, 1774–1783 (2018).
McAdams, C. J. & Maunsell, J. H. R. Effects of attention on orientation–tuning functions of single neurons in macaque cortical area V4. J. Neurosci. 19, 431–441 (1999).
Ha, D., Dai, A. M. & Le, Q. V. Hypernetworks. In 5th International Conference on Learning Representations (ICLR 2017) openreview.net/forum?id=rkpACe1lx (OpenReview.net, 2017).
Eliasmith, C. et al. A large-scale model of the functioning brain. Science 338, 1202–1205 (2012).
Galanti, T. & Wolf, L. On the modularity of hypernetworks. In Advances in Neural Information Processing Systems 33 (eds Larochelle, H. et al.) 10409–10419 (Curran Associates, 2020).
Tomov, M. S., Yagati, S., Kumar, A., Yang, W. & Gershman, S. J. Discovery of hierarchical representations for efficient planning. PLoS Comput. Biol. 16, e1007594 (2020).
Olson, C. R. Brain representation of object-centered space in monkeys and humans. Annu. Rev. Neurosci. 26, 331–354 (2003).
George, D. et al. Clone-structured graph representations enable flexible learning and vicarious evaluation of cognitive maps. Nat. Commun. 12, 2392 (2021).
Friston, K. J., Rosch, R., Parr, T., Price, C. & Bowman, H. Deep temporal models and active inference. Neurosci. Biobehav. Rev. 77, 388–402 (2017).
Acknowledgements
I thank A. Fisher, D. Gklezakos, P. Jiang, P. Rangarajan and V. Sathish for many discussions and the collaborative work cited in the text. I also thank K. Friston, C. Eliasmith and members of his laboratory, researchers at Numenta, S. Mirbagheri, N. Steinmetz and G. Burachas for discussions and feedback. This work was supported by National Science Foundation EFRI grant 2223495, National Institutes of Health grant 1UF1NS126485-01, the Defense Advanced Research Projects Agency under contract HR001120C0021, a UW + Amazon Science Hub grant, a Weill Neurohub Investigator grant, a Frameworks grant from the Templeton World Charity Foundation and a Cherng Jia and Elizabeth Yun Hwang Professorship. The opinions expressed in this publication are those of the author and do not necessarily reflect the views of the funders.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing interests
The author declares no competing interests.
Peer review
Peer review information
Nature Neuroscience thanks Karl Friston, Aleena Garner, and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Rao, R.P.N. A sensory–motor theory of the neocortex. Nat Neurosci 27, 1221–1235 (2024). https://doi.org/10.1038/s41593-024-01673-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41593-024-01673-9
