Abstract
Is perception of the whole based on perception of its parts? There is psychological1 and physiological2,3 evidence for parts-based representations in the brain, and certain computational theories of object recognition rely on such representations4,5. But little is known about how brains or computers might learn the parts of objects. Here we demonstrate an algorithm for non-negative matrix factorization that is able to learn parts of faces and semantic features of text. This is in contrast to other methods, such as principal components analysis and vector quantization, that learn holistic, not parts-based, representations. Non-negative matrix factorization is distinguished from the other methods by its use of non-negativity constraints. These constraints lead to a parts-based representation because they allow only additive, not subtractive, combinations. When non-negative matrix factorization is implemented as a neural network, parts-based representations emerge by virtue of two properties: the firing rates of neurons are never negative and synaptic strengths do not change sign.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 51 print issues and online access
$199.00 per year
only $3.90 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Palmer,S. E. Hierarchical structure in perceptual representation. Cogn. Psychol. 9, 441–474 ( 1977).
Wachsmuth,E., Oram,M. W. & Perrett, D. I. Recognition of objects and their component parts: responses of single units in the temporal cortex of the macaque. Cereb. Cortex 4, 509–522 (1994).
Logothetis,N. K. & Sheinberg,D. L. Visual object recognition. Annu. Rev. Neurosci. 19, 577 –621 (1996).
Biederman,I. Recognition-by-components: a theory of human image understanding. Psychol. Rev. 94, 115–147 (1987).
Ullman,S. High-Level Vision: Object Recognition and Visual Cognition (MIT Press, Cambridge, MA, 1996).
Turk,M. & Pentland,A. Eigenfaces for recognition. J. Cogn. Neurosci. 3, 71–86 (1991).
Field,D. J. What is the goal of sensory coding? Neural Comput. 6, 559–601 (1994).
Foldiak,P. & Young,M. Sparse coding in the primate cortex. The Handbook of Brain Theory and Neural Networks 895 –898 (MIT Press, Cambridge, MA, 1995 ).
Olshausen,B. A. & Field,D. J. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381, 607–609 ( 1996).
Lee,D. D. & Seung,H. S. Unsupervised learning by convex and conic coding. Adv. Neural Info. Proc. Syst. 9, 515–521 (1997).
Paatero,P. Least squares formulation of robust non-negative factor analysis. Chemometr. Intell. Lab. 37, 23–35 (1997).
Nakayama,K. & Shimojo,S. Experiencing and perceiving visual surfaces. Science 257, 1357– 1363 (1992).
Hinton,G. E., Dayan,P., Frey,B. J. & Neal,R. M. The “wake-sleep” algorithm for unsupervised neural networks. Science 268, 1158–1161 (1995).
Salton,G. & McGill,M. J. Introduction to Modern Information Retrieval (McGraw-Hill, New York, 1983).
Landauer,T. K. & Dumais,S. T. The latent semantic analysis theory of knowledge. Psychol. Rev. 104, 211–240 (1997).
Jutten,C. & Herault,J. Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture. Signal Proc. 24, 1–10 ( 1991).
Bell,A. J. & Sejnowski,T. J. An information maximization approach to blind separation and blind deconvolution. Neural Comput. 7, 1129–1159 ( 1995).
Bartlett,M. S., Lades,H. M. & Sejnowski, T. J. Independent component representations for face recognition. Proc. SPIE 3299, 528–539 (1998).
Shepp,L. A. & Vardi,Y. Maximum likelihood reconstruction for emission tomography. IEEE Trans. Med. Imaging. 2, 113–122 (1982).
Richardson,W. H. Bayesian-based iterative method of image restoration. J. Opt. Soc. Am. 62, 55–59 ( 1972).
Lucy,L. B. An iterative technique for the rectification of observed distributions. Astron. J. 74, 745–754 ( 1974).
Dempster,A. P., Laired,N. M. & Rubin, D. B. Maximum likelihood from incomplete data via the EM algorithm. J. Royal Stat. Soc. 39, 1– 38 (1977).
Saul,L. & Pereira,F. Proceedings of the Second Conference on Empirical Methods n Natural Language Processing (eds Cardie, C. & Weischedel, R.) 81–89 (Morgan Kaufmann, San Francisco, 1997).
Acknowledgements
We acknowledge the support of Bell Laboratories and MIT. C. Papageorgiou and T. Poggio provided us with the database of faces, and R. Sproat with the Grolier encyclopedia corpus. We thank L. Saul for convincing us of the advantages of EM-type algorithms. We have benefited from discussions with B. Anderson, K. Clarkson, R. Freund, L. Kaufman, E. Rietman, S. Roweis, N. Rubin, J. Tenenbaum, N. Tishby, M. Tsodyks, T. Tyson and M. Wright.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Lee, D., Seung, H. Learning the parts of objects by non-negative matrix factorization. Nature 401, 788–791 (1999). https://doi.org/10.1038/44565
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.1038/44565
This article is cited by
-
Abnormal interlimb coordination of motor developmental delay during infant crawling based on kinematic synergy analysis
BioMedical Engineering OnLine (2024)
-
Muscle synergies inherent in simulated hypogravity running reveal flexible but not unconstrained locomotor control
Scientific Reports (2024)
-
Quantifying changes in individual-specific template-based representations of center-of-mass dynamics during walking with ankle exoskeletons using Hybrid-SINDy
Scientific Reports (2024)
-
High-resolution tracking of unconfined zebrafish behavior reveals stimulatory and anxiolytic effects of psilocybin
Molecular Psychiatry (2024)
-
Learning low-rank latent mesoscale structures in networks
Nature Communications (2024)
Comments
By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.
