• Free to Read

Estimating mutual information

Alexander Kraskov, Harald Stögbauer, and Peter Grassberger
Phys. Rev. E 69, 066138 – Published 23 June 2004; Erratum Phys. Rev. E 83, 019903 (2011)
An article within the collection: PRE Milestones and the Physical Review E 25th Anniversary Milestones

Abstract

We present two classes of improved estimators for mutual information M(X,Y), from samples of random points distributed according to some joint probability density μ(x,y). In contrast to conventional estimators based on binnings, they are based on entropy estimates from k-nearest neighbor distances. This means that they are data efficient (with k=1 we resolve structures down to the smallest possible scales), adaptive (the resolution is higher where data are more numerous), and have minimal bias. Indeed, the bias of the underlying entropy estimates is mainly due to nonuniformity of the density at the smallest resolved scale, giving typically systematic errors which scale as functions of kN for N points. Numerically, we find that both families become exact for independent distributions, i.e. the estimator M̂(X,Y) vanishes (up to statistical fluctuations) if μ(x,y)=μ(x)μ(y). This holds for all tested marginal distributions and for all dimensions of x and y. In addition, we give estimators for redundancies between more than two random variables. We compare our algorithms in detail with existing algorithms. Finally, we demonstrate the usefulness of our estimators for assessing the actual independence of components obtained from independent component analysis (ICA), for improving ICA, and for estimating the reliability of blind source separation.

  • Figure
  • Figure
  • Figure
  • Figure
  • Figure
  • Figure
  • Figure
13 More
  • Received 28 May 2003

DOI:https://doi.org/10.1103/PhysRevE.69.066138

©2004 American Physical Society

Erratum

Erratum: Estimating mutual information [Phys. Rev. E 69, 066138 (2004)]

Alexander Kraskov, Harald Stögbauer, and Peter Grassberger
Phys. Rev. E 83, 019903 (2011)

Collections

This article appears in the following collections:

PRE Milestones

Physical Review E published its 50,000th paper in September 2015. To celebrate this, the journal presents a series of milestone papers that were published since its inception in 1993. This is an eclectic collection of papers that made significant contributions to their field, chosen by the editors. A new milestone will be added each week.

Physical Review E 25th Anniversary Milestones

The year 2018 marks the 25th anniversary of Physical Review E. To celebrate the journal’s rich legacy, during the upcoming year we highlight a series of papers that made important contributions to their field. These milestone articles were nominated by members of the Editorial Board of Physical Review E, in collaboration with the journal’s editors. The 25 milestone articles, including an article for each calendar year from 1993 through 2017 and spanning all major subject areas of the journal, will be unveiled in chronological order and will be featured on the journal website.

Authors & Affiliations

Alexander Kraskov, Harald Stögbauer, and Peter Grassberger

  • John-von-Neumann Institute for Computing, Forschungszentrum Jülich, D-52425 Jülich, Germany

Article Text

Click to Expand

References

Click to Expand
Issue

Vol. 69, Iss. 6 — June 2004

Reuse & Permissions
Author publication services for translation and copyediting assistance advertisement

Authorization Required


×
×

Images

×

Sign up to receive regular email alerts from Physical Review E

Log In

Cancel
×

Search


Article Lookup

Paste a citation or DOI

Enter a citation
×