A Tutorial for Information Theory in Neuroscience

The big-picture view of information theory

Probability distributions and initial analysis steps

Converting neuroscience data to probability distributions

If casting data in terms of probability distributions is necessary to information theory analyses, how do we convert raw data from neuroscience experiments into probability distributions? Indeed, this is a nontrivial problem that has the potential to dramatically affect the outcome of information theory analyses. This process (so-called state space reconstruction) is of vital importance (Wibral et al., 2014a).

Many neuroscience experiments involve multiple observations of the system/organism under study, via either multiple trials or a single recording of a system’s behavior through time (Fig. 1A). The data from the experiment can then be discretized (if necessary) into states s (Fig. 1B, see Data Binning for more information about binning and see Handling Continuous Data and Further Refinements for alternatives to binning). (Note, we are not referring to time binning at this point, but rather binning of multiple observations of some variables across trials or time.) Admittedly, understanding this binning procedure is one of the most difficult aspects of employing an information theory analysis. Part of the problem lies in the fact that this step is highly flexible and system specific. Suppose the data consist of individual electrode voltage values through time. In this case, we could bin the values into certain voltage ranges (e.g., less than –1 mV, –1 to 1 mV, and >1 mV). (We will discuss why/how to pick certain ranges below.) Then, we could refer to all voltage measurements below –1 mV as state 1, between –1 and 1 mV as state 2, and above 1 mV as state 3.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

General information theory analysis protocol. A, A neuroscience experiment or simulation is performed to gather environmental data (e.g., stimuli), physiologic data (e.g., voltage recordings), and/or behavioral data (e.g., animal location). B, If necessary, the data are then discretized (see Data Binning). Some types of data (e.g., spike data) do not require discretization. In this example, two sets of data were produced, but analysis of any number of data sets is possible. C, The discretized data are then converted to probability distributions by first counting the number of times each unique set of states was observed. In the case of single trial data (gray tables), the joint states for all of the data are counted to estimate the probability distribution. In the case of trial-based data (green and orange tables), the joint states are counted for all data at certain time bins across trials. D, The desired information theory measure is applied to the probability distribution.

At the most basic level, the probability of a state is estimated as the total number of observations of that state divided by the total number of observations for all states (Fig. 1C). For instance, if we note s as the state of the variable being recorded (e.g., electrode voltage), as the number of experimental observations of state s (a so-called frequency distribution; e.g., the number of time bins where the voltage was in a certain range), and as the total number of experimental observations, then the probability for the state s would be estimated by (3)

So, if we recorded the voltage for 10,000 time bins and 1298 of those time bins produced a voltage less than –1 mV, we would find . It should be noted that this is a form of maximum likelihood estimation (Myung, 2003), in that we assume that the data observed represent the most likely outcome of the underlying probability distribution. Once the probability distribution is estimated, the appropriate information theory measure can be applied (Fig. 1D, see sections below on various information theory measures).

Depending on the information measure to be used, the states could consist of a single data type (e.g., voltage on one electrode) or multiple data types (e.g., voltage recordings from multiple electrodes). If multiple data types are used, the state is then a joint state for all the variables (e.g., voltage recording 1 is less than –1 mV and voltage recording 2 is >1 mV). If the analysis utilizes single-trial data, the states could be experimentally recorded values through time, with each time bin being a unique state observation. Such an analysis will produce a time-averaged information value throughout the recording (see discussion of stationarity concerns below). If the analysis utilizes trial-based data, the states could be experimentally recorded values at a given time point across trials (e.g., 100 ms after the stimulus), with each trial being a unique state observation instead of each time bin throughout a recording being a state observation. Such an analysis will produce an instantaneous information value at a given time point in the trial (Lizier et al., 2008; Wibral et al., 2014a for other interpretations of instantaneous information values). If multiple values are being recorded through time, delays can be introduced in the time order of the states (e.g., stimulus at a given time in a trial and neural signal 200 ms later in a trial).

It is important to note that the state being observed is very flexible. For instance, with neurons the distinction between rate coding and spike timing coding has frequently been discussed (Victor and Purpura, 1996; Borst and Theunissen, 1999; Panzeri and Schultz, 2001; Van Rullen and Thorpe, 2001; Stanley, 2013). In this case, the state could be the number of spikes in a given time bin when a rate coding scheme is being investigated, or the state could be a specific pattern of spikes when a spike timing coding scheme is being investigated.

Enough observations must be performed to adequately sample the space of possible joint states. As a bare minimum, the number of observations must be greater than the number of possible joint states, though more observations are usually necessary to perform an information theory analysis. To the best of our knowledge, there are no agreed-on standards for the number of observations necessary to adequately estimate a joint probability distribution, though informal discussions with other researchers suggest >10 observations per possible state is ideal. Note that the use of significance testing via surrogate data (see Significance Testing) can minimize the occurrence of type 1 errors (i.e., reporting a significant information theory result when none is actually present). However, small data sets will increase the likelihood of type 2 errors (i.e., failing to report a significant information theory result when one is actually present) from surrogate data significance testing and produce bias (see Bias in Entropy and Mutual Information). The number of observations necessary to estimate a probability distribution has been explored to some extent in the literature (Ramos and Macau, 2017), but a great deal of attention has been paid to other methods to assess bias and estimate probability distributions (see Handling Continuous Data and Further Refinements).

In addition to concerns related to the number of observations, experimenters must also consider a fundamental assumption of this estimation method that each observation is produced from the same underlying probability distribution. This assumption is frequently referred to as “stationarity.” In other words, we must assume that each observation contributes to a picture of the same probability distribution (which we cannot directly access). If the underlying probability distribution is changing through our observations, the method outlined in Eqn. 3 will not produce a valid estimate of the probability distribution. Thus, the experiment must be designed in such a way that stationarity can be assumed. This can be especially important in single-trial data analysis, where the underlying probability distribution may be suspected to change throughout a recording. For instance, research should consider whether animal behavior is changing over observations (e.g., is the animal becoming satiated?) or whether neural behavior is changing (e.g., are firing rates changing?).

Data binning

A further complication regarding estimation of probability distributions is the fact that data from neuroscience experiments can be continuous (e.g., action potential times, voltage, calcium signal, BOLD signal, movements, positions, principle components, etc.) or discrete (e.g., action potential magnitudes, some types of stimuli, some types of behaviors, animal strain, etc.). Data that are naturally discrete, such as binary stimuli (e.g., light on versus light off), are relatively easy to use with the simple estimation method in (Eqn. 3). Other data can have discrete and nondiscrete features, such as action potentials whose magnitudes are discrete (spike versus no spike) but whose timing is continuous (though typically binned in time by the recording resolution of the experimental system). However, continuous data never precisely repeat through observations, so the number of observations for each state would trivially be 1. In other words, a particular voltage observation could be 5.3476 mV, and that voltage value will not be repeated throughout an experiment. Thus, counting the number of observations at a specific voltage value will not provide an estimate of the underlying probability distribution. (However, repeated observations can provide an estimate of the density of a continuous probability distribution; see Handling Continuous Data and Further Refinements.)

How should we handle continuous data? The primary solution to this problem that we will discuss in this article is to convert the continuous data to discrete data via some type of binning or discretization procedure. [See Handling Continuous Data and Further Refinements for a discussion of other methods for handling continuous data and Daw et al. (2003) for a general review of discretization in data analysis]. Once this binning procedure is performed, the continuous data have been made discrete, and the probability distribution estimation method described in Eqn. 3 can be applied. We will discuss two general procedures, all of which are implemented in the information theory toolbox (see Software). Note that it is frequently necessary to look at the data to see which discretization method will be most appropriate for the analysis and to set discretization parameters. For instance, knowledge of a neuron’s firing rate can help influence decisions about time bin size and the ranges of spike counts per bin (e.g., low-firing-rate neurons need larger time bins and finer spike count resolution, while high-firing-rate neurons need smaller time bins and less spike count resolution to adequately capture the dynamic range of the spiking behavior). In all cases, users should avoid “parameter fishing” to find binning methods that yield expected results (see Significance Testing.).

The first binning procedure we will discuss simply involves dividing the total range of the data (i.e., minimum value observed to maximum value observed) into number of equal-width bins (Fig. 2A,B). Note that this method preserves some large-scale patterns in the continuous distribution of the data, though that resolution is dependent on the choice of , which is a parameter of the analysis. We generally refer to this method as the “uniform width” binning method.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

Example data discretization. A, 200 example data points were randomly generated (vertical black lines represent individual data points, black plot represents a fine-resolution histogram). The data were discretized into four uniform width bins or states (top colored regions) or four uniform count bins or states (bottom colored regions). B,C, The number of data points in each bin divided by the total number of data points was then used as the probability for each bin (state). Uniform width bins (B) can preserve general data distribution features (e.g., two peaks in this case), but produce some bins with low probabilities. Uniform count bins (C) produce uniform probability distributions, which have certain information theory advantages, but do not preserve general data distribution features.

The second binning procedure we will discuss uses a similar method, except the data are binned into equal-count bins (Fig. 2A,C), typically referred to herein as “uniform count” binning method. Importantly, this method does not preserve large-scale structure in continuous data distributions, but this method is especially powerful when examining relationships between variables because it maximizes the available information signal between variables (see Entropy and Mutual Information). Furthermore, this method can allow for the use of a single null model for the analysis of multiple variables, which can significantly decrease computation time in significance testing (see Significance Testing.). Finally, this method is not impacted by the range of the data or the various scales in the data. With equal width binning, an outlier data point may dramatically affect the binned structure of the data by skewing the bins. For these reasons, we used the uniform count method throughout the demonstrations presented herein, though the Neuroscience Information Theory Toolbox contains numerous demonstrations of other binning procedures (see Software; Extended Data).

Extended Data

Download Extended Data, ZIP file

Once one of these binning methods is chosen and applied to the data, the probability distribution can be estimated via the method described in Eqn. 3, where each bin corresponds to a unique state.

Entropy

Entropy is the fundamental information theory quantity. It measures the uncertainty contained in a variable. Because information theory conceptualizes information as a reduction in uncertainty, it is necessary to quantify uncertainty before information. In other words, we must understand uncertainty before we can understand information.

The entropy of a discrete random variable (call this variable X with individual states x ) is (Eqn. 4) (Shannon, 1948; Cover and Thomas, 2006) (4)

Note that refers to all of the possible states x can take.

As a first step in understanding entropy, it is useful to consider the simple fair coin example from above. In that case, x could take two possible states (heads and tails), and the likelihood for both outcomes is 0.5. Therefore, if we refer to this coin as , then we have (Eqn. 5): (5) (6)

Therefore, the uncertainty of a fair coin is 1 bit. Three other example systems with four possible states instead of two are shown in Fig. 3. Note that for systems with probability distributions that are more concentrated, the entropy is lower (Model 1 in Fig. 3), while for systems with evenly spread probability distributions, the entropy is higher (Model 3 in Fig. 3). If a variable is likely to be in one state (i.e., concentrated probability distribution), it has low uncertainty. Conversely, if a variable is equally likely to be in many different states (i.e., dispersed probability distribution), it has high uncertainty. This intuitively agrees with our definition of uncertainty. Note that the uniform counts binning procedure (see Data Binning) will produce a uniform probability distribution, which will maximize entropy (Cover and Thomas, 2006). Furthermore, a variable that is perfectly concentrated (i.e., it has only one possible state and that state has a likelihood of 1) will produce an entropy of 0. This makes sense because such a variable has no uncertainty. Finally, the entropy cannot be negative. This is because the likelihoods of the individual states cannot be larger than 1, so the argument of the logarithm cannot be <1, so the logarithm will always be positive.

• Download figure
• Open in new tab
• Download powerpoint

Figure 3.

Example entropy calculations. A, Example probability distributions for three models (red, blue, and green); B, their associated entropy values. Model 1 was most likely to be in state 1, so it had low entropy. Model 3 was equally likely to be in all four states, so it had maximum entropy. Uniform count binning (see Data Binning) will produce equally likely states and maximize entropy, similar to Model 3.

As is typically done in information theory, we use logarithms with base 2 in our entropy calculations (Eqn. 4), but we could have chosen any other base. The choice of base 2 produces entropy values in units of bits. Other base values produce different units for entropy (e.g., using the natural logarithm produces units of nats). Throughout this paper, we will use units of bits because they provide two useful conceptual connections. First, computer memory is expressed in units of bytes, which are related to bits via 1 byte = 8 bits. Second, as we discussed above, we can think of bits as yes/no questions. When we conceptualize bits this way, we can think of the entropy as the average number of yes/no questions necessary to determine the state of the variable. Returning to the fair coin example, we can see that one yes/no question (e.g., “Is the coin in the heads state?”) can always allow us to determine the state of the coin, which is why the result of our entropy calculation in (Eqn. 6) yielded 1 bit of entropy.

Just as probability distributions can have more than one variable in a joint probability distribution, the joint entropy can be calculated for systems with more than one variable. The joint entropy of two discrete random variables (variable X with individual states x , variable Y with individual states y ) is given by (Eqn. 7) (Cover and Thomas, 2006) (7)

For the system of two independent fair coins from above (likelihood of each combination of heads and tails is 0.25), we obtain (8)

This result agrees with our interpretation of entropy as the average number of yes/no questions necessary to determine the state. We found from Eqn. 6 that we needed one question to determine the state of one coin. Because the coins are independent, we should need two yes/no questions to determine the state of two coins, which is exactly what we find in Eqn. 8. Indeed, in general, when the X and Y variable are independent (i.e., ), the joint entropy of the two variables is simply the sum of the individual entropies (recall ) (Eqn. 9): (9)

The final entropy quantity of interest is the conditional entropy, which quantifies the average uncertainty in a variable given the state of another variable. The conditional entropy of two discrete random variables (variable X with individual states x given variable Y with individual states y ) is given by (Eqn. 10) (Cover and Thomas, 2006) (10)

As an example for the joint entropy, we will again turn to the system of two independent fair coins from above. In this case, the conditional likelihood of each combination of heads and tails is 0.5, because the likelihood for the first coin to be heads is always 0.5 regardless of the state of the second coin. Thus, we obtain (Eqn. 11) (11)

Because the coins are independent, the conditional entropy of the first coin given the second coin should be the same as the entropy of the first coin alone, which is what we found in Eqn. 6.

As a final example of entropy, consider the system of dependent coins described in Tables 1 and 2. In this case, the joint entropy is found to be (Eqn. 12) (12)

Because the coins are dependent, their joint entropy (1.73 bits) is less than was found for two independent coins (2 bits; Eqn. 8). In other words, for these dependent coins, there is less uncertainty about the state of the system because it is more likely to be in states where the coins match than states where they do not match. The conditional entropy for each coin in this system is identical and is found to be (Eqn. 13): (13)

While each coin in isolation produces heads and tails with equal likelihood and therefore has an entropy of 1 bit (Eqn. 6), the uncertainty in the state of each coin is reduced when the other is known. Therefore, the conditional entropy of each coin is <1 bit.

In all cases, not just systems with independent variables, the following intuitive relationship between entropy, joint entropy, and conditional entropy can be shown (Cover and Thomas, 2006): (14)

In words, (Eqn. 14) says that the joint entropy of X and Y is equal to the entropy of X along, plus the remaining entropy of Y when X is known.

While the formula given in Eqn. 4 may seem unusual, it actually has several important properties that motivated its creation (Shannon, 1948; Cover and Thomas, 2006). First, . In other words, the entropy can never be negative. This is very helpful because it is unclear how we would interpret negative uncertainty. Second, systems with one absolutely certain state have no entropy. This makes sense because a system that is always in one state cannot provide information, nor can another variable provide information about it. Finally, the entropy of two independent variables is simply the sum of their individual entropies (Eqn. 9) (i.e., the entropies are additive).

Mutual information

Information theory conceptualizes information as a reduction in uncertainty in one variable when another variable is known. In other words, if learning the state of one variable reduces our uncertainty in another variable on average, then the first variable provides information about the second variable. Importantly, we are now able to quantify uncertainty using entropy, so we can quantify this reduction in uncertainty and, therefore, information. While this notion of information is conceptually similar to correlation measures like explained variance, it is important to note that the definition of information developed in information theory possesses several distinct advantages (see Why use information theory?, see Fig. 5 below).

Recall that the conditional entropy (Eqn. 10) expresses the entropy that remains in X given knowledge about Y . Thus, the total entropy of X must be equal to the entropy that remains in X after Y is learned plus the information provided by Y about X . Therefore, (15)

Note that information will also be measured in bits, because the units of entropy are bits. Because entropy is always positive, it follows from Eqn. 15 that . Therefore, it is frequently advantageous in analyses of real data to discretize data using a uniform count method to maximize each variable’s entropy and thereby maximize the available information signal (see Data Binning).

The relatively simple expression in Eqn. 15 can be rearranged and rewritten using probability distributions (Eqns. 4 and 10) to provide an expression for information, or as it is more generally referred to in the literature, the “mutual information” (Cover and Thomas, 2006): (16)

In Fig. 4, we present three example systems with their corresponding mutual information values. In model 1, knowledge of either variable does not provide information about the other variable, so the mutual information is zero. For instance, if X is known to be in state 1, Y is equally likely to be in states 1 and 2. In model 2, some information about either variable is provided by the other, so the mutual information is nonzero. In model 3, each variable perfectly predicts the other, so the mutual information is maximized at 1 bit. In this case, 1 bit is the maximum mutual information because each variable has an entropy of 1 bit.

• Download figure
• Open in new tab
• Download powerpoint

Figure 4.

Example mutual information calculations. A, Example probability distributions for three models (red, blue, and green); B, their associated mutual information values. In model 1, the X and Y variables were independent, so their mutual information was zero. In model 2, knowledge of X or Y reduces our uncertainty in the state of the other variable to some extent, so nonzero mutual information was observed. In model 3, X and Y are identical, so their mutual information was maximal.

In addition to conceptualizing information as a reduction in uncertainty, another interpretation of information is provided by the sum expression in Eqn. 16. Recall that when two variables are independent . Thus, in Eqn. 16, for independent variables, the argument of the logarithm becomes one for all states, which produces an information of zero. This agrees with intuition, because independent variables cannot provide information about each other. In this way, the mutual information is viewed as the Kullback–Leibler distance between the true joint distribution and the joint distribution under the assumption the data are independent (Cover and Thomas, 2006).

Note that the sum term in Eqn. 16 is symmetric in X and Y , which implies that (this symmetry can also be noted in an alternative expression for the mutual information: ; Cover and Thomas, 2006). In other words, the information Y provides about X is equal to the information X provides about Y . For instance, in the example where I tell you the state of a flipped coin from above, my message contained information about the state of the coin, but the coin provides the same amount of information about my message. Because of this symmetry, from Eqn. 16 is most commonly referred to as the “mutual information.” We will employ this nomenclature throughout the remainder of this article to maintain consistency with the established literature.

To demonstrate the differences between linear analyses and information theory using mutual information, we created three model systems: one with linear interactions and two with nonlinear interactions (Fig. 5). A linear analysis method like correlation easily detects a significant correlation coefficient among linearly related variables but does not detect a significant interaction among two data sets with nonlinear interactions. Conversely, mutual information is able to detect a significant interaction in all three cases. While this example clearly demonstrates information theory’s ability to detect nonlinear interactions, it is important to note that the resulting mutual information values do not produce a model that describes the relationship between the variables (see What can information theory tell you? and Model Building).

• Download figure
• Open in new tab
• Download powerpoint

Figure 5.

Example of linear versus nonlinear analysis methods. A, Example data for three models (red, blue, and green) with linear (red) and nonlinear (blue and green) interactions; B, the associated correlation coefficient and mutual information (MI) values for all three models (star: p < 0.05, correlation coefficient and p-value calculated via MATLAB, mutual information and p-value calculated via the Neuroscience Information Theory Toolbox; see Data Binning and Significance Testing, 4 bins and 1000 null data sets).

Now that we have an expression for mutual information between two variables, it is natural to expand to systems of three or more variables. The most straightforward method for measuring the information between three variables is to use mutual information between two variables, but make one of the variables a joint (or “vector valued”) variable of two variables. When two variables are combined in this way, we consider each unique combination of states for the two joined variables as a unique state for the joint set. For instance, a joint variable consisting of two coins has four states: each possible combination of heads and tails. The mutual information between a joint variable constructed from two X variables ( and ) and another variable Y can be expressed using (Eqn. 17) (17)

Using this method, we can calculate the mutual information between one variable and two other variables considered together. This type of analysis can be very helpful in neuroscience applications. For instance, we might ask how much information two neurons provide about a stimulus or behavior together instead of individually. Below, we will explore other methods for quantifying the information between more than two variables.

Mutual information can be further expanded by considering the mutual information between two variables conditioned on a third variable. This measure is referred to as conditional mutual information and is given by (Eqn. 18) (18)

The conditional mutual information allows us to examine interactions between two variables while taking into account the effects of a third variable. For instance, suppose we had three magic coins that always produced identical results. In this case, the mutual information between any pair of coins would be one bit. However, the conditional mutual information between any pair given the third coin would be zero bits. This is because the information each coin provides about the other can be explained or provided by the third coin. Another example illustrates that the inclusion of a third variable can increase the information between two variables. Suppose we had two independent, unbiased coins and a third magic coin that always produced a heads when either the first coin or the second coin produced a heads and tails only when both coins produced tails (i.e., an OR operation). In that case, the mutual information between the first two coins would be 0, but the conditional mutual information between them conditioned on the third coin would be 0.19 bits. Thus, the third coin creates conditional dependence between the otherwise independent coins. Conditional mutual information will be especially helpful when examining causal relations.

Transfer entropy

As we discussed above, the probability distributions used in information theory calculations can be produced from many different types of data, possibly with different temporal relations. When a certain temporal ordering is used with conditional mutual information, an information theory measure called “transfer entropy” is produced. Transfer entropy is given by (Schreiber, 2000) (19)

Transfer entropy measures the information about the future state of a variable ( ) provided by another variable in the past ( ) given the information provided by the past state of the variable ( ). Once these temporal relationships are defined, transfer entropy can be interpreted as a better measure of causal influence from X to Y than merely the mutual information between and , because transfer entropy measures the changes caused in Y from X that are not accounted for by the history of Y alone. However, it is possible for no causal interaction to exist from to , but yet to still observe a nonzero transfer entropy result. For instance, even when considering , it is possible that some third unmeasured variable controls and , rendering the interaction from to noncausal.

As an example of transfer entropy, consider simultaneously measuring the firing rate through time of two neurons, X and Y . Assume that X sends an inhibitory connection to Y such that when X fires, Y stops firing. Knowing the past state of X allows you to predict the future of Y better than predicting the future of Y with only the past of Y . This type of interaction would result in increased transfer entropy from X to Y .

Fundamentally, transfer entropy is simply conditional mutual information with certain assumptions about temporal order and variable source, which allows it to serve as a measure of causal influence. Due to the widespread interest in neural connectivity (Bullmore and Sporns, 2009; Friston, 2011), transfer entropy has been widely used in the literature (for example, Honey et al., 2007; Lizier et al., 2008; Ito et al., 2011; Vicente et al., 2011; Timme et al., 2014b, 2016; Wibral et al., 2014b; Nigam et al., 2016; Bossomaier et al., 2016). Numerous methods have been employed to define past and future state (Staniek and Lehnertz, 2008; Ito et al., 2011; Wibral et al., 2013; Timme et al., 2014b). These methods allow for the exploration of interactions over certain time scales, the search for interactions with set delays (e.g., synaptic connectivity), or interactions involving patterns of activity. An interesting alternative measure called the directed information has also been used in the literature (Quinn et al., 2011).

In Fig. 6, we present four example single-trial systems (neuron spike trains) with example data through time and their resulting transfer entropy values. For the sake of simplicity in this introduction, we will assume the past state occurs one time bin before the future state. Models 1 through 3 possessed increasing interaction strength between the state of X at a given time with the state of Y one time step later. As expected, these models produced increasing transfer entropy values. Model 4 possessed a strong interaction between the state of X at a given time with the state of Y one time step later, but that interaction could be explained with the past state of Y , resulting in zero transfer entropy. This demonstrates the conditional aspect of transfer entropy.

• Download figure
• Open in new tab
• Download powerpoint

Figure 6.

Example transfer entropy calculations. A, Example model spike trains (color bands: spikes); B, their associated transfer entropy values. Model 1 contained independent neurons, so it produced zero transfer entropy. Models 2 and 3 contained interactions from neuron X to Y. In model 3, neuron X’s state precisely determined neuron Y’s state one time step in the future, which produced maximal transfer entropy. In model 4, neuron X’s state precisely determines neuron Y’s state, but the past of neuron Y also determines its future, so it produced zero transfer entropy.

Partial information decomposition

As we mentioned above, we can move beyond pairs of variables to consider information between groups of variables. The simplest such extension is to consider the mutual information between two variables taken together ( and ) and a third variable Y via Eqn. 17 (as a more advanced alternative, see (Pica et al. (2017)) for recent work on treating all variables equally). While this step alone is helpful in many circumstances, it is possible to decompose this mutual information into several other useful and intuitive components. For instance, we might ask what portion of is provided redundantly by both and (what information overlap about Y exists between and ), or uniquely by or alone (what information does provide about Y that does not and vice versa), or synergistically by both and together (what bonus information do and provide about Y when both are known simultaneously). This dissection is generally referred to as the partial information decomposition (Williams and Beer, 2010). Quantifying these relationships can provide a great deal of insight into how a system functions. If all of the information in is provided redundantly, then we know and are doing the same thing, at least in an information theoretic sense. If all of the information in is provided uniquely by , then we know is not providing information about Y . If all of the information in is provided synergistically, then we know and are somehow working together or engaged in some type of complex interaction with Y .

Note that we have not provided any explicit mathematical definition for the synergy, redundancy, and unique information concepts that we are invoking. Rather, we are relying on intuition regarding what “synergy” means in the context of information. The following equations express the intuitive relationships between the relevant mutual information terms and the synergy S , redundancy R , and unique information and components: (20) (21) (22)In (Eqn. 20), we take the total information provided by and together to be equal to the four underlying components (synergy, redundancy, and unique information terms). In Eqn. 21, we take the information provided by just to be equal to the redundancy and the unique information provided by . We include only these two terms because the unique information from is not provided by and because the synergy is not provided by alone.

While the mutual information expressions in Eqns. 20, 21, 22 can be calculated easily using Eqns. 16–17), it is not possible to derive mathematical expressions for the synergy, redundancy, and unique information without further measures. However, notice that if we had access to a measure for redundancy or unique information, the other components could be found easily via basic algebra.

Several researchers have put forward candidate measures for redundancy or unique information to address this problem (Williams and Beer, 2010; Harder et al., 2013; Bertschinger et al., 2014; Griffith et al., 2014; Ince, 2017; Finn and Lizier, 2018; see Quax et al., 2017 for an alternative information theory approach to measuring synergy). In fact, this area of information theory is currently undergoing rapid development, with numerous previous methods for analyzing synergy and redundancy in neuroscience contexts (Brenner et al., 2000; Schneidman et al., 2003a,b) being improved on. We feel strongly that it is beyond the scope of this introductory tutorial to thoroughly review all of these newly introduced measures for redundancy and unique information. Doing so properly would require an entire article. However, we also feel strongly that some of these new methods should be included in this tutorial because they represent new powerful tools for the analyses of neural data that will be of interest to readers and because related concepts have been used in neuroscience for many years.

In an attempt to thread this pedagogical needle, we have decided to focus on the first measure introduced: the minimum information (Williams and Beer, 2010). The minimum information possesses several advantages over other candidate measures in that it is relatively straightforward, it can be expanded to any number of X variables (in principle, though see Ince (2017) as an exception), and it does not require numerical approximation, which is especially relevant when analyzing large amounts of data. As described by Williams and Beer (2010), the minimum information ( ) can be interpreted as a measure of redundancy. It is given by (Eqn. 23) (23)

Note that the minimum information is very similar to the mutual information as expressed in Eqn. 16, except that there is a minimum operation over and . Thus, the minimum information measures the smallest overlap in information provided by both and about each state of Y individually. We then interpret this smallest overlap as the redundancy portion because it is the amount of information provided by both and individually. For instance, if provided 0.2 bits of information about a state of Y , but provided 0.3 bits for that state of Y , then only 0.2 bits of information would be redundantly provided by both and individually. Once we equate the minimum information with the redundancy (i.e., set ), the remainder of the decomposition terms can be found easily via (Eqns. 20, 21, 22).

Our discussion of the partial information decomposition has so far ignored time ordering. However, it is frequently useful in neuroscience applications to consider a time-ordered structure where information moves from and to Y (converging) or from Y to and (diverging) (Fig. 7A). In the converging case, we can think of Y as processing information, computing, or jointly encoding information from and . This manner of conceptualizing the partial information decomposition could be useful in experiments where two types of stimuli are presented to an animal simultaneously while a neural variable is recorded, or when two presynaptic neurons are simultaneously recorded with a shared postsynaptic target neuron. This converging framework has also been combined with transfer entropy to include the history of the receiver variable (Williams and Beer, 2011). In the diverging case, we can think of and as representing or encoding different features of Y . A common neuroscience experiment that could use this framework would include a stimulus presented to an animal while simultaneously recording two neural variables.

• Download figure
• Open in new tab
• Download powerpoint

Figure 7.

Partial information interpretations and example systems. A, Though the partial information decomposition does not require explicit time ordering, it is frequently helpful to apply converging or diverging ordering to the interactions. B, Example of purely redundant systems. The X variables provided the same amount of information about each state of Y . C, Example purely synergistic systems. The X variables alone provided no information about Y , though they did together via a nonlinear operation (Op). D, Example purely unique systems. In the converging example, only provided information about Y . In the diverging example, each X variable provided information about different states of Y . The joint probability distributions for these systems are listed as extended data in Fig. 7-1.

While we intuitively motivated the definitions of redundancy, synergy, and unique information above, it is often very helpful to consider specific examples (see below for several neuroscience-specific examples of redundancy, synergy, and unique information). For redundant systems, we present converging and diverging examples in Fig. 7B. In the converging case, and behave identically in both cases, and the states of and precisely determine the state of Y . In the diverging case, the same output from Y is always sent to both and . In both of these cases, and each provide information about Y , but it is the same amount of information about the same states of Y , so the information is redundant.

Example synergistic systems are shown in Fig. 7C. In the converging case, Y performs a nonlinear operation (Op) on and such that when and match, Y is one state, but when and do not match, Y is in another state. In the diverging case, the state of Y determines if and will be in the same state. In both of these cases, it is necessary to know the states of and together to gain information about Y . Neither nor provide information about Y alone, but together they do. Thus, the information must result from and working together, which makes the interactions synergistic.

Finally, example systems with unique information are shown in Fig. 7D. In the converging case, only determines the state of Y . In fact, provides no information about Y . In the diverging case, and encode non-overlapping states of Y . Therefore, the information from and each is unique, although they both provide the same amount of information individually.

We wish to emphasize that our use of the minimum information is not a declaration that it is the superior measure. While the minimum information possesses distinct advantages over other methods, it has been shown that it measures the redundant amount of information as opposed to the redundant content, which several authors find counterintuitive (Harder et al., 2013; Bertschinger et al., 2014; Griffith et al., 2014; Ince, 2017). For instance, consider an example system with the joint probability distribution given by Table 3. This system produces a redundancy of 1 bit and a synergy of 1 bit when analyzed with . However, the and variables provide information about different combinations of states of Y . differentiates between and , while differentiates between and . Thus, we can see that and provide the same amount of information (1 bit) about the same individual states of Y , so finds a redundancy of 1 bit, despite the difference in the content in the information. We feel this is an important feature of that readers should keep in mind, not an indication that produces invalid results. If you find this interpretation lacking, we recommend you explore the other methods currently available that seek to improve on (Harder et al., 2013; Bertschinger et al., 2014; Griffith et al., 2014; Ince, 2017; Finn and Lizier, 2018).

In addition to the three-variable formulation discussed above, the partial information decomposition can be expanded to include additional X variables. The mathematics involved in this expansion is somewhat complex (Williams and Beer, 2010), so we will forgo discussing it here. The partial information decomposition has also been adapted to transfer entropy measures between three variables (Williams and Beer, 2011), as well as measures of information transfer and information gain (Beer and Williams, 2014). Information transfer is conceptually similar to transfer entropy, where information is moving from some past state to a future state, except that information transfer measures information moving from one variable to another variable about a third variable. Information gain measures the information a variable gains in time about another variable. The information gained by Y through time about X is given by (Eqn. 24) (24)

The information transferred from Y to Z about X is given by (Eqn. 25) (25)

Note that the precise delays and time bin structure for past and future states must be selected, similar to transfer entropy. Because these measures represent more advanced techniques, we will provide only the expressions here and direct the interested reader to further examples in the literature (Beer and Williams, 2014). An alternative method for measuring information transmission about some other variable can be found in Ince et al. (2015).

Bias in entropy and mutual information

Several researchers have previously discussed biases associated with different methods for estimating information values from continuous data and/or small data sets (Treves and Panzeri, 1995; Wolpert and Wolf, 1995; Panzeri and Treves, 1996; Paninski, 2003; Nemenman et al., 2004; Panzeri et al., 2007; Bonachela et al., 2008), noting that limited data tend to bias information results. Below we will discuss several software packages that use various bias-correction algorithms to address this problem. Given the introductory nature of this article and the fact that these bias-correction techniques have not been developed for all of the information theory measures we will discuss, we will not employ these techniques in this tutorial. However, we feel it is important to discuss how bias can affect an analysis.

In simple cases, these biases can be understood as an interplay between the inherent noise associated with any analysis using limited amounts of data and the fact that information theory metrics must be greater than or equal to zero. Thus, noise in the estimate of the probability distribution produces a nonzero information result, even if the true underlying probability distribution would produce an information result of zero. Frequently, the most straightforward means of accounting for this bias when comparisons are performed between two information theory values is via significance testing (see Signficant testing). However, it is important to note the possible presence of bias when quoting information values. In other words, a measurement of 0.2 bits of entropy for the spiking activity of a neuron may be highly relevant within an analysis, but it is important to note that the true entropy of the spiking activity of the neuron may not be 0.2 bits, depending on the number of data points used in the calculation and the analysis techniques employed (e.g., number of bins).

As examples of the effects of bias associated with small amounts of data in entropy and mutual information calculations, we produced several simple model systems. First, we produced models of low (0.33 bits) and high (2 bits) entropy. We randomly selected observations from these probability distributions, estimated the probability distributions using these observations and the methods described above, and calculated the entropy. In Fig. 8A we can see that as more observations were performed, the estimated entropy values approached the entropy values for the true probability distributions. For few observations, the estimated values were biased downward, though several individual trials produced elevated entropy values for the low entropy model. We performed a similar simulation with simple low (0 bits) and high (0.53 bits) mutual information models (Fig. 8B). Again, when few observations were performed, the estimated values varied widely, with a bias toward higher mutual information values.

• Download figure
• Open in new tab
• Download powerpoint

Figure 8.

Example bias in entropy and mutual information calculations. A, Distributions of entropy values for low (0.33 bits) and high (2 bits) models as a function of number of observations. Entropy values tended to be biased downwards, though some trials produced elevated entropy values for trials with few observations. The probability distribution models were and . The binning method (four total bins) allowed for a maximum entropy of 2 bits. B, Distributions of mutual information values for low (0 bits) and high (0.53 bits) models as a function of number of observations. Mutual information values tended to be biased upwards, though some trials produced lower mutual information values for trials with few observations. Both models had two variables, each with two states. In the low-mutual-information model, all joint states were equally likely (i.e., independent variables). In the high-entropy model, the matching joint states had a probability of 0.45 and the other joint states had a probability of 0.05. The binning method (four total joint states) allowed for a maximum mutual information of 1 bit. Dark fringe represents interquartile range, and light fringe represents extremum range over 1000 trial simulations for each model and each unique number of observations.

Significance testing

So far, we have discussed the logistics of converting neuroscience data to probability distributions and numerous information theory measures that can be applied to probability distributions to gain useful insights into interactions in a system. The results of such analyses will always be numbers greater than or equal to zero. Importantly, a real experimental system will rarely produce an information theory measurement of precisely zero even when no interactions actually exist between the variables because of the presence of noise. In addition, bias can dramatically alter the results of information theory analyses (see Bias in entropy and mutual information). Therefore, a vital step of any information theory analysis is to assess which information theory measurements are significant.

Surrogate data testing or Monte Carlo analysis is frequently the solution to significance testing in information theory analyses (Lindner et al., 2011; Timme et al., 2014b; Wibral et al., 2014a; Asaad et al., 2017). This type of analysis is performed by generating surrogate null model data that preserve certain aspects of the data while randomizing other aspects. Once the information theory analysis is applied to the surrogate data, a distribution of null model information theory values can be compared to the information theory value from the real data. The proportion of null model information theory values that are found to be larger than or equal to the real data are then taken as an estimate of the p-value for the information theory result from the real data.

The randomization procedures necessary to generate surrogate null model data can be performed before creating probability distributions (e.g., spike jittering (Rolston et al., 2007; Timme et al., 2014b)) or after creating probability distributions. Randomization before converting to probability distributions is highly system specific. Thus, we will primarily focus on randomization of the probability distribution, because those techniques can be generally applied to any type of data. Furthermore, if the equal counts method of data binning is used (see Data Binning), the same null model data can be applied to different variables in many circumstances, greatly improving computational efficiency. This is possible with equal counts binning because the marginal distributions are identical across different variables. The marginal distributions have been rendered uniform via the equal counts binning method to maximize entropy. Typically, preserving these marginal distributions is the only constraint on the randomization method (see below). Still, in any case, it is important to randomize conservatively. For instance, in the case of spike train jittering, it is frequently best to jitter by small amounts to retain large-scale changes in firing rate caused by bursts. Furthermore, if transfer entropy is to be applied to the data, it is often best to not jitter the receiver neuron spike train to preserve the autocorrelation between the past and future states of Y (Eqn. 19).

Null model data can be created after generating probability distributions by randomizing the number of observations in joint states while preserving the number of observations for each state of each variable (i.e., by preserving the marginal distributions). For instance, suppose we have two magic coins that always produce the same flip result. Furthermore, suppose we flip each coin 10 times and produce 5 heads and 5 tails for each coin. The true observations from these data, as well as the observations from a randomly chosen null model where the joint observations are randomized while preserving the number of observation for each coin alone, are shown in Table 4.

The null model observations in Table 4 can be converted to a probability distribution via Eqn. 3, and any information measure that can be applied to the real data can be applied to the null model data. This process can be repeated many times to generate a null distribution of information values, which allows for an estimate of the p-value for the real data via the process described above.

To demonstrate the significance testing process using randomized probability distributions, we will use a simple model system with two variables ( X and Y ) and a measurement of their mutual information. Each variable can take one of two possible states. We varied the interactions in the model using a parameter a such that (Eqn. 26) (26)

Thus, when , there was no relationship between X and Y , and their mutual information should be zero in a perfect system. When , X and Y were copies and their mutual information should be one in a perfect system. However, in real experiments, it is necessary to estimate the probably distribution from observations (i.e., Eqn. 3), so the estimated distribution will not be identical to the real probability distribution, which will most likely produce nonzero mutual information results even when .

We first considered an example hypothetical experiment that conducted 100 observations from a model with no interactions between X and Y (Fig. 9A). Despite the fact that the true underlying probability distribution governing this experiment had no mutual information, a nonzero mutual information result was observed due to random fluctuations in the observations. However, when the data were randomized 10,000 times, the distribution of null information values enveloped the value from the real data. Thus, a high p-value estimate was produced, and this result would not typically be recognized as significant.

• Download figure
• Open in new tab
• Download powerpoint

Figure 9.

Example significance testing for mutual information via surrogate data null models. A,B, Example histogram of null model (randomized real data) mutual information values and the mutual information value from the real data (red line) for a system with no interactions (A) and for a system with interactions (B). As expected, the p-value in A indicates that the null model ( X and Y are independent) cannot not be rejected. In B, the p-value is low enough to reject the null model. C, p-values for models with different numbers of observations as a function of interaction strength (100 models generated for each a value and number of observations, solid line: median, fringe: interquartile range). Larger interaction strengths produced lower p-values, and models with more observations could detect weaker interactions. The minimum p-value resolution available in this demonstration was 0.0001 because 10,000 surrogate data sets were generated for each real data set.

Next, we considered an example hypothetical experiment that also had 100 observations, but the model contained interactions between X and Y (Fig. 9B). A nonzero mutual information result was observed, but the distribution of mutual information values from the null model were almost exclusively less than the mutual information result produced by the real data. Thus, a low p-value estimate was produced and this result would typically be recognized as significant.

Finally, we examined models with different interaction strengths and number of observations (Fig. 9C). As expected, when more observations were performed, weaker interactions could be detected as significant.

Selecting the number of surrogate data sets to generate is an important concern with this method of p-value estimation. In the examples shown above, we used 10,000 surrogate data sets, which allowed for a p-value resolution of 0.0001 (i.e., 1/10,000). Note that using this method, if all of the surrogate data sets produce information values less than the real value, it is only possible to estimate the p-value as [or, to provide a fixed value, ], not . In addition to improved resolution, additional surrogate data sets may be helpful when performing multiple comparison corrections for many information values (e.g., stimulus encoding by many neurons simultaneously) by lowering the minimum p-value. Therefore, it is advantageous to use as many surrogate data sets as possible. However, researchers must weigh the desired or required p-value resolution against available computational resources when determining the right number of surrogate data sets to use in their analyses.

Frequently the question arises of how to judge the magnitude of an information theory result. If an analysis yields a result of 0.5 bits, is that a lot? The answer to this question is highly analysis specific, which prevents interpreting these values in the absence of additional testing. If the data are discretized as discussed in Data Binning, the maximum allowed information result will be related to the number of bins used, with more bins yielding higher maximum information values. Frequently, a helpful method for judging the magnitude of an information theory result other than entropy is to compare the value to the entropy. For instance, if the mutual information between two variables is 0.5 bits, but each of the variables has an entropy of 0.5 bits, then the variables completely predict each other. In a different case where each variable has an entropy of 5 bits, a mutual information of 0.5 bits indicates less predictive ability.

Because the information theory analyses performed herein serve as demonstrations, we will forgo significance testing in the demonstrations. That said, we wish to emphasize that significance testing is necessary to draw meaningful conclusions from real data. To aid in this process, the software package included with this article is capable of conducting significance testing as discussed above (see Software).

In addition to randomizing the supplied data on a measurement-by-measurement basis, the software also allows for the user to supply a previously calculated null model. Using a previously calculated null model has the potential to yield better p-value resolution by running one null model with many surrogate data sets and greatly reduce calculation by removing the need to analyze surrogate data for each measurement. However, this tactic will yield these advantages only if the same null model can be applied to numerous measurements, which requires the same marginal distributions across all the measurements. Using uniform counts binning will frequently produce the same marginal distributions for the underlying variables, making this process possible. For instance, if all variables in an analysis possess the same number of observations and uniform counts binning is used, then all variables will have the same uniform marginal distributions. (Note that care must be taken to test that the marginal distributions are identical in the event of observations with identical values.)

Finally, we must acknowledge two other important issues surrounding significance testing. First, it is vital to consider the number of significance tests performed and to control for multiple comparisons (e.g., via Bonferroni correction or false discovery rate control; Benjamini and Hochberg, 1995; Benjamini and Yekutieli, 2001). This is especially relevant when setting analysis parameters such as number of bins, bin size, and delays. It is not appropriate to go parameter fishing to find a bin size that produces significant results. Parameters should be set initially based on the amount of available data and prior knowledge of the system so the number of significance tests can be reduced. However, in any case, multiple comparison corrections must be performed.

Second, the entire paradigm of null hypothesis significance testing is itself a topic of some controversy (McShane et al., 2017). To a large extent, we agree that an over emphasis on p-values without descriptions of effect sizes, effect uncertainty, or models that explain the data are not helpful to the advancement of science. A thorough discussion of this topic is beyond the scope of this tutorial, but we wish to emphasize that simply quoting a significant result in an information theory analysis (e.g., animal strain 1 showed significant mutual information, but animal strain 2 did not) is less than ideal. It is important to note that information theory allows researchers to move beyond simply quoting p-values because information theory analyses produce results in bits, which allows for a direct measurement of effect size (though bias effects must be considered as well; see Bias in Entropy and Mutual Information). In other words, a difference in mutual information results of 1 bit indicates a smaller effect size than a difference of 2 bits. The ability to measure effect sizes and perform significance testing in a model-free manner makes information theory a valuable tool, but, especially in the context of debates about null hypothesis significance testing, it is not the only tool that should be used (see Model Building).

Model building

We feel it is important to discuss the place of an information theory analysis in neuroscientific studies. As we discussed above (see What can information theory tell you?), information theory is very helpful for detecting complex interactions between variables, but it is not capable of providing a model that explains the gathered data. In other words, information theory can provide insights about which variables are related and how they are related (e.g., neuron 1 influences neuron 2, but not vice versa), but it cannot provide a unique model that explains those interactions (e.g., neuron 1 makes an excitatory synapse on neuron 2). Indeed, it is possible for systems governed by different rules to produce identical information theory results (James and Crutchfield, 2017). Of course, the model-free nature of information theory makes it an extremely powerful tool because it requires no assumptions about how the variables are related (though it can require some assumptions related to data analysis such as binning, see Discussion). This is especially relevant in complex systems where researchers frequently do not know what model should explain the data or do not want to restrict their analyses to certain types of models. However, the creation of these models is one of the most significant goals of science. In general, we seek to create compact, simplified models that explain vast oceans of data in every field of science. Therefore, to completely describe the place of information theory analyses in scientific research, it is necessary to emphasize the crucial step of model building as the step following information theory analyses. Information theory is only capable of eliminating some possible explanations for data, not identifying the one true explanation. Unlike the information theory analyses discussed above, model building is highly system dependent. Furthermore, we are aware of no studies that both perform an information theory analysis and then build a model using the results of that analysis for guidance. Thus, in this article it will be possible only to provide general suggestions about model building techniques to use following an information theory analysis.

A first general step to guide model building intuition is to examine the probability distribution to which the information theory analysis was applied. If, for instance, the mutual information was calculated between the firing rate of a neuron and a stimulus, the probability distribution can provide information about whether the neuron became more active when the stimulus was on or if it became less active. If, for instance, the transfer entropy was calculated between the firing rates of two neurons, the probability distribution can often determine if the relationship is excitatory or inhibitory. Unfortunately, simply examining the probability distribution can reveal complex nonlinear relationships, especially when more than two variables are involved. Furthermore, automated methods to perform these types of assessments throughout entire data sets are difficult to create.

Beyond the simple step of examining the probability distribution, we believe it is often best to perform some type of Bayesian analysis (Friston et al., 2003; Koller and Friedman, 2009; Kruschke, 2015). This process involves creating a model and fitting it to the data to obtain model parameters. For instance, if neural connectivity was assessed with transfer entropy, it would probably be best to assume the neurons exist in a network with certain types of excitatory and inhibitory interactions. The existence of a connection between any two neurons could be determined by the existence of a significant transfer entropy result between those neurons. Then, the results of the Bayesian analysis would describe the type of connection that exists between each pair of connected neurons in terms of the assumed model structure. Similar models could be constructed to explain encoding relationships between stimuli and neural activity, relationships between neural activity and behavior, and complex interactions between neural signals such as neural computations.

If the general process we are discussing for analyzing real neuroscience data is to perform an information theory analysis and then build a model, then the demonstration models shown below can be thought of as something similar to this process in reverse. For these demonstrations, the true model that generated the data is known, so we will be able to see how various features of the models correspond to features of the information theoretic results. In a real analysis, one could perform the information theory analysis first, then construct a model that is based on the information theory analysis, and fit the data, for instance.

Demonstration models

To demonstrate the utility of information theory analyses, we developed several models that primarily use Izhikevich neurons because they are easy to implement, are computationally efficient, and have been widely used in the literature (Izhikevich, 2003, 2007). We wish to emphasize that these models (or simulations) are used here only as a means to produce data very similar to real neural data for demonstration purposes. The purpose of analyzing data from these simulations is not to test novel scientific hypotheses; rather, these simulations are used purely for demonstration purposes. For all models, we attempted to use a realistic number of observations and number of experiments. In all cases, we generated and analyzed 20 models, which is roughly similar to conducting experiments with 20 neural recordings. Also, for all models, we used the equal counts method of binning the data to maximize the entropy of underlying variables and, when neuron spikes are concerned, we examine the information in terms of spike counts (i.e., rate coding). For all models, we include the necessary software to generate and analyze the data used in this article (see Software).

Software

To facilitate the use of information theory analyses in neuroscience, we have created a MATLAB software package (the Neuroscience Information Theory Toolbox) to carry out the analyses discussed in this tutorial article Extended Data. The software uses standard MATLAB functionality throughout, so once the user’s data are rendered as MATLAB variables, the remainder of the analysis can be conducted entirely in MATLAB. Our overall goal was to create software that functioned like any other basic MATLAB functions. In short, if you know how to manipulate matrices and use built-in functions in MATLAB, you will be able to quickly use this software. The analysis software is thoroughly documented within each function similar to built-in MATLAB functions, and overall guidance is supplied via a README file. Numerous simple demonstrations are included in addition to the software to generate the data and perform information theory analyses for all of the simulations discussed in this article. These simple demonstrations serve to highlight the coding necessary to implement various parameters and settings associated with different analyses. Furthermore, the software is capable of performing significance testing using both the real data and predefined null models (see Significance Testing), though this functionality was not employed in the simulations presented herein. Finally, basic functions are provided so the user can build more complicated analysis software to suit their needs, as well as macro-style functions that can be quickly and easily used to perform information theory analyses.

In fact, the generality of the software means that the software can be used to analyze data from disciplines other than neuroscience. We chose to use the word “neuroscience” in the name of the software package to highlight its intended role, but it can just as easily be applied to data from economics, physics, computer science, or sociology as it can to data from neuroscience.

While we feel our software package fills a valuable role in the field, we wish to emphasize that other excellent information theory software packages exist and that these packages may be more useful depending on the desired application (Ince et al., 2010; Quiroga and Panzeri, 2013; Lizier, 2014; Szabo, 2014; Moore et al., 2017). We explored 10 other available software packages and documented their important features to aid readers in comparing software options (Table 8). We found that many of these other packages are focused on a narrower type of analysis. Several software packages have been introduced to calculate transfer entropy (Ito et al., 2011; Lindner et al., 2011; Montalto et al., 2014; Pastore et al., 2016), often with the emphasis on estimating neural connectivity. Also, several software packages have focused on estimating entropy and mutual information using more advanced techniques (e.g., binless and kernel estimation techniques, as well as bias correction) than those presented herein to address problems surrounding continuous data and binning (Goldberg et al., 2009; Ince et al., 2009; Magri et al., 2009; Lindner et al., 2011; Lizier, 2014). In total, these software packages are capable of providing superior analyses of certain types of data, though their underlying assumptions (e.g., Gaussian distributed data in some cases) must be carefully weighed. Furthermore, it should be noted that, to the best of our knowledge, no such advanced estimation techniques have been extended to the partial information decomposition. We intentionally chose to not include bias correction algorithms in our software because of the presence of significance testing algorithms. We feel this aligns better with the introductory goals of this paper and typical neuroscience experiments where two values are compared while the absolute value of the result is frequently less important.

We attempted to perform computation speed comparisons with all of the software packages listed in Table 8. Unfortunately, we were only able to perform comparisons of identical analyses using the JIDT Toolbox (Lizier, 2014; 10 times faster than the Neuroscience Information Theory Toolbox) and the Transfer Entropy Toolbox (Ito et al., 2011; 100 times faster than the Information Theory Toolbox). This speed performance was not surprising given the focus of the Neuroscience Information Theory Toolbox on flexibility and standard MATLAB functionality. We were unable to test the other packages for two main reasons. First, several packages require the use of more advanced probability estimation techniques that are not available in the Neuroscience Information Theory Toolbox (Lindner et al., 2011; Montalto et al., 2014; Szabo, 2014). As a result, these packages took significantly longer to perform a given information theory calculation, but this was not a fair comparison due to the additional calculations performed by the other software package. Second, several software packages could not be made to run on our machines or required programming languages with which we are not familiar (Goldberg et al., 2009; Ince et al., 2009; Magri et al., 2009; Pastore et al., 2016; Moore et al., 2017; dit-contributors, 2018). Obviously, our inability to test the software may be a larger reflection on our programming abilities (or our lack of available time to devote to learning new languages) than on the quality of the software itself. However, we feel these are still relevant details for readers who may find themselves in similar positions. We anticipate that many of these software packages are orders of magnitude faster than the Neuroscience Information Theory Toolbox based on the languages in which they are written or their use of mex files in MATLAB.

In summation, we highly recommend that the interested reader pursue these other software packages if his or her research question better aligns with the goals of another software package. While many different software packages exist, we found that most of them differ in subtle but important ways regarding functionality, data types, purpose, and programming language. For instance, the dit toolbox lacks methods for handling experimental data, but it maintains a very large set of available information measures for discrete probability distributions (Dit-contributors, 2018). For the purposes of analyzing experimental data in a neuroscience context, we found the JIDT software package to be very helpful (Lizier, 2014). It possesses almost all of the information measured discussed here, data can be analyzed via an intuitive GUI, it can implement more complicated information estimation techniques, and, though it is written in JAVA, implementation in MATLAB is straightforward in most respects. JIDT does have some issues with importing/organizing data from MATLAB, different information measures, and interpretation of time-dependent information measures, but we would probably recommend it for most users as the next software to use after the Neuroscience Information Theory Toolbox. At the very least, we hope that this paper and our software package provide the reader with a useful introduction to information theory and information theoretic analyses of neuroscience data.