Table 2
A List of Important Information Metrics
| Metric | Brief Description | Formula |
| Fisher information | Fisher50 51 can be credited with developing the theory of statistical information, positing that the distribution of an event is altered by changes in the value(s) of parameters. This measure is useful in determining the dispersion of a feature of the data. |  where f(x;θ) is the density of the event represented by X
|
| Shannon information | Shannon's (1948)52 definition of information measures variation in a distribution. This measure is the product of the amount of information provided by an event (which is inversely proportional to the probability of its occurrence) and the probability of the occurrence of the event. |  where pi is the probability of the event expressed by the random variable X.
|
| Kullback–Leibler information | Kullback (1951, 1959)53 54 presents a definition of information for discriminating between two distributions f,g that model 2 different populations. |  where f is the density of event A and g is the density of event B
|
| Mutual information | mutual information is defined as the amount of uncertainty reduction in an event B when an event A is known. It can take the form of Kullback information, measuring the distance from independence by comparing the joint distribution with the product of the marginal distributions. |  where H(B) is the entropy of the set B and H(B|A) is the entropy of the set Bwhen the set A is known or  where f(a,b) is the joint distribution of the two events.
|
| AIC | algorithmic information content (AIC) is a deterministic information criterion and equals the length of the most concise program that instructs a universal computer to produce the given dataset. | |
