dawnsong / npeet_lnc

local non-uniformity correction for mutual information estimation

NPEET_LNC

Mutual information Estimation with Local Non-uniformity Correction (a branch of NPEET Non-parametric Entropy Estimation Toolbox)

This package contains Python code implementing mutual information estimation functions for continuous variables. This estimator gives a correction term to the traditional kNN estimator and can estimate mutual information more accurately than Kraskov estimator for strongly dependent variables with limited samples.

To use this package, it requires scipy 0.12 or greater.

Example installation and usage:

git clone https://github.com/BiuBiuBiLL/NPEET_LNC.git

>>> from lnc import MI
>>> import numpy as np
>>> x = [1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0]
>>> y = [1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0]
>>> MI.mi_LNC([x,y],k=5,base=np.exp(1),alpha=0.25) #Apply LNC estimator
Output: 25.29758574548632
>>> MI.mi_Kraskov([x,y],k=5,base=np.exp(1)) #Same data applied to Kraskov estimator
Output: 0.62745310245310382

To run the test example in test.py, simply type the following in terminal

$python test.py
Output:
Testing 2D linear relationship Y=X+Uniform_Noise
noise level=1e-07, Nsamples = 500
True MI(x:y) 16.0472260191
Kraskov MI(x:y) 5.79082342999
LNC MI(x:y) 15.9355050641

Testing 2D quadratic relationship Y=X^2+Uniform_Noise
noise level=1e-07, Nsamples = 1000
True MI(x:y) 15.8206637545
Kraskov MI(x:y) 6.48347086055
LNC MI(x:y) 11.4586276609

Testing 3D linear relationship Y=X+Uniform_Noise, Z=X+Uniform_Noise
noise level=1e-07, Nsamples = 500
True MI(x:y:z) 32.2836569188
Kraskov MI(x:y:z) 11.58164686
LNC MI(x:y:z) 32.1846129957

Testing 3D quadratic relationship Y=X^2+Uniform_Noise, Z=X^2+Uniform_Noise
noise level=1e-07, Nsamples = 500
True MI(x:y:z) 31.5020968975
Kraskov MI(x:y:z) 11.57764686
LNC MI(x:y:z) 25.6686276941

One need to specify the thresholding parameter alpha when using LNC estimator. This parameter is related to the nearest-neighbor parameter k and dimensionality d, see alpha.xlsx for the detailed alpha value to use.

Also see the references on the implemented estimators.

			A Kraskov, H Stögbauer, P Grassberger. 
			http://pre.aps.org/abstract/PRE/v69/i6/e066138
			Estimating Mutual Information
			PRE, 2004.

			Shuyang Gao, Greg Ver Steeg and Aram Galstyan 
			http://arxiv.org/abs/1411.2003
			Efficient Estimation of Mutual Information for Strongly Dependent Variables
			AISTATS, 2015.