mennthor / awkde

Adaptive Width KDE with Gaussian Kernels

Adaptive Width KDE with Gaussian Kernels

Installation

This uses the awesome pybind11 package which makes creating C++ bindings super convenient. Only the evaluation is written in a small C++ snippet to speed it up, the rest is a pure python implementation.

The setup is taken from the example at https://github.com/pybind/python_example Just clone the repository and invoke pip:

git clone https://github.com/mennthor/awkde
pip install [--user] [-e] ./awkde

Note: --user and -e are optional. Use --user to install in your user site-package directory (recommended) and -e if you want to only link the files to the site-package directories, so that when you make any changes in your local copy, you don't have to reinstall all the time.

Try to run the example/examply.py:

cd awkde/example
python example.py

You should get this plot in the examples folder (you need matplotlib for the plot):

Algorithm

The unweighted kernel density estimator is defined as

where the product h * lambda takes the role of a local sigma.

The kernel bandwith is choosen locally to account for variations in the density of the data. Areas with large density gets smaller kernels and vice versa. This smoothes the tails and gets high resolution in high statistics regions. The local bandwidth paramter is defined as

where

is some normalization and ^f(X_i) the KDE estimate at the data point X_i. The local bandwidth is multiplied to the global bandwidth for each kernel.

Furthermore different scales in data is accounted for by scaling it via its covariance matrix to an equal spread. First a global kernel bandwidth is applied to the transformed data and then based on that density a local bandwidth parameter is applied.

All credit for the method goes to [1] and to S. Schoenen and L. Raedel for huge parts of the implementation. For information on Silverman or Scott rule, see [2] or [3].

References

• [1] B. Wang and X. Wang, "Bandwidth Selection for Weighted Kernel Density Estimation", Sep. 2007, DOI: 10.1214/154957804100000000.
• [2] D.W. Scott, "Multivariate Density Estimation: Theory, Practice, and Visualization", John Wiley & Sons, New York, Chicester, 1992.
• [3] B.W. Silverman, "Density Estimation for Statistics and Data Analysis", Vol. 26, Monographs on Statistics and Applied Probability, Chapman and Hall, London, 1986.

awkde's Issues

Problem installing

I am really interested in using your code.

When I try to install it using the command provided in read_me I run into an error.
I'm on Mac Catalina and I want use annaconda to work on python.
Here is the error I encounter when I try to install the package :

_> pip install [--user] -e ./awkde

ERROR: Exception:
Traceback (most recent call last):
File "/opt/anaconda3/lib/python3.7/site-packages/pip/_vendor/packaging/requirements.py", line 98, in init
req = REQUIREMENT.parseString(requirement_string)
File "/opt/anaconda3/lib/python3.7/site-packages/pip/_vendor/pyparsing.py", line 1947, in parseString
raise exc
File "/opt/anaconda3/lib/python3.7/site-packages/pip/_vendor/pyparsing.py", line 1937, in parseString
loc, tokens = self._parse(instring, 0)
File "/opt/anaconda3/lib/python3.7/site-packages/pip/_vendor/pyparsing.py", line 1677, in _parseNoCache
loc, tokens = self.parseImpl(instring, preloc, doActions)
File "/opt/anaconda3/lib/python3.7/site-packages/pip/_vendor/pyparsing.py", line 4052, in parseImpl
loc, exprtokens = e._parse(instring, loc, doActions)
File "/opt/anaconda3/lib/python3.7/site-packages/pip/_vendor/pyparsing.py", line 1681, in _parseNoCache
loc, tokens = self.parseImpl(instring, preloc, doActions)
File "/opt/anaconda3/lib/python3.7/site-packages/pip/_vendor/pyparsing.py", line 3798, in parseImpl
raise ParseException(instring, loc, self.errmsg, self)
pip._vendor.pyparsing.ParseException: Expected stringEnd, found '[' (at char 11), (line:1, col:12)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/anaconda3/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
status = self.run(options, args)
File "/opt/anaconda3/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 301, in run
wheel_cache
File "/opt/anaconda3/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 249, in populate_requirement_set
wheel_cache=wheel_cache
File "/opt/anaconda3/lib/python3.7/site-packages/pip/_internal/req/constructors.py", line 395, in install_req_from_line
parts = parse_req_from_line(name, line_source)
File "/opt/anaconda3/lib/python3.7/site-packages/pip/_internal/req/constructors.py", line 346, in parse_req_from_line
extras = convert_extras(extras_as_string)
File "/opt/anaconda3/lib/python3.7/site-packages/pip/_internal/req/constructors.py", line 77, in convert_extras
return Requirement("placeholder" + extras.lower()).extras
File "/opt/anaconda3/lib/python3.7/site-packages/pip/_vendor/packaging/requirements.py", line 102, in init
requirement_string[e.loc : e.loc + 8], e.msg
pip._vendor.packaging.requirements.InvalidRequirement: Parse error at "'[--user]'": Expected stringEnd
_

PyPi Publication

It would be nice to have this library available on PyPi, so that it can be easily integrated into Jupyter online notebooks that don't accept git sources.

I'm just wondering why it has never been published. Is the C++ binding an issue?

I could publish it from a fork, if you don't mind.

Issue with pip install after new changes

Hi,
I was trying to install awKDE with new updates following instructions but I am getting this error.
pip install git+https://github.com/mennthor/awkde.git@master
Collecting git+https://github.com/mennthor/awkde.git@master
Cloning https://github.com/mennthor/awkde.git (to revision master) to /tmp/pip-req-build-qkqacjs1
Running command git clone --filter=blob:none --quiet https://github.com/mennthor/awkde.git /tmp/pip-req-build-qkqacjs1
Resolved https://github.com/mennthor/awkde.git to commit d3dd004
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [43 lines of output]
running egg_info
creating /tmp/pip-pip-egg-info-iuq_5ils/awkde.egg-info
writing /tmp/pip-pip-egg-info-iuq_5ils/awkde.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-pip-egg-info-iuq_5ils/awkde.egg-info/dependency_links.txt
writing requirements to /tmp/pip-pip-egg-info-iuq_5ils/awkde.egg-info/requires.txt
writing top-level names to /tmp/pip-pip-egg-info-iuq_5ils/awkde.egg-info/top_level.txt
writing manifest file '/tmp/pip-pip-egg-info-iuq_5ils/awkde.egg-info/SOURCES.txt'
/home/ahnitz/projects/env/lib/python3.8/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
warnings.warn(
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "/tmp/pip-req-build-qkqacjs1/setup.py", line 34, in
setup(
File "/home/ahnitz/projects/env/lib/python3.8/site-packages/setuptools/init.py", line 87, in setup
return distutils.core.setup(**attrs)
File "/home/ahnitz/projects/env/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 148, in setup
return run_commands(dist)
File "/home/ahnitz/projects/env/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
dist.run_commands()
File "/home/ahnitz/projects/env/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
self.run_command(cmd)
File "/home/ahnitz/projects/env/lib/python3.8/site-packages/setuptools/dist.py", line 1214, in run_command
super().run_command(command)
File "/home/ahnitz/projects/env/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
cmd_obj.run()
File "/home/ahnitz/projects/env/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 308, in run
self.find_sources()
File "/home/ahnitz/projects/env/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 315, in find_sources
mm.run()
File "/home/ahnitz/projects/env/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 550, in run
self.add_defaults()
File "/home/ahnitz/projects/env/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 587, in add_defaults
sdist.add_defaults(self)
File "/home/ahnitz/projects/env/lib/python3.8/site-packages/setuptools/_distutils/command/sdist.py", line 228, in add_defaults
self._add_defaults_ext()
File "/home/ahnitz/projects/env/lib/python3.8/site-packages/setuptools/_distutils/command/sdist.py", line 311, in _add_defaults_ext
build_ext = self.get_finalized_command('build_ext')
File "/home/ahnitz/projects/env/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 298, in get_finalized_command
cmd_obj = self.distribution.get_command_obj(command, create)
File "/home/ahnitz/projects/env/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 859, in get_command_obj
cmd_obj = self.command_obj[command] = klass(self)
TypeError: 'NoneType' object is not callable
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

how can i install this?

hi,

i have what looks like a long tail dsitibution and the current kde in scikit either oversmooths the larger area of the curve and oversmooths the tails. therefore i wish to use this method. however when i run

pip install [--user] -e ./awkde

i get

string end error? how can i install this

ModuleNotFoundError: No module named 'awkde.backend'

I have cloned the repo and tried to install it.

The installation seems to be working fine however, import awkde throws ModuleNotFoundError: No module named 'awkde.backend'

Any idea?

Does it works with 2 dimension Kernel Density?

Hi! Does this library also works for two dimensional data for Kernel Density Estimate? I've tried it with a dataset on which i'm working separately with the Sklearn KDE. The Awkde library leds to an incredible oversmoothing? Is it possible that I'm doing something wrong?

Here the comparison between the sample of 2000 points done with Sklearn KDE in first image and with Awkde (oversmooth) in second image. I want to underline that i've used the same global bandwidth.

1D example

Hello I was curious if there was a 1D example of this being used for something like a non-parametric smoother for scatter plot data.