Coder Social home page Coder Social logo

eth-bloom's Introduction

eth-bloom

Join the conversation on Discord Build Status PyPI version Python versions

A python implementation of the bloom filter used by Ethereum.

This library and repository was previously located at https://github.com/pipermerriam/ethereum-bloom. It was transferred to the Ethereum foundation github in November 2017 and renamed to eth-bloom. The PyPi package was also renamed from ethereum-bloom to `eth-bloom.

Read more in the documentation below. View the change log.

For more information on what Ethereum Bloom Filters are see here.

Quickstart

python -m pip install eth-bloom

Usage

The BloomFilter object

>>> from eth_bloom import BloomFilter
>>> b = BloomFilter()
>>> b'a value' in b  # check whether a value is present
False
>>> b.add(b'a value')  # add a single value
>>> b'a value' in b
True
>>> int(b)  # cast to an integer
3458628712844765018311492773359360516229024449585949240367644166080576879632652362184119765613545163153674691520749911733485693171622325900647078772681584616740134230153806267998022370194756399579977294154062696916779055028045657302214591620589415314367270329881298073237757853875497241510733954508399863880080986777555986663988492288946856978031023631618215522505971170427986911575695114157059398791122395379400594948096
>>> bin(b)  # cast to a binary string
'0b100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'

You can also add an iterable of items to a bloom filter.

>>> b = BloomFilter()
>>> b'value-a' in b
False
>>> b'value-b' in b
False
>>> b.extend([b'value-a', b'value-b'])
>>> b'value-a' in b
True
>>> b'value-b' in b
True

You can initialize a bloom filter from an iterable of byte strings.

>>> b = BloomFilter.from_iterable([b'value-a', b'value-b'])  # initialize from an iterable of values.
>>> b'value-a' in b
True
>>> b'value-b' in b
True

You can initialize a bloom filter from the integer representation of the bloom bits.

>>> b = BloomFilter(3458628712844765018311492773359360516229024449585949240367644166080576879632652362184119765613545163153674691520749911733485693171622325900647078772681584616740134230153806267998022370194756399579977294154062696916779055028045657302214591620589415314367270329881298073237757853875497241510733954508399863880080986777555986663988492288946856978031023631618215522505971170427986911575695114157059398791122395379400594948096)
>>> b'a value' in b
True

You can also merge bloom filters

>>> from eth_bloom import BloomFilter
>>> b1 = BloomFilter()
>>> b2 = BloomFilter()
>>> b1.add(b'a')
>>> b1.add(b'common')
>>> b2.add(b'b')
>>> b2.add(b'common')
>>> b'a' in b1
True
>>> b'b' in b1
False
>>> b'common' in b1
True
>>> b'a' in b2
False
>>> b'b' in b2
True
>>> b'common' in b2
True
>>> b3 = b1 + b2  # using addition
>>> b'a' in b3
True
>>> b'b' in b3
True
>>> b'common' in b3
True
>>> b4 = b1 | b2  # or using bitwise or
>>> b'a' in b4
True
>>> b'b' in b4
True
>>> b'common' in b4
True
>>> b1 |= b2  # or using in-place operations (works with += too)
>>> b'a' in b1
True
>>> b'b' in b1
True
>>> b'common' in b1
True

Developer Setup

If you would like to hack on eth-bloom, please check out the Snake Charmers Tactical Manual for information on how we do:

  • Testing
  • Pull Requests
  • Documentation

We use pre-commit to maintain consistent code style. Once installed, it will run automatically with every commit. You can also run it manually with make lint. If you need to make a commit that skips the pre-commit checks, you can do so with git commit --no-verify.

Development Environment Setup

You can set up your dev environment with:

git clone [email protected]:ethereum/eth-bloom.git
cd eth-bloom
virtualenv -p python3 venv
. venv/bin/activate
python -m pip install -e ".[dev]"
pre-commit install

Release setup

To release a new version:

make release bump=$$VERSION_PART_TO_BUMP$$

How to bumpversion

The version format for this repo is {major}.{minor}.{patch} for stable, and {major}.{minor}.{patch}-{stage}.{devnum} for unstable (stage can be alpha or beta).

To issue the next version in line, specify which part to bump, like make release bump=minor or make release bump=devnum. This is typically done from the main branch, except when releasing a beta (in which case the beta is released from main, and the previous stable branch is released from said branch).

If you are in a beta version, make release bump=stage will switch to a stable.

To issue an unstable version when the current version is stable, specify the new version explicitly, like make release bump="--new-version 4.0.0-alpha.1 devnum"

eth-bloom's People

Contributors

bhargavasomu avatar carver avatar cburgdorf avatar davesque avatar dylanjw avatar fselmo avatar g-r-a-n-t avatar kclowes avatar miguel550 avatar njgheorghita avatar onyb avatar pacrob avatar pipermerriam avatar reedsa avatar tmckenzie51 avatar wolovim avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

eth-bloom's Issues

Documentation

  • Version: na
  • Python: all
  • OS: all

What was wrong?

Documentation of this repository is a little sparse. While the installation/use instructions are good, there's no mention in the README or otherwise of exactly what is the ethereum bloom filter, what a bloom filter is, or anything.

What did you expect it to do?

A link to the respective whitepaper(god knows there's enough of those in the ethereum world), or some kind of reference to exactly why this repository exists and what it's used for beyond code examples.

How close is this?

Bloom filters are a type of data structure that use cryptographic hashes
to help data stored within them to be retrieved or stored. They work like other
data structures, but in a probabilistic way: it allows for false positive
matches but not false negative matches. Bloom filters storage space use is
low relative to other kinds of data structures.
( For more information on Bloom filters, see Wikipedia: https://en.wikipedia.org/wiki/Bloom_filter )

A trie is an a kind of tree data structure for use with dynamic data

Ethereum bloom filters are bloom filters implemented using a "trie" data
structure and the SHA-256 ("keccak") cryptographic hash function.

To see the bloom filter used in the context of the full description of Ethereum / the "Yellow Paper" see

DR. GAVIN WOOD - ETHEREUM: A SECURE DECENTRALISED GENERALISED TRANSACTION LEDGER, EIP-150 REVISION, FOUNDER, ETHEREUM & ETHCORE, [email protected]
http://gavwood.com/Paper.pdf

Code to reproduce the error

pip shows wrong description

  • Version: 1.0.3
  • Python: 3.x
  • OS: osx

What was wrong?

Pip lists eth-bloom like this:
eth-bloom (1.0.3) - Python implementation of the Ethereum Trie structure

What did you expect it to do?

It should mention bloom filters instead of tries.

Code to reproduce the error

pip3 search eth-bloom

Python 3.7 support

Add support for Python 3.7

  • add to tox.ini
  • add to classifiers in setup.py
  • update .travis.yml with new python 3.7 runs
  • fix any issues that are exposed.

CI broken

What was wrong?

PR #34 fails the CI run, because it could not clone this (or another?) repository.
Circleci displays this error:

Using SSH Config Dir '/home/circleci/.ssh'
git version 2.20.1
Cloning git repository
Cloning into '.'...
Warning: Permanently added the RSA host key for IP address '140.82.114.4' to the list of known hosts.
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

What did you expect it to do?

Run the tests.

Code to reproduce the error

Create a PR in this repository.

Modernize project

What is wrong?

This project was create prior to our project template:

https://github.com/carver/ethereum-python-project-template

What should be done?

  • remove requirements-dev in favor of [dev] via extras_require as will as installation instructions in the README.
  • add python_requires and exclude 3.5.2
  • update lint run in tox.ini to use extras_require['lint'] style installation.

Add type hints and expose via PEP561

Largely copy/pasted from ethereum/py-evm#1398

Background

Type hints allow us to perform static type checking, among other things. They raise the security bar by catching bugs at development time that, without type support, may turn into runtime bugs.

This stackoverflow answer does a great job at describing their main benefits.

What is wrong?

This library currently does not have any type hints.

This needs to be fixed by:

  1. Adding all missing type hints.
  2. Enforcing (stricter) type checking in CI

How

There does exist tooling (monkeytype) to the generation of type hints for existing code bases. From my personal experience monkeytype can be helpful but does still require manual fine tuning. Also, manually adding these type hints does serve as a great boost to the general understanding of the code base as it forces one to think about the code.

  1. Run mypy --follow-imports=silent --warn-unused-ignores --ignore-missing-imports --no-strict-optional --check-untyped-defs --disallow-incomplete-defs --disallow-untyped-defs --disallow-any-generics -p eth_bloom

  2. Eliminate every reported error by adding the right type hint

Because this library supports older versions of python, the type hints will not be able to use the modern python3.6 syntax.

Definition of done

This issue is done when the following criteria are met:

  1. mypy is run in CI

Add a new command to the flake8 environment in the tox.ini file that runs:

mypy --follow-imports=silent --warn-unused-ignores --ignore-missing-imports --no-strict-optional --check-untyped-defs --disallow-incomplete-defs --disallow-untyped-defs --disallow-any-generics -p eth_bloom`
  1. Usage of type: ignore (silencing the type checker) is minimized and there's a reasonable explanation for its usage

Stretch goals

When this issue is done, stretch goals can be applied (and individually get funded) to tighten type support to qualify:

  1. mypy --strict --follow-imports=silent --ignore-missing-imports --no-strict-optional -p eth_bloom

Outdated eth-hash version dependency

  • Version: 1.0.0
  • Python:3.6
  • OS: osx/linux/win

What was wrong?

setup.py refers to outdated eth-hash dependency:
"eth-hash>=0.1.0a3,<0.2.0",

Current eth-hash version is 0.2.0

Code to reproduce the error

Some projects (e.g. eth-evm (trinity)) will fail to install\run properly:

# trinity
Traceback (most recent call last):
  File "/ara/venv_python/python3/lib/python3.6/site-packages/pkg_resources/__init__.py", line 578, in _build_master
    ws.require(__requires__)
  File "/ara/venv_python/python3/lib/python3.6/site-packages/pkg_resources/__init__.py", line 895, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/ara/venv_python/python3/lib/python3.6/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (eth-hash 0.2.0 (/ara/venv_python/python3/lib/python3.6/site-packages), Requirement.parse('eth-hash<0.2.0,>=0.1.0a3'), {'eth-bloom'})

How to fix?

Replace appropriate line in setup.py with e.g.:
"eth-hash>=0.1.0a3,<1.0.0",

eth-hash Dependency Issue

  • Version: 1.0.3
  • Python: 3.9
  • OS: osx

What was wrong?

$ pip install eth-tester
$ pip install web3
$ pip install py-evm
...
Requirement already satisfied: toolz>=0.8.0 in ./venv/lib/python3.9/site-packages (from cytoolz<1.0.0,>=0.10.1; implementation_name == "cpython"->eth-utils<2.0.0,>=1.9.4->py-evm) (0.11.1)
Installing collected packages: mypy-extensions, pyethash, cached-property, sortedcontainers, typing-extensions, trie, py-ecc, blake2b-py, eth-bloom, py-evm
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

eth-bloom 1.0.3 requires eth-hash<0.3.0,>=0.1.0a3, but you'll have eth-hash 0.3.1 which is incompatible.
Successfully installed blake2b-py-0.1.3 cached-property-1.5.2 eth-bloom-1.0.3 mypy-extensions-0.4.3 py-ecc-4.1.0 py-evm-0.3.0a20 pyethash-0.1.27 sortedcontainers-2.3.0 trie-2.0.0a5 typing-extensions-3.7.4.3

How should we fix it?

Maybe we should update the eth-hash requirement. It is on version 0.3.1 now (https://github.com/ethereum/eth-hash).

python 3.6 support

  • Version: 0.5.2
  • Python: <-- this
  • OS: ubuntu/debian gnu/linux

What was wrong?

both the tox.ini file and setup.py explicitly note support for python 3.4 3.5 (which was available in debian/ubuntu until recently, so it works now, but won't in the future) but not 3.6 (which will be standard in the future).

What did you expect it to do?

work in environments where the only python interpreter is 3.6+

Code to reproduce the error

tox

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.