ethereum / eth-utils

Utility functions for working with ethereum related codebases.

Ethereum Utilities

Common utility functions for python code that interacts with Ethereum

Read more in the documentation on ReadTheDocs. View the change log.

Quickstart

python -m pip install eth-utils

Developer Setup

If you would like to hack on eth-utils, please check out the Snake Charmers Tactical Manual for information on how we do:

• Testing
• Pull Requests
• Documentation

We use pre-commit to maintain consistent code style. Once installed, it will run automatically with every commit. You can also run it manually with make lint. If you need to make a commit that skips the pre-commit checks, you can do so with git commit --no-verify.

Development Environment Setup

You can set up your dev environment with:

git clone [email protected]:ethereum/eth-utils.git
cd eth-utils
virtualenv -p python3 venv
. venv/bin/activate
python -m pip install -e ".[dev]"
pre-commit install

Update Networks

The list of networks resides in the JSON file under eth_utils/__json/eth_networks.json. This file is used to initialize Networks, which can be used to obtain network information with a chain ID.

Run the script to update the JSON file with the response from the remote list.

python update_networks.py

If there are new networks they will appear in the JSON file. After checking the updates, open a PR to make them available in a new release.

Release setup

To release a new version:

make release bump=$$VERSION_PART_TO_BUMP$$

How to bumpversion

The version format for this repo is {major}.{minor}.{patch} for stable, and {major}.{minor}.{patch}-{stage}.{devnum} for unstable (stage can be alpha or beta).

To issue the next version in line, specify which part to bump, like make release bump=minor or make release bump=devnum. This is typically done from the main branch, except when releasing a beta (in which case the beta is released from main, and the previous stable branch is released from said branch).

If you are in a beta version, make release bump=stage will switch to a stable.

To issue an unstable version when the current version is stable, specify the new version explicitly, like make release bump="--new-version 4.0.0-alpha.1 devnum"

eth-utils's Issues

Typo and samples in README

In is_string sample, there is a typo 'bytearra' instead of 'bytearray'
In * is_text* all the samples are using is_string

I'll open a PR to fix that

Is `eth_utils.abi.collapse_if_tuple` private or public API?

Originally posted by @pipermerriam in #143 (comment)

Validate text and hexstr parameters are str type

• Version: 1

What was wrong?

See this comment about bytes being passed to to_text as text=. In that case to_text() returned a bytes value, which seems wrong.

How can it be fixed?

Raise an exception if the text or hexstr parameters are not str, in any of the conversion methods.

eth-utils 1.0.0-beta.1 not found in PyPI

What was wrong?

The versions of eth-keyfiles and eth-keys were published on PyPI and depend on the version eth-utils 1.0.0-beta.1 that is not published on PyPI / or is hidden by default

How can it be fixed?

Publish 1.0.0-beta.1 on PyPI? (Or fix the dependencies for now)

ImportError: cannot import name 'apply_formatter_at_index'

• Version: 1.0.3
• Python: 3.5
• OS: linux(Ubuntu 16.04)

ImportError: cannot import name 'apply_formatter_at_index'

from web3 import Web3
  File "/lib/python3.5/site-packages/web3/__init__.py", line 7, in <module>
    from eth_account import Account  # noqa: E402
  File "/lib/python3.5/site-packages/eth_account/__init__.py", line 1, in <module>
    from eth_account.account import Account  # noqa: F401
  File "/lib/python3.5/site-packages/eth_account/account.py", line 21, in <module>
    from eth_utils.curried import (
  File "/lib/python3.5/site-packages/eth_utils/curried/__init__.py", line 7, in <module>
    from eth_utils import (
ImportError: cannot import name 'apply_formatter_at_index'

How can it be fixed?

AssertionError on import (wrong keccak function?) OSX, python 2.7.13

Not sure what I've got wrong here:

~ $ pip install ethereum-utils
Requirement already satisfied: ethereum-utils in /usr/local/lib/python2.7/site-packages
Requirement already satisfied: pysha3>=0.3 in /usr/local/lib/python2.7/site-packages (from ethereum-utils)
~ $ python
Python 2.7.13 (default, Apr  4 2017, 08:44:49) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import eth_utils
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/eth_utils/__init__.py", line 5, in <module>
    from .abi import (  # noqa: F401
  File "/usr/local/lib/python2.7/site-packages/eth_utils/abi.py", line 3, in <module>
    from .crypto import keccak
  File "/usr/local/lib/python2.7/site-packages/eth_utils/crypto.py", line 18, in <module>
    assert keccak('') == b"\xc5\xd2F\x01\x86\xf7#<\x92~}\xb2\xdc\xc7\x03\xc0\xe5\x00\xb6S\xca\x82';{\xfa\xd8\x04]\x85\xa4p"  # noqa: E501
AssertionError

Add a replace_exceptions decorator

What was wrong?

Exception replacement is a common need when stitching together a lot of libraries.

Web3.py and eth-tester can both use it, at least.

How can it be fixed?

Add a replace_exceptions implementation, like the one in eth-tester:

ethereum/eth-tester@9f92225

Stable 0.x branch and backport fixes

Need to create a v0 branch from the latest v0.x.x release and backport #45 into it.

setuptools-markdown dependency unnecessary

PyPI supports Markdown formatted README files nowadays: https://packaging.python.org/guides/making-a-pypi-friendly-readme/

Thus, the dependency on setuptools-markdown can be removed. Also from the other eth-* packages.

Missing to_normalized_address example in README

Hi!

Looks like the to_normalized_address in README is merely copy of to_checksum_address.
It will be great to include the correct explanation and sample of it

pytest No module named eth_utils

• Version: 1.0.0
• Python: 2.7.14-2ubuntu1 / 3.6.3-0ubuntu2
• OS: ubuntu 17.10 artful

What was wrong?

compiling with either python3 or python, it doesn't seem to matter: eth_utils is not found by pytest.

make test
py.test tests
========================================================================== test session starts ===========================================================================
platform linux2 -- Python 2.7.14+, pytest-3.2.1, py-1.4.34, pluggy-0.4.0 -- /usr/bin/python
cachedir: .cache
rootdir: /home/z/python/eth-utils/github-eth-utils/eth-utils, inifile: pytest.ini
plugins: hypothesis-3.12.0
collected 0 items / 11 errors

================================================================================= ERRORS =================================================================================
___________________________________________________________ ERROR collecting tests/abi-utils/test_abi_utils.py ___________________________________________________________
ImportError while importing test module '/home/z/python/eth-utils/github-eth-utils/eth-utils/tests/abi-utils/test_abi_utils.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/abi-utils/test_abi_utils.py:3: in
from eth_utils.hexidecimal import (
E ImportError: No module named eth_utils.hexidecimal
_______________________________________________________ ERROR collecting tests/address-utils/test_address_utils.py _______________________________________________________
ImportError while importing test module '/home/z/python/eth-utils/github-eth-utils/eth-utils/tests/address-utils/test_address_utils.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/address-utils/test_address_utils.py:5: in
from eth_utils.address import (
E ImportError: No module named eth_utils.address
______________________________________________________ ERROR collecting tests/currency-utils/test_currency_tools.py ______________________________________________________
ImportError while importing test module '/home/z/python/eth-utils/github-eth-utils/eth-utils/tests/currency-utils/test_currency_tools.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/currency-utils/test_currency_tools.py:9: in
from eth_utils.currency import (
E ImportError: No module named eth_utils.currency
____________________________________________________ ERROR collecting tests/encoding-utils/test_big_endian_integer.py ____________________________________________________
ImportError while importing test module '/home/z/python/eth-utils/github-eth-utils/eth-utils/tests/encoding-utils/test_big_endian_integer.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/encoding-utils/test_big_endian_integer.py:10: in
from eth_utils.encoding import (
E ImportError: No module named eth_utils.encoding
_________________________________________________ ERROR collecting tests/formatting-utils/test_padding_left_and_right.py _________________________________________________
ImportError while importing test module '/home/z/python/eth-utils/github-eth-utils/eth-utils/tests/formatting-utils/test_padding_left_and_right.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/formatting-utils/test_padding_left_and_right.py:5: in
from eth_utils.formatting import (
E ImportError: No module named eth_utils.formatting
________________________________________________ ERROR collecting tests/functional-utils/test_return_value_decorators.py _________________________________________________
ImportError while importing test module '/home/z/python/eth-utils/github-eth-utils/eth-utils/tests/functional-utils/test_return_value_decorators.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/functional-utils/test_return_value_decorators.py:3: in
from eth_utils import (
E ImportError: No module named eth_utils
____________________________________________ ERROR collecting tests/hexidecimal-utils/test_0x_prefix_addition_and_removal.py _____________________________________________
ImportError while importing test module '/home/z/python/eth-utils/github-eth-utils/eth-utils/tests/hexidecimal-utils/test_0x_prefix_addition_and_removal.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/hexidecimal-utils/test_0x_prefix_addition_and_removal.py:5: in
from eth_utils.hexidecimal import (
E ImportError: No module named eth_utils.hexidecimal
____________________________________________________ ERROR collecting tests/hexidecimal-utils/test_is_0x_prefixed.py _____________________________________________________
ImportError while importing test module '/home/z/python/eth-utils/github-eth-utils/eth-utils/tests/hexidecimal-utils/test_is_0x_prefixed.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/hexidecimal-utils/test_is_0x_prefixed.py:5: in
from eth_utils.hexidecimal import (
E ImportError: No module named eth_utils.hexidecimal
________________________________________________________ ERROR collecting tests/hexidecimal-utils/test_is_hex.py _________________________________________________________
ImportError while importing test module '/home/z/python/eth-utils/github-eth-utils/eth-utils/tests/hexidecimal-utils/test_is_hex.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/hexidecimal-utils/test_is_hex.py:5: in
from eth_utils import (
E ImportError: No module named eth_utils
______________________________________________________ ERROR collecting tests/string-utils/test_string_coersions.py ______________________________________________________
ImportError while importing test module '/home/z/python/eth-utils/github-eth-utils/eth-utils/tests/string-utils/test_string_coersions.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/string-utils/test_string_coersions.py:10: in
from eth_utils.string import (
E ImportError: No module named eth_utils.string
_________________________________________________________ ERROR collecting tests/types-utils/test_types_utils.py _________________________________________________________
ImportError while importing test module '/home/z/python/eth-utils/github-eth-utils/eth-utils/tests/types-utils/test_types_utils.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/types-utils/test_types_utils.py:3: in
from eth_utils.types import (
E ImportError: No module named eth_utils.types
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 11 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================== 11 error in 0.56 seconds ========================================================================
Makefile:28: recipe for target 'test' failed
make: *** [test] Error 2

How can it be fixed?

??

Enable discovery of type data via PEP561

Do the same as was done in ethereum/eth-typing#10

Many many warnings when application is run

• Version: 0.8.0
• Python: 3.5
• OS: osx/linux/win

What was wrong?

I have many-many lines when I run my program an I can't omit them:

/home/user/.virtualenvs/test/lib/python3.5/site-packages/eth_utils/string.py:23: DeprecationWarning: The `force_obj_to_text` function has been deprecated and will be removed in a subsequent release of the eth-utils library. UTF8 cannot encode some byte values in the 0-255 range which makes naive coersion between bytes and text representations impossible without explicitly declared encodings.
  "declared encodings.".format(fn.__name__)
/home/user/.virtualenvs/test/lib/python3.5/site-packages/eth_utils/string.py:23: DeprecationWarning: The `force_text` function has been deprecated and will be removed in a subsequent release of the eth-utils library. UTF8 cannot encode some byte values in the 0-255 range which makes naive coersion between bytes and text representations impossible without explicitly declared encodings.
  "declared encodings.".format(fn.__name__)
/home/user/.virtualenvs/test/lib/python3.5/site-packages/eth_utils/string.py:23: DeprecationWarning: The `force_obj_to_text` function has been deprecated and will be removed in a subsequent release of the eth-utils library. UTF8 cannot encode some byte values in the 0-255 range which makes naive coersion between bytes and text representations impossible without explicitly declared encodings.
  "declared encodings.".format(fn.__name__)

Code for reproduce

import warnings

warnings.filterwarnings("ignore", category=DeprecationWarning)
warnings.filterwarnings("ignore", category=PendingDeprecationWarning)

from web3 import IPCProvider, Web3

web3 = Web3(IPCProvider('/home/user/.ethereum/geth.ipc'))

latest_block = web3.eth.getBlock('latest')

for i in range(latest_block.number):
    block = web3.eth.getBlock(i)
    ...

How can it be fixed?

Please don't use filters for warning in library code. They are only for applications:

warnings.simplefilter('always', DeprecationWarning)

There is no easy way to switch off warnings from my application.

Convert documentation to readthedocs

What was wrong?

Documentation is currently all README based

How can it be fixed?

Convert documentation to be Sphinx based and host on readthedocs.

Default context of decimals changes after import

What was wrong?

Problem is here:
https://github.com/ethereum/eth-utils/blob/master/eth_utils/currency.py#L11

decimal.DefaultContext.prec = 999

My application uses other precision prec=28 but package changes global variable after import. It's not good behavior for libraries.

How can it be fixed?

I suppose this line should be removed from library or it should use a local context instead of default context.

Add `ValidationError`

What was wrong?

ethereum/py-evm#1125 (comment)

It would be useful to have a common ValidationError exception that can be used across libraries.

How can it be fixed?

Add a new module eth_utils.exceptions and add a ValidationError exception to that module. Make it importable as from eth_utils import ValidationError

Follow-up:

• use new exception in web3.py
• use new exception in eth-abi?
• use new exception in eth (in py-evm code base)
• use new exception in p2p (in py-evm code base)
• use new exception in trinity (in py-evm code base)
• others?

Import from the curried versions of toolz

What was wrong?

eth_utils.toolz does not provide access to the curried versions of the methods

How can it be fixed?

Add another eth_utils.curried.toolz module like the eth_utils.toolz one, but importing from toolz.curried.*.

Add smoke tests for at least one method from each subsection of https://toolz.readthedocs.io/en/latest/api.html -- like:

from eth_utils.curried.toolz import drop
drop2 = drop(2)
assert list(drop2([4, 3, 2, 1])) == [2, 1]

Should also smoke test that non-curryable functions are working, like:

from eth_utils.curried.toolz import compose
hexint = compose(hex, int)
assert hexint('10') == '0xa'

Constrain versions to disallow Python 3.5.2

What was wrong?

See ethereum/web3.py#1012 (comment)

Python 3.5.2 has a bug...

How can it be fixed?

In the setup.py file, add a new line for python_requires like this from web3.py: https://github.com/ethereum/web3.py/blob/master/setup.py#L33

Use the requirement string: >=3.5,!=3.5.2,<4 to restrict the python versions. It might just be a ! without the equals, I can't remember.

Add type hints

What was wrong?

Python's dynamic-typing nature makes it impossible to perform static type checking

How can it be fixed?

We can add type hints (like in ethereum/eth-keys#19) and use mypy to perform static type checking, and hopefully catch bugs before they affect our users. Just like in the PR above, pyannotate can be used to generate initial type hints

python 3.6 (3.7?) in setup.py and elsewhere

• Version: 1.0.0
• Python: 3.6 / 3.7
• OS: ubuntu 18.04 bionic / *

What was wrong?

Right now in setup.py and elsewhere there's some hardcoded mentioning of python 3.4 3.5 ...but now with 3.6 (and 3.7?) out it's probably a good idea to explicitly bump that version in there too.

ie
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5',
],

to
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
],

tox.ini:
basepython =
py27: python2.7
py34: python3.4
py35: python3.5

to
basepython =
py27: python2.7
py34: python3.4
py35: python3.5
py36: python3.6
etc

How can it be fixed?

see above

Add tests to release tarball (in PyPI)

Could you include tests in the source tarball in PyPI? I think it's good practice:

• https://docs.python.org/2/distutils/sourcedist.html#specifying-the-files-to-distribute
• https://stackoverflow.com/a/20180000

This way one can run the tests after installing the package. Also, I'm currently packaging this for a Linux distro, and it would be really helpful to be able to run the tests as a part of the automatic package building process while using PyPI for the release tarballs.

Similar request applies to eth-typing at least.

to_wei calculates inaccurate when value less then 1 submitted

What is wrong?

I've fell into issue with to_wei when working with web3.py library and found out that it uses eth_utils.currency.to_wei to proceed calculation.

Short description on how I fell into issue.

I was working with smart-contract where I need to setup amount of ether it should accept to prevent cases when user submitted wrong amount.
Variable stores in wei and data is setup in constructor.

web3 = self.get_web3()

contract = web3.eth.contract(abi=ContractHelper.get_abi(self.CONTRACT_NAME),
                             bytecode=ContractHelper.get_bytecode(self.CONTRACT_NAME),
                             contract_name=self.CONTRACT_NAME)
...
self.transaction_id = contract.deploy(transaction={'from': AccountHelper.get_base_account(),
                             "gasPrice": web3.toWei(25, 'gwei')}, args=[web3.toWei(0.05, 'ether')])

For simplicity purpose I've replaced variables with raw data supply.

After deploying contract I didn't pay attention to events result until I found out that contract doesn't accept payment that was working without issue in ethereum IDE

Code seemed fine but then I've looked into events fired by contract.

Instead of 50000000000000000 value in wei was 50000000000000002.

How can it be fixed?

In general problem was with too big precision (999) that is set in default context. It goes much too deep into floating than it actually needs.

I forked repo and fixed issue. Will submit PR soon.

is_hex() doesn't work as shown in readme

• Version: 1.0.2
• Python: 3.6
• OS: osx

What was wrong

is_hex() doesn't work as shown in readme:

>>> is_hex(b'') => TypeError: is_hex requires text typed arguments. Got: b''
>>> is_hex(b'0x12') => TypeError: is_hex requires text typed arguments. Got: b'0x12'

It appears to only expect a str.

ImportError when importing pad_left and pad_right from eth_utils

• Version: Latest
• Python: 3.5
• OS: osx/linux

What was wrong?

Code:
from eth_utils import (
pad_left,
pad_right,
)

Error message:
"Exception Value: | cannot import name 'pad_left'"

How can it be fixed?

It looks like in the more recent version of eth_utils, there is no longer a file called "formatting.py" which contains the function defs for pad_left() and pad_right() functions. There is also no longer an import for these in the init.py in the root "eth_utils" folder.

I have a docker setup which is installing fresh from pip each deployment so it would be nice if this was fixed. The import for pad_left and pad_right is actually called from the "eth_abi" library, not my custom code. The location it is called in eth_abi is: eth_abi/utils/padding.py

Thank you!

Title is wrong

small spelling error in README:

etherum-utils -> ethereum-utils

Wrong function_signature_to_4byte_selector sample in README

The sample call to function_signature_to_4byte_selector in README is wrong, using function_abi_to_4byte_selector (probably copy/pasted)

I'll send a PR to fix it

Switch to `ctoolz` library.

What was wrong?

The toolz library is pretty awesome, but it is all pure python and thus, as slow as other pure python stuff.

Turns out there is a c implementation of the entire library! https://pypi.python.org/pypi/cytoolz

How can it be fixed?

Change to use ctoolz

make `keccak` work for bytearrays

• Version: 1.0.0b2
• Python: 3.6
• OS: osx

What was wrong?

When trying to hash a bytearray with the keccak function (keccak(bytearray(b'abc'))) it throws the following error:
TypeError: expected an int in first arg, or keyword of hexstr or text

There are to problems:

• The error message seems wrong, as the first argument can be a boolean or bytes
• More important: hashing should work for bytearrays as well. Or is there a reason this isn't implemented?

How can it be fixed?

Make keccak work for bytearrays

Decorator to convert to NamedTuple

What was wrong?

Maybe it would be better if we had a decorator to convert key value pairs coming out in the form of generators or dictionaries to a NamedTuple. Something on the lines of to_NamedTuple(tupleType).

How can it be fixed?

It should be simple to add the decorator from the existing API similar to to_dict etc

New tests to validate type safety is preserved

What was wrong?

From: #151

Looks like flatten_return, reversed_return and sorted_return don't do a good job of retaining type safety. It's not easy to write a test for this now.

How can it be fixed?

Figure out how to easily write tests that demonstrate both positive and negative assertions about mypy checks.

Validating Ethereum address

What is the best way to validate Ethereum address, assuming it might or might not be checksummed? This was not immediately clear from the APIs.

Add `import_string` to docs

What was wrong?

The import_string function isn't documented.

How can it be fixed?

Add basic docs.

Should is_hex() return False if the argument is not a str?

What was wrong?

It might be inconvenient for people to have is_hex() raise a TypeError if the argument is a string. It always forces a double-check:

if is_string(val) and is_hex(val):
    ...

How can it be fixed?

We could simply return False if the value is not a str. If this is the route we go, it should be added to v2-alpha, but not v1. There are backwards incompatibility problems, like:

try:
  if is_hex(val):
    return val
  else:
    # this code block currently gets to presume a string, but would crash during concatenation after the change
    raise ValidationError("string " + value + " must be hex")
except TypeError:
  # after the change this wouldn't run, which is fine I guess
  raise ValidationError(f"{value} must be a string")

@pipermerriam At first, I was very in favor of returning False, but after thinking a bit longer, I'm not so sure. People regularly confuse bytes <-> str, so it might just feel broken to them when they try as is_hex(bytes_val) and it always returns false. It's nice to be in their face that they are trying to do something that doesn't make sense.

Maybe there could be a new is_hexstr() that would have this new behavior of returning False on anything that's not a str. Then hopefully is_hexstr(bytes_val) would look more... wrong. This could be backported into v1 as well.

Deprecate and remove `force_text` and `force_bytes` and dependent functions.

What was wrong?

The force_bytes and force_text functions are bad. They use latin1 encoding due to the fact that utf8 cannot encode all of the bytes values between 0-255. This has caused much confusion given that utf8 is the default encoding that most people expect.

How can it be fixed?

Fully deprecate the automatic coercion of text -> bytes and bytes -> text including all of the functions which use the derivative functions like coerce_return_to_*, and coerce_args_to_*.

Introduce new functions to_text, to_bytes and maybe others, pulled from Web3.py in web3.utils.encoding.

Any function which previously dependend on the coerce_args_to_* should then be migrated to the new call format fn(primitive=None, hexstr=None, text=None) format to remove implicit type conversions.

Any function which depends on coerce_return_to* should be converted to do explicit conversions.

Expand currency test suite to fuzz test using float values.

What was wrong?

The test suite for to_wei is not doing fuzz testing against inputs of float type. This should be done as I'm not currently sure that there aren't odd float values that won't produce un-expected results.

How can it be fixed?

Add float inputs to the fuzz testing suite to see that from_wei(to_wei(v, 'ether'), 'ether') == decimal.Decimal(str(v)) for float inputs.

To normalized address throws ValueError

• Version: 1.0.3
• Python: 3.6
• OS: osx

Running the same example off the Readme:
to_normalized_address('0x000000000000000000000000c305c901078781c232a2a521c2af7980f8385ee9')
throws:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File ".../lib/python3.6/site-packages/eth_utils/address.py", line 68, in to_normalized_address
    "Unknown format {}, attempted to normalize to {}".format(address, hex_address)
ValueError: Unknown format 0x000000000000000000000000c305c901078781c232a2a521c2af7980f8385ee9, attempted to normalize to 0x000000000000000000000000c305c901078781c232a2a521c2af7980f8385ee9

`to_wei` does not handle floating point numbers correctly

What is wrong

The to_wei function doesn't handle floating point numbers in the expected way.

In [121]: to_wei(1.2, 'ether')
Out[121]: 1199999999999999955

How can it be fixed

We should be able to use the string representation of floating point numbers to achieve precise conversions in this case.

Stop encoding keccak arguments with latin-1

What was wrong?

eth_utils.keccak() is encoding text arguments with latin-1. This is different from behavior of other eth libraries.

This is a sub-task of #36

See https://gist.github.com/holiman/6f57cb5b082b6712b9e8c19fe48ca960#gistcomment-2332077

How can it be fixed?

One of:

• only accept bytes as argument
• convert text to utf-8
• Edit: use the (primitive=None, texstr=None, hexstr=None) argument signature approach -- preferred

Probably requires a major version bump

makefile testall scenario slightly off

• Version: 1.0.0
• Python: 3.6
• OS: ubuntu 17.10 artful

What was wrong?

Makefile shows:

@echo "testall - run tests on every Python version with tox"

but the actual scenario is

test-all:
tox

How can it be fixed?

either change test-all to testall or testall to test-all.

apply_key_map behaviour is ambiguous if remapping would overwrite other field.

• Version: 1.0

What was wrong?

The apply_key_map function can be both non-deterministic as well as ambiguous to what the proper behavior is when a remapped key collides with a non-remapped key.

>>> apply_key_map({'a': 'b'}, {'b': 3})
{'b': 3}

>>> apply_key_map({'a': 'b'}, {'a': 2, 'b': 3})
{'b': 3}

>>> apply_key_map({'a': 'b'}, collections.OrderedDict((('a', 2), ('b', 3))))
{'b': 3}

>>> apply_key_map({'a': 'b'}, collections.OrderedDict((('b', 3), ('a', 2))))
{'b': 2}

How can it be fixed?

I think the proper behavior to fix both of these issue is to do the following check

key_conflicts = set(  # get the keys from the starting value
    original_value.keys()
).difference(  # remove any keys that are going to be remapped
    key_map.keys()
).intersection(  # find the intersection with the result of the remap.
    v 
    for k, v 
    in key_map.items() 
    if k in original_value
)

If there are any key conflicts then refuse to apply the remapping, raising a KeyError indicating what keys were in conflict.

Backfill changelog

What was wrong?

Looks like we've forgotten to update the CHANGELOG since v0.7.3

How can it be fixed?

Browse through the list of closed pull requests and/or issues in the commit ranges between each version, or the commits themselves and populate the CHANGELOG.

The `to_tuple` type decorators don't correctly handle generics for item types.

What was wrong?

When using to_tuple on a function which operates on a generic type, type safety is not preserved.

TVal = TypeVar('TVal')

@to_tuple
def typing_to_tuple_generic(values: Sequence[TVal]) -> Iterable[TVal]:
    yield from values

v_tuple_generic = typing_to_tuple_generic([1, 2, 3])
reveal_type(v_tuple_generic)  # error: Revealed type is 'builtins.tuple[TVal`-1]'

How can it be fixed?

???

Fail fast if decode_hex is passed bytes

• Version: 1.0.0-beta.2
• Python: 3.6
• OS: linux

What was wrong?

decode_hex() was passed a bytes value, and it wasn't caught until deep in the eth-utils stack. Took an extra second to debug.

How can it be fixed?

Immediately reject non-str values to decode_hex

Ubuntu-16.04 cannot run populus and error message is "ImportError: cannot import name 'force_text'"

• Version: 2.2.0
• Python: 3.5.2
• OS: linux
• eth_utils: 1.0.1

Problem Description

1. Use pip install populus
   pip install populus

2. Type this command

populus compile

3. Show error message

Traceback (most recent call last):
  File "/home/manager/Github_Project/ICO_contracts/venv_3/bin/populus", line 7, in <module>
    from populus.cli import main
  File "/home/manager/Github_Project/ICO_contracts/venv_3/lib/python3.5/site-packages/populus/__init__.py", line 24, in <module>
    from .project import Project  # NOQA
  File "/home/manager/Github_Project/ICO_contracts/venv_3/lib/python3.5/site-packages/populus/project.py", line 10, in <module>
    from populus.config.defaults import (
  File "/home/manager/Github_Project/ICO_contracts/venv_3/lib/python3.5/site-packages/populus/config/__init__.py", line 1, in <module>
    from .base import (  # noqa: F401
  File "/home/manager/Github_Project/ICO_contracts/venv_3/lib/python3.5/site-packages/populus/config/base.py", line 12, in <module>
    from populus.utils.mappings import (
  File "/home/manager/Github_Project/ICO_contracts/venv_3/lib/python3.5/site-packages/populus/utils/mappings.py", line 8, in <module>
    from eth_utils import (
ImportError: cannot import name 'force_text'

How can it be fixed?

Probably the new version of eth-utils has no function named 'force_text' and populus uses it.

Need a method that can compile populus from source code

to_bytes() adds significant overhead to keccak()

That function seems to add a big overhead to keccak(), and given its performance-critical nature I think we should consider dropping it (making keccak() accept only bytes). Here are some numbers showing the overhead it adds when compared to running the backend implementation directly:

$ python -m timeit -n 100000 -v -s "import os; from eth_hash.utils import auto_choose_backend; backend = auto_choose_backend()" "backend.keccak256(os.urandom(32))"
raw times: 1.81 1.82 1.85
100000 loops, best of 3: 18.1 usec per loop

$ python -m timeit -n 100000 -v -s "import os; from eth_utils import keccak" "keccak(os.urandom(32))"
raw times: 2.51 2.45 2.44
100000 loops, best of 3: 24.4 usec per loop

unexpected upper() called in to_checksum_address(address)

eth-utils/eth_utils/address.py

Line 118 in c06d261

norm_address[i].upper()

norm_address[i] should not invoke upper().

In that case, address is '0x8eb009288b68c343de5bc393a7e13c28f5425780', checksum_address is '0x8EB009288b68C343DE5Bc393a7e13C28F5425780'.

is_checksum_address(value) returns False. The two addresses should be compared case-insensitively, although unnecessary

Don't accept `str` as a primitive in `to_hex()`

• Version: 1.0.2
• Python: 3.5
• OS: osx/linux/win

What was wrong?

to_hex() shouldn't presume the kind of str value in primitive. It's the kind of ambiguity we aim to avoid, using hexstr and text keyword args:
https://github.com/ethereum/eth-utils/blob/master/eth_utils/conversions.py#L41-L42

How can it be fixed?

to_hex() should reject str values, by raising a TypeError and listing the accepted types: (bytes, bytearray, int, bool)

Double-check if this breaks anything in py-evm or web3.py

Extract many utilities from web3 into eth-utils

What was wrong?

Web3.py has added a number of utilities that other Ethereum libraries might find useful. For example, we are in the process of extracting the local signing code from web3 to eth-account (working name), and it depends on datastructures, conversions, and decorators that don't really have much to do with Web3, but are currently housed there.

How can it be fixed?

Extract to eth-utils

• decorators
  • combomethod,
• encoding
  • hexstr_if_str,
  • text_if_str,
  • to_bytes,
  • to_int,
  • to_hex,
  • to_text,
  • ?
• ?

Add to a new eth-rlp

• HashableRLP
• Transaction(rlp.Serializable)
• ?

Add to a new hexbytes

HexBytes

Compose from available libraries

FrozenAttributeDict = compose(AttrDict, frozendict)

vyper 0.0.4 has requirement ethereum==2.1.3

• Python: 3.6.4
• OS: osx

What was wrong?

➜  eth-utils git:(master) pip install eth-utils
Requirement already satisfied: eth-utils in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (1.0.3)
Requirement already satisfied: cytoolz<1.0.0,>=0.8.2 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from eth-utils) (0.9.0.1)
Requirement already satisfied: eth-hash<1.0.0,>=0.1.0 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from eth-utils) (0.1.2)
Requirement already satisfied: toolz>=0.8.0 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from cytoolz<1.0.0,>=0.8.2->eth-utils) (0.9.0)
vyper 0.0.4 has requirement ethereum==2.1.3, but you'll have ethereum 2.3.0 which is incompatible.

How can it be fixed?

Make the requirement ethereum >= 2.1.3 ?

Add support for `bytearray`s in the various conversion utils

What was wrong?

We don't support bytearrays in most/all of the various conversion utils. Since bytearray and bytes types are interchangeable in many ways, and supporting both is trivial, we should add support.

How can it be fixed?

Add support to the conversion utils for bytearray types in the primitive field, like: to_int, to_bytes, to_hex, to_text and anywhere else that supports a bytes type as an input.