polkascan / py-substrate-interface

Python Substrate Interface

Polkascan Open-Source

Polkascan Open-Source Application

Quick deployment (Use hosted Polkascan API endpoints)

Step 1: Clone repository:

git clone https://github.com/polkascan/polkascan-os.git

Step 2: Change directory:

cd polkascan-os

Step 3: Check available releases:

git tag

Step 4: Checkout latest releases:

git checkout v0.x.x

Step 5: Make sure to also clone submodules within the cloned directory:

git submodule update --init --recursive

Step 6: Then build the other docker containers

docker-compose -p kusama -f docker-compose.kusama-quick.yml up --build

Use public Substrate RPC endpoints

Step 1: Clone repository:

git clone https://github.com/polkascan/polkascan-os.git

Step 2: Change directory:

cd polkascan-os

Step 3: Check available releases:

git tag

Step 4: Checkout latest releases:

git checkout v0.x.x

Step 5: Make sure to also clone submodules within the cloned directory:

git submodule update --init --recursive

Step 6: During the first run let MySQL initialize (wait for about a minute)

docker-compose -p kusama -f docker-compose.kusama-public.yml up -d mysql

Step 7: Then build the other docker containers

docker-compose -p kusama -f docker-compose.kusama-public.yml up --build

Full deployment

The following steps will run a full Polkascan-stack that harvests blocks from a new local network.

Step 1: Clone repository:

git clone https://github.com/polkascan/polkascan-os.git

Step 2: Change directory:

cd polkascan-os

Step 3: Check available releases:

git tag

Step 4: Checkout latest releases:

git checkout v0.x.x

Step 5: Make sure to also clone submodules within the cloned directory:

git submodule update --init --recursive

Step 6: During the first run let MySQL initialize (wait for about a minute)

docker-compose -p kusama -f docker-compose.kusama-full.yml up -d mysql

Step 7: Then build the other docker containers

docker-compose -p kusama -f docker-compose.kusama-full.yml up --build

Links to applications

• Polkascan Explorer GUI: http://127.0.0.1:8080
• Monitor harvester progress: http://127.0.0.1:8080/kusama/harvester/admin
• Harvester Task Monitor: http://127.0.0.1:5555
• Polkadot JS Apps: http://127.0.0.1:8081

Other networks

• Polkadot: Use docker-compose.polkadot-quick.yml, docker-compose.polkadot-public.yml and docker-compose.polkadot-full.yml
• Substrate Node Template (https://github.com/substrate-developer-hub/substrate-node-template): Use docker-compose.substrate-node-template.yml

Add custom types for Substrate Node Template

• Modify harvester/app/type_registry/substrate-node-template.json to match the introduced types of the custom chain
• Truncate runtime and runtime_* tables on database
• Start harvester
• Check http://127.0.0.1:8080/node-template/runtime-type if all type are now correctly supported
• Monitor http://127.0.0.1:8080/node-template/harvester/admin if blocks are being processed and try to restart by pressing "Process blocks in harvester queue" if process is interupted.

Cleanup Docker

Use the following commands with caution to cleanup your Docker environment.

Prune images

docker system prune

Prune images (force)

docker system prune -a

Prune volumes

docker volume prune

API specification

The Polkascan API implements the https://jsonapi.org/ specification. An overview of available endpoints can be found here: https://github.com/polkascan/polkascan-pre-explorer-api/blob/master/app/main.py#L60

Troubleshooting

When certain block are not being processed or no blocks at all then most likely there is a missing or invalid type definition in the type registry.

Some steps to check:

• The Substrate node should be started in archive mode (--pruning archive)
• Be sure to regularly update, also the Git submodules: git submodule update --init --recursive for latest version of py-scale-codec
• Check which blocks are not being processed and try to restart manually at: http://127.0.0.1:8080/<NETWORK_NAME>/harvester/admin
• Check Celery queue for more details about failing blocks: http://127.0.0.1:5555/tasks?state=FAILURE
• Check and match types in ./harvester/app/type_registry/custom_types.json with added types in Substrate

You can also dive into Python to pinpoint which types are failing to decode:

import json
from scalecodec.type_registry import load_type_registry_file
from substrateinterface import SubstrateInterface

substrate = SubstrateInterface(
    url='ws://127.0.0.1:9944',
    type_registry_preset='substrate-node-template',
    type_registry=load_type_registry_file('harvester/app/type_registry/custom_types.json'),
)

block_hash = substrate.get_block_hash(block_id=3899710)

extrinsics = substrate.get_block_extrinsics(block_hash=block_hash)

print('Extrinsincs:', json.dumps([e.value for e in extrinsics], indent=4))

events = substrate.get_events(block_hash)

print("Events:", json.dumps([e.value for e in events], indent=4))

py-substrate-interface's Issues

[Feature request] Example of signing and submitting an extrinsic

Thanks for your work guys! I was just wondering if you would be so kind of providing an example of signing and submitting extrinsics in the README?
Thanks!

Optimize metadata lookups

Create lookup dict of metadata storage functions/events/calls in stead of nested if-statements to improve performance

Outdated examples at pkg.go.dev

I just tried the examples (simpleTransfer example in particular) at https://pkg.go.dev/github.com/centrifuge/go-substrate-rpc-client#pkg-examples and ran into a bunch of compatibility issue.

Eventually, I got it working using the main_test.go file and the answer at centrifuge/go-substrate-rpc-client#99 .

Can someone update the docs on the pkg.go.dev site? I'm new to go and not sure how to do it. If you can point me in the right direction, I can update the examples myself.

Add support for MetadataV12

Add support for paritytech/substrate#6969

Fixed index for modules

Mortal extrinsic

I want to use the era field on extrinsic. Could you update the feature? plz

py-substrate-interface/substrateinterface/__init__.py

Line 969 in 0f5c62e

raise NotImplementedError("Mortal transactions not yet implemented")

compose_call() results in payload that makes sign-transaction crash

N.B.: subkey sign-transaction ... v2.0.0 itself has a bug because why would it otherwise now suddenly fail with its own (May 2019) parameters, see "long story short" here: paritytech/substrate#5180 (comment). But that does not mean that there is not a second bug in the Python code ... ;-)

---

Could it be that the compose_call() returns a faulty hex blob? Try this:

Get an old version of subkey that was working for sure:

git clone substrate
cd substrate
git checkout e74463feb2f79ef68a1515e7ad626d8695daa51d
cd subkey
cargo run -- sign-transaction     --call 0300ff408aa36b945720031fbb2e661fc9c9f1ed799be5369f991bbfce02103b5f0e44e514     --nonce 0     --suri //Alice///     --password password     --prior-block-hash f8349fe94e9fd86e8a85745bbb70a4d2d5f4d8fc32a1f8fc08088faf1840e705

that does not crash but results in

0x250281ffd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d820c9dae649279bc7323e3905b326d7a052c8ba3e644f4cf62ab97354df6167a226af6a7d0d90ed8a0a90261c7bd3512b08b487e77a8d254e76e335fdaabeb0a00000300ff408aa36b945720031fbb2e661fc9c9f1ed799be5369f991bbfce02103b5f0e44e514

but when I now swap the --call for a payload output of your compose_call()

cargo run -- sign-transaction     --call a8040400ff8eaf04151687736326c9fea17e25fc5287613693c912909cb226aa4794f26a48070010a5d4e8   --nonce 0     --suri //Alice///     --password password     --prior-block-hash f8349fe94e9fd86e8a85745bbb70a4d2d5f4d8fc32a1f8fc08088faf1840e705

the same command crashes with

thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', /rustc/f3e1a954d2ead4e2fc197c7da7d71e6c61bad196/src/libcore/macros/mod.rs:15:40

P.S.: Yes new version

pip install --upgrade substrate-interface

Successfully installed scalecodec-0.9.19 substrate-interface-0.9.6

then I generated the above 'a8040400ff8eaf041....` payload like this (correct?):

import substrateinterface
substrate = substrateinterface.SubstrateInterface(url="ws://127.0.0.1:9944/")

BOB_ADDRESS = '5FHneW46xGXgs5mUiveU4sbTyGBzmstUspZC92UhjJM694ty'

payload = substrate.compose_call(call_module='Balances',
                                 call_function='transfer',
                                 call_params={'dest': BOB_ADDRESS,
                                              'value': 1000000000000})
print (payload.replace("0x", ""))

Bounty types

Implement bounty types: paritytech/substrate#5715

Failed to send: {'code': 1010, 'message': 'Invalid Transaction', 'data': 'BadProof'}

in the multi-node network, when i send the extrinsic by rpc, and the error occur.
Failed to send: {'code': 1010, 'message': 'Invalid Transaction', 'data': 'BadProof'}

but when i run substrate by a single node and send the extrinsic, it success.
there are my code:

from substrateinterface import SubstrateInterface, Keypair, SubstrateRequestException
from substrateinterface.utils.ss58 import ss58_encode
import threading

# 5CdDzb25wGQ41XnjsX3LHmTAWtJhxGyKVM9LsFZMgdGQzfrY
# 发送方
def run():

	try:
		substrate = SubstrateInterface(
			url="wss://hk.listen.io/",
			address_type=42,
			type_registry_preset='default'
		)

		import time
		time.sleep(1)
		keypair = Keypair.create_from_mnemonic("adult hunt thank force make satisfy saddle pumpkin reject very avoid goat")

		# print("Created address: {}".format(keypair.ss58_address))

		mnemonic = Keypair.generate_mnemonic()

		# 接收方随机生成
		keypair1 = Keypair.create_from_mnemonic(mnemonic, 2)

		# 可以直接调用自己定义的模块  不需要特殊处理
		call = substrate.compose_call(
			call_module='Listen',
			call_function='air_drop',
			call_params={
				# 接收方
				# 'dest': '5GnGKSCitk1QPpMNugtTGX9t6TqzDGvL5BqKzLfHNsLSrwqN',
				'des': keypair1.ss58_address,
				# 'value': 10 * 10**14

			}
		)

		extrinsic = substrate.create_signed_extrinsic(call=call, keypair=keypair)

		try:
			result = substrate.submit_extrinsic(extrinsic, wait_for_inclusion=True)
			print("Extrinsic '{}' sent and included in block '{}'".format(result['extrinsic_hash'], result['block_hash']))
			# substrate.s

		except SubstrateRequestException as e:
			print("Failed to send: {}".format(e))
	except Exception as e:
		print(e)
if __name__ == "__main__":

	# run(substrate)
	for i in range(100):
		run()

Support of Python3.6 in the latest versions

Part of our project runs with Python 3.6.10 and the latest version of this package available there seems to be 0.9.20.
Any way to make the versions after that compatible with Python 3.6?

Unit test test_generate_signature_payload fails in latest Polkadot runtime

Unit test test_generate_signature_payload fails in latest Polkadot runtime:

AssertionError: '0x05[72 chars]0800180000000500000091b171bb158e2d3848fa23a9f1[98 chars]90c3' != '0x05[72 chars]0800170000000500000091b171bb158e2d3848fa23a9f1[98 chars]90c3'
- 0x0500586cb27c291c813ce74e86a60dad270609abf2fc8bee107e44a80ac00225c409411f000800180000000500000091b171bb158e2d3848fa23a9f1c25182fb8e20313b2c1eb49219da7a70ce90c391b171bb158e2d3848fa23a9f1c25182fb8e20313b2c1eb49219da7a70ce90c3
?                                                                                  ^
+ 0x0500586cb27c291c813ce74e86a60dad270609abf2fc8bee107e44a80ac00225c409411f000800170000000500000091b171bb158e2d3848fa23a9f1c25182fb8e20313b2c1eb49219da7a70ce90c391b171bb158e2d3848fa23a9f1c25182fb8e20313b2c1eb49219da7a70ce90c3
?                                                                                  ^      

how to parse transaction from block

how to parse transaction from block, pls help me!

Signing of proxy transaction creates invalid transaction signature

I'm trying to sign a proxy transaction but the polkadot node is giving me an invalid transaction error. Example here with addresses redacted:

>>> print(proxy_call.value)
{'call_args': {'call': {'call_args': {'targets': [<nomination_address_list>]},
                        'call_function': 'nominate',
                        'call_module': 'Staking'},
               'force_proxy_type': 'Staking',
               'real': '<controller_address>'},
 'call_function': 'proxy',
 'call_module': 'Proxy'}

nomination_extrinsic = polkadot_connection.create_signed_extrinsic(call=proxy_call, keypair=<controller_proxy_keypair>)
result = polkadot_connection.submit_extrinsic(nomination_extrinsic, wait_for_inclusion=True)

Throws this exception
substrateinterface.exceptions.SubstrateRequestException: {'code': 1010, 'message': 'Invalid Transaction', 'data': 'Transaction has a bad signature'}

example for get_storage(module, function, params) please

I have been going through all the functions:

Connected to: https://dev-node.substrate.dev:9933/

[no parameters]
          get_system_name : substrate-node
              get_version : 2.0.0
 get_chain_finalised_head : 0x189dafe56cf99d25ddf6a8a6eec2e08834d3435d721a91b898c633573e8ba7a0
           get_chain_head : 0xd41eade474e58b90bbe7d7b490c0306da66dc1b36cdd4d6595aeda847a03ef8f

[param block_hash=0xd41eade474e58b90bbe7d7b490c0306da66dc1b36cdd4d6595aeda847a03ef8f]
          get_chain_block : dict_keys(['block', 'justification'])  -->  block : dict_keys(['extrinsics', 'header']) 
         get_block_header : dict_keys(['digest', 'extrinsicsRoot', 'number', 'parentHash', 'stateRoot'])  -->  digest : dict_keys(['logs']) 
         get_block_events : dict_keys(['jsonrpc', 'result', 'id']) 
get_block_runtime_version : dict_keys(['apis', 'authoringVersion', 'implName', 'implVersion', 'specName', 'specVersion']) 
         get_block_number : 905 

[param block_hash=0xd41eade474e58b90bbe7d7b490c0306da66dc1b36cdd4d6595aeda847a03ef8f]
       get_block_metadata : dict_keys(['magicNumber', 'metadata'])
                         -->
                  magicNumber : 1635018093
                          key : MetadataV8
      MetadataV8-->modules[0] : dict_keys(['name', 'prefix', 'storage', 'calls', 'events', 'constants', 'errors'])

[param block_id=905]
          get_chain_block : dict_keys(['block', 'justification'])  -->  block : dict_keys(['extrinsics', 'header']) 
           get_block_hash : 0xd41eade474e58b90bbe7d7b490c0306da66dc1b36cdd4d6595aeda847a03ef8f 

[params: storage_key=xxh6464('Sudo Key')='0x50a63a871aced22e88ee6466fe5aa5d9', block_hash=0xd41eade474e58b90bbe7d7b490c0306da66dc1b36cdd4d6595aeda847a03ef8f]
get_storage_by_key(storage_key='0x50a63a871aced22e88ee6466fe5aa5d9', block_hash=0xd41eade474e58b90bbe7d7b490c0306da66dc1b36cdd4d6595aeda847a03ef8f)
'0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d'

Now that worked all very well. Thanks a lot. Makes my life much easier.
The last one could benefit from one small improvement, see that feature request.

However, this remaining 1, I could not yet get working:

get_storage(module='Balances', function='FreeBalance', params='d43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d', block_hash=0xd41eade474e58b90bbe7d7b490c0306da66dc1b36cdd4d6595aeda847a03ef8f)

results in

None

Which is better than an exception, so the query seems to be well-formed at least; it is just asking for the wrong thing probably? Following Shawn's tutorial, it should accept a list of parameters, right? But it doesn't want to take a list.

I have tried some permutations of possible parameters, using the instructions over there; but obviously I have't understood the role of the arguments (module, function, params) yet.

For most of your other functions in this library, we can live well without unittests - but please you give one or two working examples for get_storage(). Perhaps connecting to "https://dev-node.substrate.dev:9933/" because then I can easily replicate it here.

Thanks. A lot.

how to get result of the extrinsic

how to kown the result of the extrinsic is successful of failed

There is the information of the block 3233385 below, whose result is failed, but I can't find the failed content as follows

{
    "block": {
        "extrinsics": [
            {
                "valueRaw": "040200",
                "extrinsic_length": 10,
                "version_info": "04",
                "call_code": "0200",
                "call_function": "set",
                "call_module": "timestamp",
                "params": [
                    {
                        "name": "now",
                        "type": "Compact<Moment>",
                        "value": "2020-07-19T00:28:00",
                        "valueRaw": "0b806a78647301",
                    }
                ],
            },
            {
                "valueRaw": "040900",
                "extrinsic_length": 7,
                "version_info": "04",
                "call_code": "0900",
                "call_function": "final_hint",
                "call_module": "finalitytracker",
                "params": [
                    {
                        "name": "hint",
                        "type": "Compact<BlockNumber>",
                        "value": 3233382,
                        "valueRaw": "9a59c500",
                    }
                ],
            },
            {
                "valueRaw": "041400",
                "extrinsic_length": 4,
                "version_info": "04",
                "call_code": "1400",
                "call_function": "set_heads",
                "call_module": "parachains",
                "params": [
                    {
                        "name": "heads",
                        "type": "Vec<AttestedCandidate>",
                        "value": [],
                        "valueRaw": "",
                    }
                ],
            },
            {
                "valueRaw": "840400",
                "extrinsic_length": 143,
                "version_info": "84",
                "account_length": "ff",
                "account_id": "02c75d7e02037413de1f18624a99723ca6a94f4b41a10bf8aa5a311b032f1a4c",
                "account_index": None,
                "account_idx": None,
                "signature_version": 1,
                "signature": "302df07a3605c4ec297e032971e2bd0c619bcb4237071b51be8ce0418a46a8444a51c5cfdbbaaf37ed4aafcad0d75be6d54b0d3e0896236edbd085ae9fc48c8e",
                "extrinsic_hash": "e952c3e24ef34aa9239bdb62087ca6a8e09fa25aba299064b479f77597d5dff2",
                "call_code": "0400",
                "call_function": "transfer",
                "call_module": "balances",
                "nonce": 7,
                "era": "3502",
                "tip": 0,
                "params": [
                    {
                        "name": "dest",
                        "type": "Address",
                        "value": "0xe616888158c22446a7f1c560f19977bfef8ebcea1e0f9c9bfb57fbd930ffb761",
                        "valueRaw": "",
                    },
                    {
                        "name": "value",
                        "type": "Compact<Balance>",
                        "value": 5235000000000,
                        "valueRaw": "0b00be50dec204",
                    },
                ],
            },
        ],
        "header": {
            "digest": {
                "logs": [
                    {
                        "type": "(ConsensusEngineId, Bytes)",
                        "value": {
                            "engine": "BABE",
                            "data": "0x02fd0000009898d80f00000000",
                        },
                    },
                    {
                        "type": "(ConsensusEngineId, Bytes)",
                        "value": {
                            "engine": "BABE",
                            "data": "0x20282f3392b1f646ed79c843febfa6fc88df01f2a51e5cffb8cff177aab0240f5bd679f846d61492dc32c4c12241adf7b563ef38c1c9b8c301eacdb42257b68c",
                        },
                    },
                ]
            },
            "extrinsicsRoot": "0xd8375c0c01277b1b6f30454b2b3e216ccba345296dd6acffe50129fc54e112f5",
            "number": 3233385,
            "parentHash": "0xd878f206891641dbaf46edab0490b809e5af4aeabde2c773d6a926ad8a0f3c19",
            "stateRoot": "0x29ff08719047e16b7e5eb924f5073845dea32ddd9a46a3d55e29f7e891c42a34",
        },
    },
    "justification": None,
}

Port exhaustion

Current implementation causes port exhaustion. This can be solved implementing urllib3 PoolManager. Implementing this prevents opening a new port for every http request. This means less processor overhead and 30% time winst.

#Add
from urllib3 import PoolManager

class SubstrateInterface:

    def __init__

                #Add 
                self.http = PoolManager()


    def rpc_request(self, method, params):

                else:

#previous: response = requests.request("POST", self.url, data=json.dumps(payload), headers=self.default_headers)
response = self.http.request("POST", self.url, body=json.dumps(payload), headers=self.default_headers)

#previous: if response.status_code != 200:
if response.status != 200:
                              #previous: raise SubstrateRequestException("RPC request failed with HTTP status code {}".format(response.status_code ))
raise SubstrateRequestException("RPC request failed with HTTP status code {}".format(response.status))

#previous: json_body = response.json()
json_body = json.loads(response.data.decode('utf-8'))`

py35 no blake2b

I very much appreciate your work on this. Great!

Slight issue: python 3.5 had not blake2b in hashlib. Perhaps you find another way?

Also see this idea.

Kulupu decode Error

kulupu
substrate.get_runtime_block 176637 or 176638

error info:

raise ValueError("Index '{}' not present in Enum type mapping".format(self.index))
ValueError: Index '8' not present in Enum type mapping

Verify ed25519 support

Test and verify ed25519 support; RUST-binding are already available

Original request: polkascan/substrate-interface#3

Remove deprecated Subkey wrapper references from library

All functionality is now natively implemented so can be removed

Reuse websocket connection

I'm currently overriding the rpc_request function to be able to reuse a websocket connection, but having to use this package in more than one context in the project, makes it not convenient.
Any plans to implement this?
Thanks!

remove '==' requirements (exact version requirements)

Hello,

Is it possible to remove the '==' in requirements for versions?
It breaks any kind of application using your library, as all libraries have different version requirements.

Thanks!

signing transactions

Signing transactions - some temporary fix:

Chat result, hooray: Shawn will give us some kind of a PolkadotJS wrapper: https://riot.im/app/#/room/#polkadot-watercooler:matrix.org/$1583868900119793gprSi:matrix.parity.io

then this code

import substrateinterface 
substrate = substrateinterface.SubstrateInterface(url="ws://127.0.0.1:9944/")

BOB_ADDRESS = '5FHneW46xGXgs5mUiveU4sbTyGBzmstUspZC92UhjJM694ty'

payload = substrate.compose_call(call_module='Balances',
                                 call_function='transfer',
                                 call_params={'dest': BOB_ADDRESS,
                                              'value': 1000000000000})
print (payload.replace("0x", ""))

should generate the unsigned input payload for that signing tool.

And Balance.transfer(dest, value) is just an example, I want to send a signed transaction to my own Module. But once that Balance transfer tx signing works ... we have made a big step foward, right?

Include error message for ExtrinsicFailed events in get_runtime_events

Do a convenience lookup for the type of error in the metadata for ExtrinsicFailed events and include in the result for get_runtime_events calls

Key Derivation support

Add possibility to create hard and soft derived Keypair objects like //Alice and //Bob/nested//keys with the same mnemonic

a exception with substrate.get_storage | scalecodec-Decoder

I try to use substrate.get_storage with SubstrateInterface package, there is a error throwsthrowpython3.7/site-packages/scalecodec-0.1.0-py3.7.egg/scalecodec/base.py", line 243, in get_decoder_class NotImplementedError: Decoder class for "MetadataRecord<AccountId, Hash, Balance, Moment>" not found ,

identity = substrate.get_storage( block_hash=obj['block_hash'], module="OrlModule", function="Identity", params= address.replace('0x',''), return_scale_type=storage_call.get_return_type(), hasher=storage_call.type_hasher, metadata = ? ) Is the metadata a dict? how to set it

Refactor RuntimeConfiguration in py-scalecodec

Don't let RuntimeConfiguration be a singleton anymore, will lead to problems with multi-chain usage.

Make attribute of SubstrateInterface

RequestException happen when get_block_events

get_runtime_events() --> RemainingScaleBytesNotEmptyException

Playing with your new version, I encountered this little error:

block #629 (0xd0ab022ea2a4120a315e20b8649cea4ee4ed653d276aa00594151f2072e6d02d)
Traceback (most recent call last):
  File ".../issue_RemainingScaleBytesNotEmptyException.py", line 21, in <module>
    print(substrate.get_runtime_events(ch))
  File ".../env/lib/python3.7/site-packages/substrateinterface/__init__.py", line 826, in get_runtime_events
    block_hash=block_hash
  File ".../env/lib/python3.7/site-packages/substrateinterface/__init__.py", line 805, in get_runtime_state
    response['result'] = obj.decode()
  File ".../env/lib/python3.7/site-packages/scalecodec/base.py", line 281, in decode
    raise RemainingScaleBytesNotEmptyException('Current offset: {} / length: {}'.format(self.data.offset, self.data.length))

scalecodec.exceptions.RemainingScaleBytesNotEmptyException: Current offset: 15 / length: 19

for both of the last 2 lines in:

import substrateinterface
substrate = substrateinterface.SubstrateInterface(url="ws://127.0.0.1:9944/")
ch=substrate.get_chain_head()
chb=substrate.get_chain_block(ch)
chbn=int(chb['block']['header']['number'],16)
print("block #%s (%s)" % (chbn, ch))

print(substrate.get_runtime_events(ch))
print(substrate.get_runtime_events())

when run with the node-template 2.0.0-rc2-83d7157-x86_64-linux-gnu version of node-template:

node-template --dev

Thanks a lot!

P.S.: what are the correct (address_type, type_registry_preset) in SubstrateInterface(url, address_type, type_registry_preset) for that default node-template --dev chain, by the way? Asking because your README.md example chooses (42, 'kusama') and I would like to instead of kusama go with that default node-template --dev - and I suspect that needs different parameters?

P.P.S.: Nice to have would be:

import scalecodec, substrateinterface
print (substrateinterface.__version__, scalecodec.__version__)

(FR) let the function do the xxhash'ing of the storage key name

Hi! I am trying out your library. Very nice, that should save me time.

Without unittests, however it took time to get this one working:

get_storage_by_key (block_hash, storage_key)

In the end, I managed:

get_storage_by_key(storage_key='0x50a63a871aced22e88ee6466fe5aa5d9', block_hash=0x454...)

'0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d'

which is the result from Shawn's tutorial.

My feature request:

get_storage_by_key (block_hash, storage_key_name='Sudo Key')

and then the xxh6464('Sudo Key') is done within that function.

thanks for considering it.

Update refCount to u32

paritytech/substrate#7164

Get fee estimate before submiting transfer extrinsic

Is there any way to get a fee estimate for an extrinsic without actually submitting it?
AFAIK the JS wrapper provides such an option: https://polkadot.js.org/docs/api/cookbook/tx/#how-do-i-estimate-the-transaction-fees.
Any plans to implement the same here?

How to pass the wasm file and abi to contracts.putCode and contracts.instantiate?

How can we pass wasm and abi files?

An example when I try to instantiate:

call = substrate.compose_call(
    call_module='Contracts',
    call_function='instantiate',
    call_params={
        'endowment': 100000000000000000,
        'code_hash': '0x238914d69cec97bb70fcf61b30827fcc9edccb13d76d6eeec8e54efb0560f438',
        'gas_limit': 1000000000000,
        'data': fh.hex()
    }
)

The above code hash I got from manually uploading the wasm and abi through the UI.

Also I'd like to know how to pass the bytes to 'data' in Contracts.putCode.
Thanks!

Copyright notice is only 2018 - 2019

Only a minor issue but I just noticed the date range while browsing the repo.

e.g.
https://github.com/polkascan/py-substrate-interface/blob/master/substrateinterface/__init__.py

Verification Error: Execution(ApiError("Could not convert parameter `tx` ...

error message:

sending from: 5HmubXCdmtEvKmvqjJ7fXkxhPXcg6JTS62kMMphqxpEE6zcG
Failed to send: <class 'substrateinterface.exceptions.SubstrateRequestException'> with args:
{'code': 1002,
 'data': 'RuntimeApi("Execution(ApiError(\\"Could not convert parameter `tx` '
         'between node and runtime: No such variant in enum '
         'MultiSignature\\"))")',
 'message': 'Verification Error: Execution(ApiError("Could not convert '
            'parameter `tx` between node and runtime: No such variant in enum '
            'MultiSignature"))'}

when using the same 2.0.0-rc2 substrate dev chain node-template as in #13

node-template purge-chain -y --dev
node-template --dev

2020-06-10 23:39:38 Substrate Node
2020-06-10 23:39:38 ✌️  version 2.0.0-rc2-83d7157-x86_64-linux-gnu
2020-06-10 23:39:38 📋 Chain specification: Development

this python code is almost identical to your README.md example

from substrateinterface import SubstrateInterface, Keypair, SubstrateRequestException
from pprint import pformat
substrate = SubstrateInterface( url="ws://127.0.0.1:9944" )
keypair = Keypair.create_from_mnemonic('episode together nose spoon dose oil faculty zoo ankle evoke admit walnut')
print ("sending from:", keypair.ss58_address)
BOB_ADDRESS = '5FHneW46xGXgs5mUiveU4sbTyGBzmstUspZC92UhjJM694ty'
call = substrate.compose_call(
    call_module='Balances', call_function='transfer',
    call_params={
        'dest': BOB_ADDRESS,
        'value': 1 * 10**12})
try:
    extrinsic = substrate.create_signed_extrinsic(call=call, keypair=keypair)
    result = substrate.send_extrinsic(extrinsic, wait_for_inclusion=True)
    print("Extrinsic '{}' sent and included in block '{}'".format(result['extrinsic_hash'], result['block_hash']))
except (SubstrateRequestException, ValueError) as e:
    print("Failed to send: {} with args:".format(type(e)))
    print("{}".format(pformat(e.args[0])))

What am I doing wrong?

Thanks a lot!

P.S.: In subkey, we can access also the predefined accounts (Alice, Bob, Charlie, etc) by their aliases. It would be nice to be able to do something like this:

keypair = Keypair.create_from_private_key('//Alice')
keypair = Keypair.create_from_mnemonic('//Alice')
keypair = Keypair.create_from_alias('Alice')

Thanks!

Auto discovery of needed configuration like ss58Format and type registry

substrate = SubstrateInterface(
    url='http://127.0.0.1:9933',
    address_type=2,
    type_registry_preset='kusama'
)

address_type and type_registry_preset could be determined by a system_properties and system_chain RPC call during init

Decoding issue on Westend network

Hi,
I'm using the latest version of this package - substrate-interface==0.9.30
I get scalecodec.exceptions.RemainingScaleBytesNotEmptyException: No more bytes available (offset: 960 / length: 957)
for the call get_runtime_block on block 2937574 on Westend network.
Stacktrace:

    block_data = self._client.get_runtime_block(block_hash=block_hash)
  File "/app_dir/env/lib/python3.7/site-packages/substrateinterface/__init__.py", line 1739, in get_runtime_block
    extrinsic_decoder.decode()
  File "/app_dir/env/lib/python3.7/site-packages/scalecodec/base.py", line 280, in decode
    self.value = self.process()
  File "/app_dir/env/lib/python3.7/site-packages/scalecodec/block.py", line 163, in process
    arg_type_obj = self.process_type(arg.type, metadata=self.metadata)
  File "/app_dir/env/lib/python3.7/site-packages/scalecodec/base.py", line 325, in process_type
    obj.decode(check_remaining=False)
  File "/app_dir/env/lib/python3.7/site-packages/scalecodec/base.py", line 280, in decode
    self.value = self.process()
  File "/app_dir/env/lib/python3.7/site-packages/scalecodec/types.py", line 434, in process
    result[key] = self.process_type(data_type, metadata=self.metadata).value
  File "/app_dir/env/lib/python3.7/site-packages/scalecodec/base.py", line 325, in process_type
    obj.decode(check_remaining=False)
  File "/app_dir/env/lib/python3.7/site-packages/scalecodec/base.py", line 280, in decode
    self.value = self.process()
  File "/app_dir/env/lib/python3.7/site-packages/scalecodec/types.py", line 779, in process
    element = self.process_type(self.sub_type, metadata=self.metadata)
  File "/app_dir/env/lib/python3.7/site-packages/scalecodec/base.py", line 325, in process_type
    obj.decode(check_remaining=False)
  File "/app_dir/env/lib/python3.7/site-packages/scalecodec/base.py", line 280, in decode
    self.value = self.process()
  File "/app_dir/env/lib/python3.7/site-packages/scalecodec/types.py", line 434, in process
    result[key] = self.process_type(data_type, metadata=self.metadata).value
  File "/app_dir/env/lib/python3.7/site-packages/scalecodec/base.py", line 325, in process_type
    obj.decode(check_remaining=False)
  File "/app_dir/env/lib/python3.7/site-packages/scalecodec/base.py", line 287, in decode
    'No more bytes available (offset: {} / length: {})'.format(self.data.offset, self.data.length))
scalecodec.exceptions.RemainingScaleBytesNotEmptyException: No more bytes available (offset: 960 / length: 957)

Failed to send

Try to create and send signed extrinsic using example in readme with modification to using polkadot mainnet:

from substrateinterface import SubstrateInterface, SubstrateRequestException, Keypair

substrate = SubstrateInterface(url="ws://127.0.0.1:9944", address_type=0)

keypair = Keypair.create_from_mnemonic(MY_MNEMONIC, address_type=0)

call = substrate.compose_call(
    call_module='Balances',
    call_function='transfer',
    call_params={
        'dest': DEST_ACCOUNT
        'value': AMOUNT * 10**10
    }
)

extrinsic = substrate.create_signed_extrinsic(call=call, keypair=keypair)

try:
    result = substrate.submit_extrinsic(extrinsic, wait_for_inclusion=True)
    print("Extrinsic '{}' sent and included in block '{}'".format(result['extrinsic_hash'], result['block_hash']))

except SubstrateRequestException as e:
    print("Failed to send: {}".format(e))

But got an error:

Failed to send: {'code': 1002, 'message': 'Verification Error: Execution: Trap: Trap { kind: Unreachable }', 'data': 'RuntimeApi("Execution: Trap: Trap { kind: Unreachable }")'}

Operate with a contract that deployed on the substrate

Hi,

I have a tutorial smart contract 'flipper' deployed on my substrate chain.
contract name and function:
wasm: FLIPPER.WASM (INSTANCE)
fn: flip(), get()

How can I operate with the contract call by this lib? Seems like using

call = substrate.compose_call(
    call_module=xxxx,
    call_function='flip',
    call_params={
        'dest': '5E9oDs9PjpsBbxXxRE9uMaZZhnBAV38n2ouLB28oecBDdeQo',
        'value': 1 * 10**12
    }
)

shall I use the 'FLIPPER.WASM' as the call_module?

Contract instantiate produces OutputBufferTooSmall error

Hi Arjan, it's probably a substrate question, not the interface, but I would like to know what causes this OutputBufferTooSmall error when I try to run

call = substrate.compose_call(
    call_module='Contracts',
    call_function='instantiate',
    call_params={
        'endowment': 1000000,
        'gas_limit': 1000000000,
        'code_hash': code_hash,
        'data': '0x{}'.format(code_bytes.hex())
    }
)

is it something about the data? What goes into data, I didn't find in the docs. Not the contract code I assume, as it already was passed with the contracts.put_code ..?

Transactions,Total module events,Runtime Versions,Block time,Session ID,Block author

"Transactions,Total module events,Runtime Versions,Block time,Session ID,Block author", how to obtain these contents, pls help，link address:https://polkascan.io/kusama/block/3269038#inherents

Storage calls on Westend network fail

Hi,
I'm making a call like was described in the README on the Westend network:

balance_info = substrate.get_runtime_state(
    module='System',
    storage_function='Account',
    params=['5FQhydxUyKtYeo9M4DsX3Cogm1u6bKF1KSwFikxhmFfydV7m'],
    block_hash="0x16a8af5d8f1d9a6adcc813dc2e77e58c315da5304378207e3037a25857a09922"
).get('result')

And get the following error (doesn't happen in Polkadot or in any other (non-storage) Westend call):

scalecodec.exceptions.RemainingScaleBytesNotEmptyException: Current offset: 69 / length: 72

Debugging I've found out that MetadataV12Decoder is used. The version substrate-interface is 0.9.25 and scalecodec 0.10.26.
Please advise what can be done and whether this is really an issue.

Thanks,
Pavel

Decoder class for "ExtrinsicMetadata" not found

I connect to a private substrate node where I use docker to build it.
> docker run -p 9944:9944 -it --rm parity/substrate:latest --dev --ws-external --rpc-external
and try the code below:

import substrateinterface
substrate = substrateinterface.SubstrateInterface(url="ws://127.0.0.1:9944/")
block_hash = substrate.get_block_hash(1)
print(block_hash)
get_runtime_block = substrate.get_runtime_block(block_hash=None)

I can get the hash of block 1 but I cannot get runtime block. The error message shows "ExtrinsicMetadata" not found. Is this issue with substrate?

>python3 poe_access.py 0xfc496269de21da17235e69329d33411805e1564f968a12f8522f8c257c850665 Traceback (most recent call last): File "poe_access.py", line 6, in <module> get_runtime_block = substrate.get_runtime_block(block_hash=None) File "/home/ubuntu/work/python_trek_env/lib/python3.6/site-packages/substrateinterface/__init__.py", line 1226, in get_runtime_block self.init_runtime(block_hash=block_hash, block_id=block_id) File "/home/ubuntu/work/python_trek_env/lib/python3.6/site-packages/substrateinterface/__init__.py", line 565, in init_runtime self.metadata_decoder = self.get_block_metadata(block_hash=self.block_hash, decode=True) File "/home/ubuntu/work/python_trek_env/lib/python3.6/site-packages/substrateinterface/__init__.py", line 301, in get_block_metadata metadata_decoder.decode() File "/home/ubuntu/work/python_trek_env/lib/python3.6/site-packages/scalecodec/base.py", line 278, in decode self.value = self.process() File "/home/ubuntu/work/python_trek_env/lib/python3.6/site-packages/scalecodec/metadata.py", line 49, in process self.metadata = self.process_type(self.version.value) File "/home/ubuntu/work/python_trek_env/lib/python3.6/site-packages/scalecodec/base.py", line 352, in process_type obj.decode(check_remaining=False) File "/home/ubuntu/work/python_trek_env/lib/python3.6/site-packages/scalecodec/base.py", line 278, in decode self.value = self.process() File "/home/ubuntu/work/python_trek_env/lib/python3.6/site-packages/scalecodec/metadata.py", line 968, in process result_data["metadata"]["MetadataV11"]["extrinsic"] = self.process_type("ExtrinsicMetadata").value File "/home/ubuntu/work/python_trek_env/lib/python3.6/site-packages/scalecodec/base.py", line 351, in process_type obj = self.get_decoder_class(type_string, self.data, **kwargs) File "/home/ubuntu/work/python_trek_env/lib/python3.6/site-packages/scalecodec/base.py", line 347, in get_decoder_class raise NotImplementedError('Decoder class for "{}" not found'.format(type_string)) NotImplementedError: Decoder class for "ExtrinsicMetadata" not found

compose_class function causes raise TypeError("'call' must be of type Call")

I suggest to change the class name check in the substrateinterface init.py file. The current code is:

# Check requirements if call.__class__.__name__ != 'Call': raise TypeError("'call' must be of type Call")

but in the scalecodec module the file types.py has this definition:

class GenericCall(ScaleType):

changing the init.py file to

# Check requirements if call.__class__.__name__ != 'GenericCall': raise TypeError("'call' must be of type GenericCall")

solves the error. Shall I do a PR? There was another guy complaining about this earlier in the issues.

Runtime update not handled

There was a runtime update on Polkadot which is causing block parsing to fail. First failed on block 2436698.
Stacktrace:

block_data = self._client.get_runtime_block(block_hash=block_hash)
  File "/app_dir/env/lib/python3.7/site-packages/substrateinterface/__init__.py", line 1739, in get_runtime_block
    extrinsic_decoder.decode()
  File "/app_dir/env/lib/python3.7/site-packages/scalecodec/base.py", line 280, in decode
    self.value = self.process()
  File "/app_dir/env/lib/python3.7/site-packages/scalecodec/block.py", line 158, in process
    self.call = self.metadata.call_index[self.call_index][1]
KeyError: '0a00'

Are there any plans to support such network upgrades in some automatic process?

compose_call only works with dest=PUB_KEY

When playing with

compose_call( Balances, transfer, {'dest': dest,'value': value} )

I got a lot of errors like

ValueError: Invalid checksum
ValueError: Invalid address length
ValueError: Invalid account index length

until I realized that I cannot (?) send to an

address (5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY) 

but must use the

PUB_KEY (0xd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d)

instead, for the dest:

payload = substrate.compose_call(call_module='Balances',
                                 call_function='transfer',
                                 call_params={'dest': dest,
                                              'value': value})

I suggest, you mention that the https://github.com/polkascan/py-substrate-interface#compose-call explanations?

Add Python 3.9 support

Currently no RUST bindings for Python 3.9 are available so this stalls installation via pip

AttributeError: module 'asyncio' has no attribute 'run'

Transaction info when constructing the transaction offline

In the polkadot documentation they propose to construct the transaction this way:

import { methods } from "@substrate/txwrapper";

const unsigned = methods.balances.transferKeepAlive(
  {
    dest: "15vrtLsCQFG3qRYUcaEeeEih4JwepocNJHkpsrqojqnZPc2y",
    value: 500000000000,
  },
  {
    address: "121X5bEgTZcGQx5NZjwuTjqqKoiG8B2wEAvrUFjuw24ZGZf2",
    blockHash: "0x1fc7493f3c1e9ac758a183839906475f8363aafb1b1d3e910fe16fab4ae1b582",
    blockNumber: 4302222,
    genesisHash: "0xe3777fa922cafbff200cadeaea1a76bd7898ad5b89f7848999058b50e715f636",
    metadataRpc, // must import from client RPC call state_getMetadata
    nonce: 2,
    specVersion: 1019,
    tip: 0,
    eraPeriod: 64, // number of blocks from checkpoint that transaction is valid
    transactionVersion: 1,
  },
  {
    metadataRpc,
    registry, // Type registry
  }
);

The dest and value parameters are indeed the parameters of the transfer function, but the additional info parameter doesn't appear in this library compose_call function. I assume not all is needed, but at least the tip and the source address should be set by me, no? Is it possible to construct it the same way in the current state of the function?

'call' must be of type Call

Traceback (most recent call last):
File "/home/wjy/桌面/Transx_Gitlab正式版本/transx/transx/target/release/test.py", line 33, in
extrinsic = substrate.create_signed_extrinsic(call=call, keypair=keypair)
File "/home/wjy/桌面/ENV/lib/python3.7/site-packages/substrateinterface/init.py", line 960, in create_signed_extrinsic
raise TypeError("'call' must be of type Call")
TypeError: 'call' must be of type Call
<class 'scalecodec.types.GenericCall'>

there are my code:

substrate = SubstrateInterface(
    url="ws://47.108.199.133:9945",
    address_type=42,
    type_registry_preset='default'
)
mnemonic = Keypair.generate_mnemonic()

keypair = Keypair.create_from_mnemonic(mnemonic, 2)

print("Created address: {}".format(keypair.ss58_address))
call = substrate.compose_call(
    call_module='Balances',
    call_function='transfer',
    call_params={
        'dest': '5GnGKSCitk1QPpMNugtTGX9t6TqzDGvL5BqKzLfHNsLSrwqN',
        'value': 100 * 10**14
    }
)

MultiAddress support

Support for the in Substrate introduced MultiAddress to sign/decode extrinsics:

pub enum MultiAddress<AccountId, AccountIndex> {
	/// It's an account ID (pubkey).
	Id(AccountId),
	/// It's an account index.
	Index(#[codec(compact)] AccountIndex),
	/// It's some arbitrary raw bytes.
	Raw(Vec<u8>),
	/// It's a 32 byte representation.
	Address32([u8; 32]),
	/// Its a 20 byte representation.
	Address20([u8; 20]),
}