Verification Error: Execution(ApiError("Could not convert parameter `tx` ... about py-substrate-interface

[arjanz]

Then I suspect you have a slightly older version of Substrate, recently the signing payload added the field 'transactionVersion'. (You can check this by inspecting the 'chain_getRuntimeVersion' RPC call, recently 'transactionVersion' is added there)

What you can try is the following:

custom_type_registry = {
  "runtime_id": 1,
  "types": {
    "ExtrinsicPayloadValue": {
      "type": "struct",
      "type_mapping": [
        ["call", "CallBytes"],
        ["era", "Era"],
        ["nonce", "Compact<Index>"],
        ["tip", "Compact<Balance>"],
        ["specVersion", "u32"],
        ["genesisHash", "Hash"],
        ["blockHash", "Hash"]
      ]
    }
  },
  "versioning": [
  ]
}

substrate = SubstrateInterface(
    url="http://127.0.0.1:9933",
    address_type=42,
    type_registry_preset='default',
    type_registry=custom_type_registry
)

This uses the older version of the signing payload. Alternatively you can place this together with all your custom types in a separate JSON file and use:

custom_type_registry = load_type_registry_file("custom_type_registry.json")

On our todo list is to support backwards compatibility for this kind of changes, but this isn't trivial to realize for now.

[arjanz]

sending from: 5HmubXCdmtEvKmvqjJ7fXkxhPXcg6JTS62kMMphqxpEE6zcG
Failed to send: <class 'substrateinterface.exceptions.SubstrateRequestException'> with args:
{'code': 1002,
'data': 'RuntimeApi("Execution(ApiError(\"Could not convert parameter tx '
'between node and runtime: No such variant in enum '
'MultiSignature\"))")',
'message': 'Verification Error: Execution(ApiError("Could not convert '
'parameter tx between node and runtime: No such variant in enum '
'MultiSignature"))'}

I reproduced this error, the Address format in the 'default' type registry is still in an outdated format, can you try the following:

custom_type_registry = {
  "runtime_id": 1,
  "types": {
    "Address": "AccountIdAddress",
    "ExtrinsicPayloadValue": {
      "type": "struct",
      "type_mapping": [
        ["call", "CallBytes"],
        ["era", "Era"],
        ["nonce", "Compact<Index>"],
        ["tip", "Compact<Balance>"],
        ["specVersion", "u32"],
        ["genesisHash", "Hash"],
        ["blockHash", "Hash"]
      ]
    }
  },
  "versioning": [
  ]
}

After we figure out to work with current substrate-node-template, we will make a dedicated type_registry_preset and pin the version of substrate-node-template to make it easier to get started.

[arjanz]

Still, I am trying your code, but get this:

substrateinterface.exceptions.ConfigurationError:
Result handlers only available for websockets (ws://) connections

so changing to

    substrate = SubstrateInterface(url="ws://127.0.0.1:9944",
                                   address_type=42,
                                   type_registry_preset='default',
                                   type_registry=custom_type_registry)

right?

Yes, or leave out the wait_for_inclusion=True keyword arg, this will work for http. The wait_for_inclusion and wait_for_finalization subscribes to the `submitExtrinsic' and will also return the block hash. Without these keyword args, there is no confirmation the extrinsic is successfully included =, but the extrinsic hash can still be used to for example create a link to Polkascan

[alexspurling]

@arjanz Thank you for your work maintaining this library and making sure it continues to work across new releases of substrate, polkadot etc. After having multiple issues connecting to various different versions of various different substrate nodes, I was able to get your example code working. However, I really don't understand how it is sustainable to rely on you personally updating all the .json type registry files in the scalecodec library (https://github.com/polkascan/py-scale-codec/tree/master/scalecodec/type_registry). Do you have a plan for the future to make this process automatic or somehow have the py-substrate-interface library query the node itself rather than pull types from github.com? How do you discover the necessary changes to the type registries anyway? How will you handle support for older versions? Sorry I'm not sure where else to ask this question.

[arjanz]

Basically I skim the updates in https://github.com/paritytech/substrate pallets and look for type changes and put those types in default.json. For example the PreimageStatus enum in the democracy pallet (https://github.com/paritytech/substrate/blob/master/frame/democracy/src/lib.rs#L309).

Then when a runtime is pinned and scheduled for release for Polkadot, Kusama, etc. I make an entry with runtime id and changes necessary for versioning in polkadot.json, kusama.json etc.

I agree this is a bit cumbersome but the only way to let the library know how to decode certain types that are not primitives.

Note that the upgrade runtime WASM is provided on-chain and will be auto-enacted on your node even if you don't update your node. I believe the runtime that is provided by the node natively is faster than if your node will run the synced on-chain runtime WASM, so it is advised to update frequently, but you will always be up to date with the latest runtime. For more info see https://wiki.polkadot.network/docs/en/build-node-management and https://wiki.polkadot.network/docs/en/learn-wasm#forkless-upgrades

[alexspurling]

Thanks, @arjanz . Are the changes to the types published somewhere ahead of time or do you just have to dig around in the substrate code to find them?

[drandreaskrueger]

updated to scalecodec-0.9.54. Now the error has changed:

ValueError: Element "transactionVersion" of struct is missing in given value

and it happens already in substrate.create_signed_extrinsic()

[drandreaskrueger]

Ah, thanks a lot. Much appreciated.

So the

version 2.0.0-rc2-83d7157-x86_64-linux-gnu

is outdated now, I suppose?

If I drop that rc2, and compile rc3 instead ... that problem will be gone too, right?

I really don't cling to any specific version, I just don't want to upgrade every 3 weeks, if possible. Looks like rc2 had a lifetime of 15 days lol --> https://github.com/substrate-developer-hub/substrate-node-template/tags

Will that rc3 and its tool support last me for a while then?

[drandreaskrueger]

Still, I am trying your code, but get this:

substrateinterface.exceptions.ConfigurationError:
Result handlers only available for websockets (ws://) connections

so changing to

    substrate = SubstrateInterface(url="ws://127.0.0.1:9944",
                                   address_type=42,
                                   type_registry_preset='default',
                                   type_registry=custom_type_registry)

right?

[drandreaskrueger]

getting this error:

sending from: 5HmubXCdmtEvKmvqjJ7fXkxhPXcg6JTS62kMMphqxpEE6zcG
Failed to send: <class 'substrateinterface.exceptions.SubstrateRequestException'> with args:
{'code': 1002,
 'data': 'RuntimeApi("Execution(ApiError(\\"Could not convert parameter `tx` '
         'between node and runtime: No such variant in enum '
         'MultiSignature\\"))")',
 'message': 'Verification Error: Execution(ApiError("Could not convert '
            'parameter `tx` between node and runtime: No such variant in enum '
            'MultiSignature"))'}

using these versions

scalecodec==0.9.54
substrate-interface==0.9.14
node-template --dev ✌️  version 2.0.0-rc2-83d7157-x86_64-linux-gnu

but no worries, I will simply compile rc3 now, hoping that it will last a while then.

[drandreaskrueger]

You can check this by inspecting the 'chain_getRuntimeVersion' RPC call, recently 'transactionVersion' is added there

Just tried that. It's not part of your interface yet

grv = substrate.chain_getRuntimeVersion()
AttributeError: 'SubstrateInterface' object has no attribute 'chain_getRuntimeVersion'

but I could call it this way:

pprint(substrate.rpc_request(method="chain_getRuntimeVersion", params=[]))

and got this:

chain_getRuntimeVersion():
{'id': 1,
 'jsonrpc': '2.0',
 'result': {'apis': [['0xdf6acb689907609b', 3],
                     ['0x37e397fc7c91f5e4', 1],
                     ['0x40fe3ad401f8959a', 4],
                     ['0xd2bc9897eed08f15', 2],
                     ['0xf78b278be53f454c', 2],
                     ['0xdd718d5cc53262d4', 1],
                     ['0xab3c0572291feb8b', 1],
                     ['0xed99c5acb25eedf5', 2]],
            'authoringVersion': 1,
            'implName': 'node-template',
            'implVersion': 1,
            'specName': 'node-template',
            'specVersion': 1,
            'transactionVersion': 1}}

that ^ was for rc2. Compiling rc3 is half ready.

[drandreaskrueger]

Suggestion for your README.md - mention the version for which you can guarantee that it works:

this is tested with this version of substrate or node-template:

substrate --version; node-template --version;
substrate 2.0.0-rc2-45b9f0a9c-x86_64-linux-gnu
node-template 2.0.0-rc2-83d7157-x86_64-linux-gnu

[drandreaskrueger]

compiling rc3 ... done. This is the version answer when talking to it:

get_version() = 2.0.0-rc3-f5acce1-x86_64-linux-gnu
chain_getRuntimeVersion():
{'id': 1,
 'jsonrpc': '2.0',
 'result': {'apis': [['0xdf6acb689907609b', 3],
                     ['0x37e397fc7c91f5e4', 1],
                     ['0x40fe3ad401f8959a', 4],
                     ['0xd2bc9897eed08f15', 2],
                     ['0xf78b278be53f454c', 2],
                     ['0xdd718d5cc53262d4', 1],
                     ['0xab3c0572291feb8b', 1],
                     ['0xed99c5acb25eedf5', 2]],
            'authoringVersion': 1,
            'implName': 'node-template',
            'implVersion': 1,
            'specName': 'node-template',
            'specVersion': 1,
            'transactionVersion': 1}}

The 'transactionVersion': 1 is the same as above for the rc2.

[drandreaskrueger]

Same error remains. Here's the code to try - thanks a lot!

https://gist.github.com/drandreaskrueger/053bfe0c8ddfaeeb50134a977708f999

[drandreaskrueger]

can you try the following:

YESSS ... that new custom_type_registry made a difference = now the error messages are different:

Default SubstrateInterface:
sending from: 5HmubXCdmtEvKmvqjJ7fXkxhPXcg6JTS62kMMphqxpEE6zcG
Failed to send: <class 'substrateinterface.exceptions.SubstrateRequestException'> with args:
{'code': 1002,
 'data': 'RuntimeApi("Execution(ApiError(\\"Could not convert parameter `tx` '
         'between node and runtime: No such variant in enum '
         'MultiSignature\\"))")',
 'message': 'Verification Error: Execution(ApiError("Could not convert '
            'parameter `tx` between node and runtime: No such variant in enum '
            'MultiSignature"))'}

SubstrateInterface with custom type registry:
sending from: 5HmubXCdmtEvKmvqjJ7fXkxhPXcg6JTS62kMMphqxpEE6zcG
Failed to send: <class 'substrateinterface.exceptions.SubstrateRequestException'> with args:
{'code': 1010, 'data': 'BadProof', 'message': 'Invalid Transaction'}

[drandreaskrueger]

Yes, or leave out the wait_for_inclusion=True keyword arg

Ah yes, that makes sense. Thanks.

subscribes to the `submitExtrinsic' and will also return the block hash. Without these keyword args, there is no confirmation the extrinsic is successfully included ...

Without the block_hash returned, perhaps a slight change to result.get('block_hash', None)) :

    # result = substrate.submit_extrinsic(extrinsic, wait_for_inclusion=True) # works only with ws not http
    result = substrate.submit_extrinsic(extrinsic) # works with http AND ws
    print("Extrinsic '{}' sent and included in block '{}'".format(result['extrinsic_hash'], result.get('block_hash', None)))

[arjanz]

can you try the following:

YESSS ... that new custom_type_registry made a difference = now the error messages are different:

Default SubstrateInterface:
sending from: 5HmubXCdmtEvKmvqjJ7fXkxhPXcg6JTS62kMMphqxpEE6zcG
Failed to send: <class 'substrateinterface.exceptions.SubstrateRequestException'> with args:
{'code': 1002,
 'data': 'RuntimeApi("Execution(ApiError(\\"Could not convert parameter `tx` '
         'between node and runtime: No such variant in enum '
         'MultiSignature\\"))")',
 'message': 'Verification Error: Execution(ApiError("Could not convert '
            'parameter `tx` between node and runtime: No such variant in enum '
            'MultiSignature"))'}

SubstrateInterface with custom type registry:
sending from: 5HmubXCdmtEvKmvqjJ7fXkxhPXcg6JTS62kMMphqxpEE6zcG
Failed to send: <class 'substrateinterface.exceptions.SubstrateRequestException'> with args:
{'code': 1010, 'data': 'BadProof', 'message': 'Invalid Transaction'}

Since the upgrade to rc3 you probably need to include transactionVersion again in the ExtrinsicPayloadValue? (so remove that type from the custom_type_registry)

Error BadProof means the signature is not accepted

[drandreaskrueger]

Thx a lot. I have updated the gist:

https://gist.github.com/drandreaskrueger/053bfe0c8ddfaeeb50134a977708f999/revisions

(substrate rc3 is still compiling)

[drandreaskrueger]

Since the upgrade to rc3 you probably need to include transactionVersion again in the ExtrinsicPayloadValue? (so remove that type from the custom_type_registry)

Remove ExtrinsicPayloadValue completely, like this?

custom_type_registry = {  "runtime_id": 1,
                          "types": {
                            "Address": "AccountIdAddress",
                          },
                          "versioning": [
                          ]
                        }

that doesn't solve it. I misunderstood you, right?

---

please you try again:

now both results (bottom) are done with rc3
https://gist.github.com/drandreaskrueger/053bfe0c8ddfaeeb50134a977708f999

Thanks!

[drandreaskrueger]

OMG ... every substrate chain color needs a different type_registry ?

I have just found this folder: scalecodec/type_registry That is ... complicated. Ouch.

Shouldn't the "chain maker" at least make sure to PUBLISH such a type_registry/<MYCHAINNAME>.json file?
Or better: Provide an RPC answer on some TCP IP port that results in exactly that information?

As I understand it now from your code examples, you already need that information when you initiate the connection with SubstrateInterface(url=URL, address_type=42, type_registry_preset='default', type_registry=custom_type_registry) , right? If there was such a port xyz RPC answer, that class initiation could call the node, get the full type_registry from the node, and configure itself. Would that work? Just an idea, how to make your and everyone's life easier. One can dream, right? :-) But back to the harsh reality:

I suppose the substrate default chain (rc3 ? ) uses default.json ?

Please you run your example README.md code, as it is now part of this https://gist.github.com/drandreaskrueger/053bfe0c8ddfaeeb50134a977708f999 .
As you already have a should-be-working type_registry ready for the default substrate (before we look into node-template) let's first get substrate running. Both rc3.

Thanks a lot. Much appreciate your help!

[drandreaskrueger]

Actually ...

Shouldn't the "chain maker" ... provide an RPC answer on some TCP IP port that results in exactly that information?

isn't that what get_metadata_call_functions() or get_metadata_storage_functions() does?

Or what is your type_registry ?

[drandreaskrueger]

I have added a "quickstart" to the gist, for anyone else trying to help:

https://gist.github.com/drandreaskrueger/053bfe0c8ddfaeeb50134a977708f999#file-compose_sign_and_send_extrinsic-substrate-L22-L29

Thanks

[drandreaskrueger]

Made simplifications, clarifications; reran with newest scalecodec version, updating the output RESULTS - but ~same outcome. Anyways, slightly newer code, same place: https://gist.github.com/drandreaskrueger/053bfe0c8ddfaeeb50134a977708f999

[drandreaskrueger]

https://riot.im/app/#/room/!HzySYSaIhtyWrwiwEV:matrix.org/$159308184511859xHcjX:matrix.org

EDIT: scrolled through that whole chat on 29/6/2020 - no answer yet.

[polkadev]

@arjanz @drandreaskrueger

I am develop Himalia ,

so use py-substrate-interface for contracts,
i use this to success commit extrinsic to canvas-node :

custom_type_registry = {
  "runtime_id": 1,
  "types": {
    "Address": "AccountIdAddress",
    "ExtrinsicPayloadValue": {
      "type": "struct",
      "type_mapping": [
        ["call", "CallBytes"],
        ["era", "Era"],
        ["nonce", "Compact<Index>"],
        ["tip", "Compact<Balance>"],
        ["specVersion", "u32"],
        ["transactionVersion", "u32"],
        ["genesisHash", "Hash"],
        ["blockHash", "Hash"]
      ]
    }
  },
  "versioning": [
  ]
}

substrate = SubstrateInterface(
    url="ws://localhost:9944",
    type_registry_preset='default',
    type_registry=custom_type_registry
)


keypair = Keypair.create_from_uri('//Alice')

# Upload WASM code
code = ContractCode.create_from_contract_files(
    metadata_file=os.path.join(os.path.dirname(__file__), 'erc20.json'),
    wasm_file=os.path.join(os.path.dirname(__file__), 'erc20.wasm'),
    substrate=substrate
)

receipt = code.upload_wasm(keypair)

Note : the current payload struct has a params:

 ["transactionVersion", "u32"],

@drandreaskrueger can try by this.

Note the substract : keyring.rs

...

let payload = (xt.function, extra.clone(), spec_version, tx_version, genesis_hash, genesis_hash);
let key = AccountKeyring::from_account_id(&signed).unwrap();

...

[arjanz]

Hi @polkadev, thanks for your feedback. Did you also try to use type_registry_preset='canvas'? There I already put AccountIdAddress as Address. The ExtrinsicPayloadValue seems the same as in default.json, am I missing something?

Anyway cool you are using the contract interface, note this is very recent work and needs some more testing, so I would appreciate further feedback from you!

[arjanz]

@polkadev latest release should add compatibility with latest substrate 2.0.0 and canvas-node

Also see: https://github.com/polkascan/py-substrate-interface/blob/master/examples/create_and_exec_erc20_contract.py

[arjanz]

@alexspurling This is a very legitimate question and I can only agree with you that putting the responsibility of type compatibility of numerous chains on the maintainers of the libraries (Polkadot-JS for example also manually administer this) is not a sustainable solution. But unfortunately at this moment there is no way to query the Substrate node how their runtime types are decomposed.

I have been lobbying for this feature just like it is possible to query the metadata in order to get feature discovery with all calls, events etc, but as far as I know automatic type decomposition is not on the planning yet.

I have seen though that automatic type decomposition all the way down to primitive types is realized for the Ink! contract pallet, so I hope this feature will come to Substrate runtime wide any time soon.

[arjanz]

I think the best way to lobby for this feature is to make yourself heard and state this once again in the Substrate Github issue tracker.

Regarding your question about type versioning, I do keep a versioning trail of changed types per runtime and this will be automatically applied when querying historic state.

[alexspurling]

Could you explain exactly how you extract type information if it is not available as part of any of the metadata APIs?

Also, how exactly do you track changes between versions? In the scalecodec repo, there is only a single json file for each chain. I can see the repo has versioned tags I don't think a single version number can track changes across multiple chains.

For example, let's say I am running a polkadot node based on v0.8.2 and used your py-substrate-interface in my client code in order to create and sign transactions. At some point in the future polkadot release a new version of their node which includes some type changes. Let's say that I choose not to update my node, however if at any point my client code runs substrate.update_type_registry_presets() then it will update to the latest type registry for polkadot and stop working with my old node. Is there any way that I can "fix" the version that I want to use for my type registry?

[leeduckgo]

Problem when I using canvas local dev node to deploy ink! contract

[arjanz]

Problem when I using canvas local dev node to deploy ink! contract

I think you're experiencing an issue not with this library but with Polkadot-JS (https://github.com/polkadot-js/). Have you tried updating to latest version? At a glance it seems to be type registry related.

[leeduckgo]

Problem when I using canvas local dev node to deploy ink! contract

I think you're experiencing an issue not with this library but with Polkadot-JS (https://github.com/polkadot-js/). Have you tried updating to latest version? At a glance it seems to be type registry related.

I used the canvas-ui online and it seems updated just five days ago.

https://github.com/paritytech/canvas-ui/deployments

[arjanz]

@leeduckgo then I advise you to post an issue at https://github.com/paritytech/canvas-ui

[leeduckgo]

@leeduckgo then I advise you to post an issue at https://github.com/paritytech/canvas-ui

thx, i have already post issue there