Comments (10)
We will need a tool to compare the weights. I am not going to review all the numbers and check if all of them are close enough.
@ggwpez already has a tool for this! He can share this. I was also not expecting for you to do this manually :D
That way weights generation could be automated and the results also trusted without manual rechecking (depending on the level of control & trust over the provided infra).
This is the same trust level as trusting the guy that opens the PR. But yeah, theoretically it would work. However, this increases then the maintenance burden etc. Even in Parity all this stuff isn't working smoothly all the time. Having just one script that you can run on a machine sounds fine for me right now.
from runtimes.
@bkontur FWIW I'm using this branch to do benchmarks locally. So something like:
cargo build --release -p chain-spec-generator --features runtime-benchmark
./target/release/chain-spec-generator bridge-hub-polkadot-local --raw >bridge-hub-polkadot-local-raw.json
../polkadot-sdk/target/release/polkadot-parachain-benchmarks benchmark pallet --chain bh-polkadot-local-raw.json --pallet pallet-bridge-grandpa --extrinsic "*"
where thepolkadot-parachain-benchmarks
is thepolkadot-parachain
binary, compiled with--features runtime-benchmark
UPD: forgot to add thatyou need to use smth like bh-polkadot-local-raw.json
for file name because of this. Think we need to change later
from runtimes.
- How and where to regenerate weights for this repo?
Anyone can do this using the standard hardware. It should be documented here prominently what the standard hardware is. Then people also verify that the posted weights are in the same ball park, aka correct.
- What
binary
version to use? Benchmark commands are insidepolkadot
orpolkadot-parachain
binary)
Use the latest version and add a comment into the pull request about the version to make this process reproducible.
- When to regenerate weight? Manually? On every release? On master?
Manually. Fellowship doesn't has any infrastructure and depends on contributors to run the benchmarks. The fellowship members approving the pull request are then required to reproduce the weights to ensure that they are correct.
from runtimes.
We will need a tool to compare the weights. I am not going to review all the numbers and check if all of them are close enough.
I already mentioned this in another issue, we should have the benchmark output a csv or json file so that we can easily process those numbers.
from runtimes.
Manually. Fellowship doesn't has any infrastructure and depends on contributors to run the benchmarks. The fellowship members approving the pull request are then required to reproduce the weights to ensure that they are correct.
Would it be interesting to explore if someone (some-company or some chain-collective) would sponsor some infrastructure for this?
That way weights generation could be automated and the results also trusted without manual rechecking (depending on the level of control & trust over the provided infra).
from runtimes.
most of the devs won’t have access to a standard hardware machine locally so we will also need some guides on setting up one cloud providers
from runtimes.
We have already this: https://wiki.polkadot.network/docs/maintain-guides-how-to-validate-polkadot
from runtimes.
@ggwpez already has a tool for this! He can share this. I was also not expecting for you to do this manually :D
I believe this is the tool: https://github.com/ggwpez/substrate-weight-compare
from runtimes.
Nice. Now we just need to have everything documented in a single place.
from runtimes.
I believe this is the tool: https://github.com/ggwpez/substrate-weight-compare
Yes indeed. For example to get a diff on the ref-time component:
subweight compare commits --path-pattern "*/**/weights/*.rs" --method asymptotic --ignore-errors 1183e6b06108bf053c08e26c4c50fc644f572c7a 5312070b39a7529828c6b764f9c4f3ced206cfc3 --threshold 30 --unit time
And for the PoV size there is --unit proof
. Output in markdown format looks like this:
File | Extrinsic | Old | New | Change [%] |
---|---|---|---|---|
system-parachains/asset-hubs/asset-hub-kusama/src/weights/pallet_assets_local.rs | destroy_approvals | 40.80ms | 139.20ms | +241.15 |
relay/polkadot/src/weights/pallet_staking.rs | cancel_deferred_slash | 5.89ms | 7.80ms | +32.29 |
relay/kusama/src/weights/pallet_staking.rs | cancel_deferred_slash | 5.92ms | 7.77ms | +31.40 |
relay/kusama/src/weights/pallet_xcm.rs | migrate_and_notify_old_targets | 921.58us | 639.08us | -30.65 |
relay/kusama/src/weights/pallet_election_provider_multi_phase.rs | create_snapshot_internal | 2.40ms | 1.66ms | -30.66 |
relay/kusama/src/weights/pallet_xcm_benchmarks_fungible.rs | transfer_reserve_asset | 906.43us | 619.93us | -31.61 |
relay/kusama/src/weights/pallet_balances_nis_counterpart_balances.rs | force_set_balance_killing | 400.80us | 270.86us | -32.42 |
relay/kusama/src/weights/pallet_xcm.rs | force_unsubscribe_version_notify | 847.29us | 564.77us | -33.34 |
relay/polkadot/src/weights/pallet_xcm.rs | send | 513.88us | 330.72us | -35.64 |
relay/kusama/src/weights/pallet_xcm.rs | notify_current_targets | 764.12us | 482.12us | -36.91 |
relay/kusama/src/weights/pallet_xcm_benchmarks_fungible.rs | deposit_reserve_asset | 756.32us | 472.16us | -37.57 |
relay/kusama/src/weights/pallet_xcm.rs | send | 612.26us | 329.64us | -46.16 |
relay/kusama/src/weights/pallet_xcm_benchmarks_fungible.rs | initiate_teleport | 883.42us | 474.10us | -46.33 |
relay/kusama/src/weights/pallet_xcm_benchmarks_fungible.rs | receive_teleported_asset | 274.06us | 146.28us | -46.62 |
There is also a web version, but i need to re-setup the server and add the runtimes repo. But it is possible to self-host as described in the readme.
Currently we have no trivial way to re-generate the weights. Sure there is the wiki description for the reference hardware and then anyone can run it, but its not straight forward, since the chain specs need to be generated and the node compiled first.
from runtimes.
Related Issues (20)
- [Ambassadors] Reduce `MaxRank` to 3 HOT 1
- `Fail when author pushes new code` should check if the user is not a fellow HOT 1
- Optimistic Project Funding HOT 22
- Implement Unbrick Collective HOT 3
- [CI] Publish unoptimized runtime blobs for debugging HOT 2
- [People] Add new registrar HOT 11
- PeopleKusama, PeoplePolkadot and Encointer don't have price for parent delivery set up HOT 2
- Add fee estimation support for XCMs over P<>K and P<>E bridges
- Treasury Configuration(s) for Multi-Chain Spends HOT 1
- [bridges][AHs] Include ExportMessage fees in XCM message from AH to BH HOT 2
- Update CI for Polkadot People Chain HOT 2
- Remove `chain-spec-generator`
- Check `merge bot` and required pipelines HOT 8
- Add test for coretime Runtime API HOT 1
- Coretime Runtime API: Refactor `burn_at_relay`
- Review bot error: Reviews may only be requested from collaborators HOT 6
- Field `image` on people chain is unusable HOT 5
- Enable StorageWeightReclaim for all the runtime HOT 2
- Consider using `RuntimeGenesisConfig` instead of directly constructing json for the preset genesis patch
- Actually benchmark `pallet_xcm::execute` for all runtimes HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from runtimes.