paritytech / capi Goto Github PK

We currently hand-type RPC methods in rpc/messages.ts. This is error-prone, as we'll need expend ongoing manual effort to keep these declarations in sync with the Rust source of truth. Also, we want to enable different combinations of these methods / custom methods depending on specified chain (related to #108).

An ideal solution would involve generating these type-level lookups. Aka. something akin to FRAME metadata, but for RPC definitions.

Thoughts on this @niklasad1?

deno task bootstrap occasionally hangs on the FRAME metadata download step. Stopping and re-running usually fixes. Still, might confuse newcomers who bootstrap locally. Timeout / retry is necessary. Related to #47.

deno task test -- -- --update is a bit much

... ensuring that dependency licenses aren't restricted in such a manner that we cannot use them.

When an RPC server disconnects unexpectedly, do we attempt to reconnect, re-send any egress messages, misc? How about for transactions? If the user has encoded the transaction with a nonce and window of validity, yes. What if they have not? Should users need to explicitly opt into retry behavior? Should they specify their own back-off algorithm? Should they specify their own fallback logic beyond simply re-sending? A lot of questions to answer regarding approach to retries.

@harrysolovay so sorry to message you here but I couldn't do it on the Rescripts repo.

I've been using this in production since it was built, and it has always been the simplest and most effective CRA tool. It's the only one that seems to use plain vanilla JS and the KISS method.

In any case, every other library seems to have come and gone and this is the only one that works for me on CRA v5. Would you consider reopening it and/or transferring it? I'd be happy to take over development as I'm using it in production for https://bookclubs.com.

Thank you for considering it!

Sincerely,

Ian Campbell (@YPCrumble)

A GH workflow that ensures the project bootstraps and that type-checks and tests pass.

How do we want to approach narrowly-typing RPC methods (defined in rpc/messages.ts)?

• There is no means of narrowing a client's type to reflect a subset of common capabilities.
• There is no means of specifying custom RPC methods

The rpc/messages.ts file defines a central lookup of RPC methods of Substrate. This lookup is simply a record of functions. Paraphrasing:

type MethodLookup = {
  ...
  chain_getBlock(hash?: U.HashHexString): T.Block;
  chain_getBlockHash(height?: number): U.HashHexString;
  chain_getHead: MethodLookup["chain_getBlockHash"];
  chain_getFinalizedHead(): U.HashHexString;
  chain_getFinalisedHead: MethodLookup["chain_getFinalizedHead"];
  chain_getHeader(hash?: U.HashHexString): T.Header;
  chain_subscribeAllHeads(): Subscription<T.Header>;
  ...
};

A few things to note:

1. We utilize branded types such as HashHexString instead of string. The core assumption here is that we may want to validate inputs. Related discussion in #96.
2. Aliased methods reference the lookup directly (chain_getFinalisedHead, for example).
3. Methods which produce a subscription return a Subscription<T>, where T is a union of all possible notifications. A good example of a type applied to T is the notification type of chainHead_unstable_follow.

Should we...

• ... rename MethodLookup to SubstrateRpcMethods?
• ... split out any unstable methods into an UnstableMethods lookup?
• ... provide out-of-the box lookups such as PolkadotRpcMethods and KusamaRpcMethods?

From these lookups, users can narrow their clients.

import * as C from "capi";

// TODO: `Beacon` branded string type
const POLKADOT_BEACON = "wss://rpc.polkadot.io" as C.Beacon<C.PolkadotRpcMethods>;
// ... or ...
const polkadotBeacon = C.beacon<C.PolkadotRpcMethods>("wss://rpc.polkadot.io");

// Produce a narrowly-typed `WsRpcClient`
const client = C.wsRpcClient(POLKADOT_BEACON);

We could of course abstract over the creation of well-known clients.

import * as C from "capi";

const polkadotClient = C.polkadotRpcClient();

This approach enables users to define custom RPC clients with ease.

interface MyCustomRpcMethods extends C.SubstrateRpcMethods {
  some_otherMethod(...myArgs: MyArgs): Subscription<MyNotification>;
}

const beacon = C.beacon<MyCustomRpcMethods>(myChainWsProxyUrl);

const rawClient = C.wsRpcClient(beacon);

This is of course a very rough approximation of a solution. I'd imagine there will be a bit of complexity in the generic piping of the beacon's lookup into the methods of client instances.

A good starting place is likely to fine-tune and break apart the lookups so that they're easier to compose within the type system (and some tsdocs, so that we get that attractive guided DX!).

We should also investigate the possibility of generating RPC method lookups per-chain (#109).

[Moved from #17 (comment)]

I think having to continually pass around two different things to everything that wants an RPC is rather messy, and since opening/reconnecting/closing needs to happen under the hood, I think there should be a type that wraps RpcClient with open/reconnect/close methods, rather than punting two parameters around everywhere.

At that point, we could largely remove the concept of a beacon from the various effects, and simply have something like:

const rpc = wsRpc(POLKADOT_RPC_URL);
const pallet = C.pallet(rpc, "System");

The return of rpc would be this wrapper type, and not actually initiate any connection until open is called.

Because the passing of the beacon to the factory is handled by the user, we would no longer have to propagate the Beacon generic everywhere.

A Smoldot wrapper. Will abide by Capi's narrowly-typed RPC interface.

A means of initializing a Zombienet chain in the same env in which it is to be utilized.

const node = await C.zombienet({
  // ...config
});

node.url; // use this to instantiate an RPC client

node.close();

paritytech/zombienet#259

Initial work to model subscription methods as effects. This should support subscription methods implemented in accordance with the new JSON RPC interface (for instance, chainHead_unstable_follow).

Related: #11

We need to get something public out that people can use. Improving the UX of the ecosystem is one of the top priority things we can do, and we can't do that nearly as well without starting to get feedback on this. We should ship early & often

Should be pretty easy given the new $.deferred

Do U.HexString and comparable branded types buy us much in terms of safety / DX?

TODO:

• what is the minimum amount of documentation necessary for Sub0 & 0.1.0?
• scope out different sections of documentation + create issues for those sections?

#331

type isn't a valid Rust identifier (unless escaped via r#type), so this minimizes chance of collision.

Re-introduce some of what was removed in aaae3f6a01864982fba9be62b2fd5f7f132119cb. Developers will benefit from the ability to generate chain-specific type-safe bindings.

This should also serve as a basis for generating chain-specific migration utilities, which will intelligently handle the transition between chain runtime versions.

It would be great to add an examples folder with some uses cases where the developers can check and understand the mind model around this new api.
Nothing fancy: building connection, making a transfer, execute an extrinsic from a custom pallet etc...

Just opened two issues in the denoland/wasmbuild repo.

1. denoland/wasmbuild#21 – missing cache variable definition in JS bindings
2. denoland/wasmbuild#22 – request for advice in how to approach multiple builds

Upon their feedback/resolution, we should update our own usage and do away with this temporary patch.

Following the installation instructions in the README I get a failure when running deno task bootstrap:

error: Uncaught (in promise) NotFound: No such file or directory (os error 2)
  const p = Deno.run({
                 ^
    at Object.opSync (deno:core/01_core.js:172:12)
    at opRun (deno:runtime/js/40_process.js:28:17)
    at Object.run (deno:runtime/js/40_process.js:111:17)
    at runCmd (file:///Users/aggron/dev/work/capi/util/runCmd.ts:2:18)
    at buildWasm (file:///Users/aggron/dev/work/capi/_tasks/build_wasm.ts:18:9)
    at async file:///Users/aggron/dev/work/capi/_tasks/build_wasm.ts:30:1

Looks like what I was missing was wasm-bindgen.

Would be good to add cargo install -f wasm-bindgen-cli to the installation requirements (and perhaps wasm-opt as well?).

Right now, frame_metadata/__snapshots__ updates arbitrarily as people download new metadata, and one cannot necessarily tell whether the update is due to an on-chain update or a bug in the library.

If we commit target/frame_metadata/*.scale, these snapshots will update less frequently (as developers will not have to download them individually), and when they do update, it will be clearer that this is not a bug (as the source .scale file will also have been changed).

Alternatively, should it fix and commit?

The work of #73 solidifies our approach to converting Rust-defined data types to their JS counterparts and vice versa. Developers will—in many cases—need to manually input data (with JS) that correctly encodes to the on-chain world’s expectations. This is difficult without first understanding our approach to conversion.

TODO: conversation documentation

I believe the following API is fitting (utilizing the env channel to require the presence of signer at execution).

declare const signer: C.AnySigner;

const transfer = C
  .pallet(CHAIN_PROXY_URL, "balances")
  .txFactory("transfer")(DEST, VALUE)

await transfer.run({ signer });

See also #85

Blocked on getting GPG keys for CI signing.

More here.

Contributing and community guidelines, misc.

We need the ability to decode extrinsics.

Alternatively, should it auto-update the bindings and commit?

A deno.jsonc task for running formatters and cspell validation. We'll want to run these pre-commit.

What is the best approach for getting an async iterator over incoming messages with an RPC client? Ideally, when the iteration is ended, we can trigger a client-specific callback (to close the connection, for example).

for await (const message of rpc.iter(client)) {
  // utilize
  if (isLastMessage(message)) {
    break;
  }
}

// `client` is no longer needed, so the connection is closed

The ability to install as normal (from NPM) will be useful to those brave enough to test Capi in these early days.

Configure udd. Ensure it plays nicely with workflow of lockfile generation.