polkadot-js / api Goto Github PK

Currently when the connection is lost (i.e. node restarted), the API will reconnect. However in this case the old/previous subscripts are still active and not updated - basically new subscriptions will work, old ones won't get data resulting in a half-working app.

So...

1. Keep a list of subscriptions,
2. On re-connect, unsubscribe from what we have (the node may be the same one with an internet hiccup) and then re-do the subscription fresh
3. The issue comes in with ids... currently subscriptions are mapped through on the Id from the node, in this case these probably needs to be a mapping layer that takes "id the app knows it at" -> "id the node knows it at"

Some through needed still.

... and make sure we have proper docs here

polkadot-js/dev#132

In Polkadot-JS API Docs https://github.com/polkadot-js/api/docs, what is the current procedure that is used (both in 'development' and in 'production') for generating the Markdown (.md) and HTML (.html) docs for our Gitbook at https://polkadot.js.org/api/?

Note that I reviewed comments between @jacogr and @amaurymartiny in #159, and created this PR #167 to address Issue #154, although I'm not sure if what I've done is correct. So I've come up with the following questions:

1. How are we generating the HTML .html files in docs/ directory?

   • If I delete a HTML doc file (i.e. docs/api-provider/classes/http_index.httpprovider.html) and then run yarn; yarn run build:htmldoc;, the Bash Terminal logs says Documentation generated at /polkadot-js/api/docs/html, but the file http_index.httpprovider.html does not exist (isn't regenerated).

2. How are we generating the Markdown .md files in docs/ directory?

   • Were the Markdown .md files that are currently there generated originally generated by running yarn clean && typedoc --theme markdown --out docs/html (i.e. using the markdown instead of the default theme)

     • Note: When I previously ran yarn; yarn run build:htmldoc;, the Bash Terminal logs also said To generate markdown please set option --theme markdown.

   • Are we currently just "manually" modifying the Markdown .md files (instead of generating them with a script)?

   • How are we updating the Markdown .md files in docs/ directory automatically after modifying the Typedoc comments in the code comments? (i.e.

      /**
       * @example
       * ```javascript
      ...
     

3. Should we by using the gitbook command at all? https://toolchain.gitbook.com/

4. Should https://polkadot.js.org/ have a {.api} link to the Github repo and a link to the docs at https://polkadot.js.org/api/?

Cleanup encoding all-round -

https://github.com/polkadot-js/apps/pull/129/files#diff-f4e153119ebcf3e652c624e3d01f6a44R33

Currently the examples that are generated from Markdown include coloured syntax highlighting as shown below

However the API Docs themselves that are generated from TypeDoc/TSDoc labels in code comments sometimes also contain an @example code snippet, however it does not contain any coloured syntax highlighting.

The approach used to achieve generation of multi-line code snippets including spaces between lines of code was to wrap the code with <PRE><CODE> ... </CODE></PRE> and to add <BR> between lines of code where a newline was desired. However it does not colour the code based on the syntax used.

One possible solution could be to use Google Code Prettify, which according to the usage instructions simply requires applying their CSS class to one of the elements i.e. <PRE class="prettyprint"> or <CODE class="prettyprint">, and somehow loading their JS file in the root HTML document (or just injecting their CDN script) that's generated by building with Gitbooks (i.e. <script src="https://cdn.rawgit.com/google/code-prettify/master/loader/run_prettify.js"></script>). However I tried following this approach, but it did not work, and may be because CSS classnames may not be added to HTML elements that are used in TypeDocs.

Current when storage entry is empty (null), we don't automagically set the default value. So for instance, we are doing this - https://github.com/polkadot-js/api/pull/231/files#diff-2e3ff95c7b73d14c1a8b2fb9c44197a9R143

These two files appear to be duplicate of each other:

https://github.com/polkadot-js/api/blob/master/packages/types/src/PropIndex.ts

https://github.com/polkadot-js/api/blob/master/packages/types/src/ProposalIndex.ts

e.g. in the case of storage, we don't currently do the transform 100% - there really should be no need to call transform on the value (e.g. encodeAddress in this case), it should just be done for you in the API.

// TODO We should unsubscribe from subscriptions
  subscribeProposals () {
    const { api } = this.props;
    const key = createStorageKey(method)();
    const transform = storageTransform(method);

    api.state
      .getStorage(key)
      .subscribe((value) => {
        this.setState({
          proposals: (transform(value, 0) as any[])
          .reduce((proposals, [propIdx, proposal, accountId]) => {
            const address = encodeAddress(accountId);

            if (!proposals[address]) {
              proposals[address] = [propIdx];
            } else {
              proposals[address].push(propIdx);
            }

            return proposals;
          }, {})
        });
      });
  }

(The above from the apps/example/tut-003)

It means we would need to pass the actual un-encoded key to the method and use that in the deconstruction. So the call could become getStorage(storage.staking.public.freeBalanceOf, <params>).subscribe(...)

Needs some thinking around the auto-generation, i.e. how do we specify these types to make it "just happen".

The goal is to:

• Revert this change 16237c4
• Get the CI happy again (meaning writing the missing doc)

This file: https://github.com/polkadot-js/api/blob/master/packages/rpc-rx/src/index.ts

All these methods could probably be simplified, using memoization, refCount, publish etc.

If we use TypeDoc for TypeScript Getters i.e. get, as follows:

export default class Rpc implements RpcInterface {
  private _provider: ProviderInterface;

  constructor (provider: ProviderInterface) {
    this._provider = provider;
  }

  /**
   * @example
   * <BR>
   *
   * ```javascript
   * <insert_example_here>
   * ```
   */
  get provider (): ProviderInterface { return this._provider; }
...

When we render it in Gitbooks by running: yarn; yarn run build; yarn gitbook serve and then viewing the relevant page after opening http://localhost:4000, it renders as getprovider() instead of get provider() (i.e. there's no space between).
Note that this may also be an issue with Setters.

Not a documentation issue, rather implementation one - createInterface and create* friends should not be visible here

• https://github.com/polkadot-js/api/tree/master/docs/api
• https://github.com/polkadot-js/api/tree/master/docs/api-rx

Once less dependency hoop

This file: https://github.com/polkadot-js/api/blob/master/packages/types/src/codec/createType.ts

When you go to https://polkadot.js.org/api/, and navigate to the ApiRx page https://polkadot.js.org/api/api/classes/_rx_index_.apirx.html,
it displays EventEmitter information such as:

Inherited from EventEmitter.prefixed
Defined in /home/travis/build/polkadot-js/api/node_modules/eventemitter3/index.d.ts:6

The only one deemed useful is on, which could probably be wrapped to override it and expose it, to show the actual event names.

Inherited from EventEmitter.on
Defined in /home/travis/build/polkadot-js/api/node_modules/eventemitter3/index.d.ts:32

However we do not want to see stuff like addListener, as it just clutters

• Updates automatically with rpc loads (trigerred on connect)
• Export api-observable whenReady (promise)
• Rename extrinsics/storage testing to static (expose upon API load)

• Storage Function here uses any type storageFn = (arg?: any). Reference: https://github.com/polkadot-js/api/blob/master/packages/type-storage/src/utils/createFunction.ts#L48

• It was discussed that the "arg is determined based on the type of the storage function, which is determined by the metadata queries from the runtime.". It could even be "something extending Base" or "something that could be fed to a Base type to create a Base instance".

It could be either Base<any> | any | undefined, where any is all of these.

If we establish that it's ok to just use the any time here, then propose to remove the comment // TODO Find better type than any at https://github.com/polkadot-js/api/blob/master/packages/type-storage/src/utils/createFunction.ts#L48

Visual learning from diagrams is currently not available in the API docs.

Opportunity to add the Mermaid.js Gitbook plugin that provides Flowcharts, Sequence Diagrams, and Gantt Charts

When you go to https://polkadot.js.org/api/, it loads the Introduction page, which is the README.md, which embeds the SUMMARY.md file.

The RpcRx link in the SUMMARY.md points to rpc-rx/classes/index.rxapi.md, however the actual markdown file is rpc-rx/classes/index.rpcrx.md.

So because the link is incorrect, the RpcRx link on that page is broken,

And the RpcRx link on the sidebar as shown below is also broken.

Rename the actual RPC wrappers -

• api -> rpc-core
• api-rx -> rpc-rx
• api-provider -> rpc-provider

Then for the actual high-level API -

• api-observable -> api

In this api-codec seems like the odd man out, I would make the published package @polkadot/codec

@amaurymartiny Thoughts?

Currently numerous sub-folders are generated by the command yarn run build, which may be performed in development, and is also performed in production by Travis CI to generate the .md and .html files for the production Gitbook. I do not think they do not need to be included in the Github repo, and could be safely Gitignored, but because they currently are included in the Github repo and aren't Gitignored, it slightly interferes with the development workflow.
For instance in development, when I run yarn run build, it generates the following, and when you then run git status it shows them all in 'green' as being Changes not staged for commit:, when we don't care as they were just built. Because of this, when you actually make a change to a file, you have to sometimes "remember" what files you changed, particularly if the files you edited is one of the files in the docs/ directory that aren't automatically generated.

• Files that are automatically generated when you run yarn run build and by the CI. I think these could be Gitignored, and Removed from Github repo

docs/GLOSSARY.html
docs/api/classes/.md
docs/api/classes/.html
docs/api/index.html
docs/api/interfaces/.md
docs/api/modules/.md
docs/api-observable/**/*
docs/api-observable/classes/.md
docs/api-observable/modules/.md
docs/index.html
docs/rpc-core/classes/.md
docs/rpc-core/classes/.html
docs/rpc-core/index.html
docs/rpc-core/interfaces/.md
docs/rpc-core/modules/.md
docs/rpc-provider/classes/.md
docs/rpc-provider/index.html
docs/rpc-provider/interfaces/.md
docs/rpc-provider/modules/.md
docs/rpc-rx/classes/.md
docs/rpc-rx/index.html
docs/rpc-rx/interfaces/.md
docs/rpc-rx/modules/.md
docs/search_index.json
docs/type-extrinsics/interfaces/.md
docs/type-extrinsics/modules/.md
docs/type-jsonrpc/modules/.md
docs/type-storage/interfaces/.md
docs/type-storage/modules/.md
docs/types/classes/.md
docs/types/enums/.md
docs/types/interfaces/.md
docs/types/modules/*.md

I think we could remove the above from the Git repo since it's all auto-generated (i.e. running git rm docs/index.html docs/api/* docs/api-observable/* docs/rpc-core/* docs/rpc-provider/* docs/rpc-rx/* docs/search_index.json docs/type-extrinsics/* docs/type-jsonrpc/* docs/type-storage/* docs/types/*

• Files/folders to keep in Github repo but that we want to Gitignore:
  docs/gitbook//*
  docs/html//*
  docs/.nojekyll

• Files to keep in Github repo but NOT to Gitignore
  docs/examples/**/*
  docs/README.md
  docs/SUMMARY.md
  docs/CONTRIBUTING.md
  docs/GLOSSARY.md

Without doing the above changes, to hide the built files from displaying when you run git status you may run git checkout -- docs/types/** docs/type-storage/** docs/type-jsonrpc/** docs/type-extrinsics/** docs/search_index.json docs/rpc-rx/** docs/rpc-provider/** docs/rpc-core/** docs/index.html docs/examples/**/*.html docs/api/** docs/api-observable/** docs/GLOSSARY.html

I've tried adding the following to the Gitignore file in the root directory, but haven't been able to get it to work and ignore the files.

# root docs folder to ignore
docs/**/*

# explicitly track certain content nested in the 'docs' folder
!docs/examples/**/*
!docs/README.md
!docs/SUMMARY.md
!docs/CONTRIBUTING.md
!docs/GLOSSARY.md

So everything looks nice, e.g. https://polkadot.js.org/api/

However, from https://polkadot.js.org/api/api/ scrolling down to Classes and then clicking on Api does not quite work... Basically it seems like this https://github.com/polkadot-js/api/tree/master/docs/api/classes does not result in a corresponding api.html file generated from api.md. (While all the others at higher levels seem to be doing fine)

Since we copy across to docs in the build script, I also too a look at my local _book folder, same there, it is missing. :(

https://github.com/polkadot-js/api/blob/master/docs/api/classes/_index_.api.md

Seems like there is a missing "```" somewhere. Example doesn't render as example, text renders as, well, text.

And a link to the gitbook pages - https://polkadot.js.org/api/api/classes/_index_.api.html and this (looks) a bit worse https://polkadot.js.org/api/api-provider/classes/_ws_index_.wsprovider.html

Uncaught TypeError: connect is not a function
    at onClose.js:18

This really should not be happening...

Move the following and bump all versions

• type-extrinsics
• type-jsonrpc
• type-params
• type-primitives
• type-storage

https://flow.org/en/docs/linting/

I cannot put it more succinctly :)

• Remote extrinsics & storage (the latter will still contain some, e.g. :code, basically all these paritytech/substrate#764)
• Pull storage, extrinsics and events (this is new) from state_getMeta
• Fixup encoders and do it properly - currently they are all-over, some in primitives, some in params
• Make everything work....
• ... I missed some stuff

cc @amaurymartiny :)

"@polkadot/api": "^0.28.24",

I did not see this issue yesterday at all but this is showing up today.

Convert to ES6 classes in the same fashion as polkadot-js/client#132

https://github.com/polkadot-js/api/blob/master/packages/api-provider/src/http/index.ts
https://github.com/polkadot-js/api/blob/master/packages/api-provider/src/ws/index.ts

Not the only 2 places in the API layer, but a start. (Bonus convert all the closures with self)

https://github.com/polkadot-js/api/blob/master/packages/api-provider/src/ws/connect.ts#L13

Currently when the construction fails, the UI just has a blank page.

polkadot-js/apps#132

paritytech/polkadot#563

This one may be a bit tricky - since we are assuming subscribe* methods. (However extend definitions to cater for it)

When I run the tests with yarn run test, in the summary it shows that it's testing duplicate copies of the same *.test.js files that are both in the src/ and build/ directories of each package. See screenshot below:

Also, when I run the tests with yarn run test, a couple of the tests were failing due to the error The name@polkadot/_____ was looked up in the Haste module map. It cannot be resolved, because there exists several different files, or packages, that provide a module for that particular name and platform. The platform is generic (no extension). You must delete or blacklist files until there remains only one of these. See screenshot below.

paritytech/polkadot#564

We have been using RpcRx to refer to rpc-rx. However in types defined for rpc-rx are prefixed with RxRpc https://github.com/polkadot-js/api/blob/master/packages/rpc-rx/src/types.d.ts

No objection was received to review comments of PR #233 when it was proposed to rename the prefixes from RxRpc to RpcRx in a separate PR.

Action is to

• rename RxRpcInterface$Method to RpcRxInterface$Method
• rename RxRpcInterface$Section to RpcRxInterface$Section
• rename RxRpcInterface$Events to RpcRxInterface$Events
• rename RxRpcInterface to RpcRxInterface

I am using the following code to send DOTs from Alice to Bob:
https://gist.github.com/chevdor/aed62310878042c81f8e71f9e13e7511

It fails with the following error:

ExtError: submitExtrinsic (extrinsic: Bytes): Hash:: [1001]: Extrinsic has invalid format.

Currently the methods named as getStorageAt does not work, it is actually getStorage with optional. Either needs to be aliassed (and check on the rpc layer) or the isOptional needs to be working all-through with the At methods removed.

I can get all the examples to work if I follow the instructions in https://polkadot.js.org/api/examples/, by starting a Polkadot Node with polkadot --dev, and then in the folder of each example running yarn; yarn run start

However, if I "do my own thing" to try the examples in a separate folder, I get errors. For example if I create a package.json with yarn init -y. Add all dependencies yarn add bn.js @polkadot/api @polkadot/api-provider @polkadot/storage @polkadot/extrinsics -W. Install all dependences in node_modules with yarn. Start a Polkadot Node with polkadot --dev. And then copy/paste the code from each example into a test.js file and run it with node test.js.

I can successfully get "Simple Connect" and "Generate Accounts" example code to work.

However I get errors when I run the other examples as follows:

• "Listen to blocks" example code as it just returns #0 (even thought it's already at block #180,000).
• "Listen to balance change" example code, it returns error TypeError: Cannot read property 'public' of undefined (in relation to the argument that 's provided storage.staking.public.freeBalanceOf).
• "Read Storage" example code returns error TypeError: Cannot read property 'public' of undefined (in relation to argument storage.session.public.validators).
• "Craft Extrinsic" example code returns error Error: Cannot find module '@polkadot/extrinsics/codec/encode' (even after performing yarn add @polkadot/extrinsics).
• "Transfer DOTs" example code returns error Error: Cannot find module '@polkadot/extrinsics/codec/encode/uncheckedLength'

Related to issue described in polkadot-js/apps#222

Implemented in https://github.com/polkadot-js/primitives/pull/10

Need to update the TypeDocs to accommodate the renaming that occurred in PR #220

We have been using ApiRx to refer to api-rx. However types defined for api-rx are prefixed with RxApi instead of ApiRx

Action is to

• rename RxApiInterface$Events to ApiRxInterface$Events
• rename RxApiInterface to ApiRxInterface

TODOs (will be expanded)

• adapt current samples to be poc-3 compliant
• jsonrpc needs all methods documented
• storage needs full documentation on use
  • static interface
  • dynamic interface
  • actual use
• extrinsics needs full documentation on use
  • static interface
  • dynamic interface
  • actual use

Potentially -

api-observable needs full documentation for all the functions (Will be renamed to @polkadot/api) (not advocating use, examples or docs - only UI use, to be revamped)

• Compact to extend UInt as a base
• Drop-in replacement for UInt with compact encoding
• Adjust Type to turn Compact<u32> into Compact32 (along with 8, 16, 64, 128, 256)