Coder Social home page Coder Social logo

scale-info's Introduction

scale-info · build Latest Version

A library to describe Rust types, geared towards providing info about the structure of SCALE encodable types.

The definitions provide third party tools (e.g. a UI client) with information about how they are able to decode types agnostic of language.

At its core is the TypeInfo trait:

pub trait TypeInfo {
    type Identity: ?Sized + 'static;
    fn type_info() -> Type;
}

Types implementing this trait build up and return a Type struct:

pub struct Type<T: Form = MetaForm> {
    /// The unique path to the type. Can be empty for built-in types
    path: Path<T>,
    /// The generic type parameters of the type in use. Empty for non generic types
    type_params: Vec<T::Type>,
    /// The actual type definition
    type_def: TypeDef<T>,
}

Types are defined as one of the following variants:

pub enum TypeDef<T: Form = MetaForm> {
    /// A composite type (e.g. a struct or a tuple)
    Composite(TypeDefComposite<T>),
    /// A variant type (e.g. an enum)
    Variant(TypeDefVariant<T>),
    /// A sequence type with runtime known length.
    Sequence(TypeDefSequence<T>),
    /// An array type with compile-time known length.
    Array(TypeDefArray<T>),
    /// A tuple type.
    Tuple(TypeDefTuple<T>),
    /// A Rust primitive type.
    Primitive(TypeDefPrimitive),
}

Built-in Type Definitions

The following "built-in" types have predefined TypeInfo definitions:

  • Primitives: bool, char, str, u8, u16, u32, u64, u128, i8, i16, i32, i64, i128.

  • Sequence: Variable size sequence of elements of T, where T implements TypeInfo. e.g. [T], &[T], &mut [T], Vec<T>

  • Array: Fixed size [T: $n] for any T which implements TypeInfo, where $n is one of the predefined sizes.

  • Tuple: Tuples consisting of up to 10 fields with types implementing TypeInfo.

User-defined Types

There are two kinds of user-defined types: Composite and Variant.

Both make use of the Path and Field types in their definition:

Fields

A fundamental building block to represent user defined types is the Field struct which defines the Type of a field together with its optional name. Builders for the user defined types enforce the invariant that either all fields have a name (e.g. structs) or all fields are unnamed (e.g. tuples).

Path

The path of a type is a unique sequence of identifiers. Rust types typically construct a path from the namespace and the identifier e.g. foo::bar::Baz is converted to the path ["foo", "bar", "Baz"].

Composite

Composite data types are composed of a set of Fields.

Structs are represented by a set of named fields, enforced during construction:

struct Foo<T> {
    bar: T,
    data: u64,
}

impl<T> TypeInfo for Foo<T>
where
    T: TypeInfo + 'static,
{
    type Identity = Self;

    fn type_info() -> Type {
        Type::builder()
            .path(Path::new("Foo", module_path!()))
            .type_params(vec![MetaType::new::<T>()])
            .composite(Fields::named()
                .field(|f| f.ty::<T>().name("bar").type_name("T"))
                .field(|f| f.ty::<u64>().name("data").type_name("u64"))
            )
    }
}

Tuples are represented by a set of unnamed fields, enforced during construction:

struct Foo(u32, bool);

impl TypeInfo for Foo {
    type Identity = Self;

    fn type_info() -> Type {
        Type::builder()
            .path(Path::new("Foo", module_path!()))
            .composite(Fields::unnamed()
                .field(|f| f.ty::<u32>().type_name("u32"))
                .field(|f| f.ty::<bool>().type_name("bool"))
            )
    }
}

Variant

Variant types aka enums or tagged unions are composed of a set of variants. Variants can have unnamed fields, named fields or no fields at all:

enum Foo<T>{
    A(T),
    B { f: u32 },
    C,
}

impl<T> TypeInfo for Foo<T>
where
    T: TypeInfo + 'static,
{
    type Identity = Self;

    fn type_info() -> Type {
        Type::builder()
            .path(Path::new("Foo", module_path!()))
            .type_params(vec![MetaType::new::<T>()])
            .variant(
                Variants::new()
                   .variant("A", |v| v.fields(Fields::unnamed().field(|f| f.ty::<T>())))
                   .variant("B", |v| v.fields(Fields::named().field(|f| f.ty::<u32>().name("f").type_name("u32"))))
                   .variant_unit("C")
            )
    }
}

If no variants contain fields then the discriminant can be set explicitly, enforced by the builder during construction:

enum Foo {
    A,
    B,
    C = 33,
}

impl TypeInfo for Foo {
    type Identity = Self;

    fn type_info() -> Type {
        Type::builder()
            .path(Path::new("Foo", module_path!()))
            .variant(
                Variants::new()
                    .variant("A", |v| v.index(1))
                    .variant("B", |v| v.index(2))
                    .variant("C", |v| v.index(33))
            )
    }
}

The Registry

Information about types is provided within the so-called type registry (Registry). Type definitions are registered there and are associated with unique IDs that the outside can refer to, providing a lightweight way to decrease overhead instead of using type identifiers.

All concrete TypeInfo structures have two forms:

  • One meta form (MetaType) that acts as a bridge to other forms
  • A portable form suitable for serialization.

The IntoPortable trait must also be implemented in order prepare a type definition for serialization using an instance of the type registry.

After transformation all type definitions are stored in the type registry. Note that the type registry should be serialized as part of the metadata structure where the registered types are utilized to allow consumers to resolve the types.

Encoding

The type registry can be encoded as:

  • JSON (with the "serde" feature enabled).
  • SCALE itself (using parity-scale-codec).

Features

The following optional cargo features are available:

  • serde includes support for json serialization/deserialization of the type registry. See example here.
  • derive reexports the scale-info-derive crate.

Known issues

When deriving TypeInfo for a type with generic compact fields e.g.

#[derive(Encode, TypeInfo)]
struct Foo<S> { #[codec(compact)] a: S }

You may experience the following error when using this generic type without the correct bounds:

error[E0275]: overflow evaluating the requirement `_::_parity_scale_codec::Compact<_>: Decode`

See #65 for more information.

Resources

scale-info's People

Contributors

ascjones avatar benluelo avatar bkchr avatar boundless-forest avatar cupnfish avatar dvdplm avatar eskimor avatar gilescope avatar gshep avatar gui1117 avatar jsdw avatar kichjang avatar koushiro avatar koute avatar lexnv avatar liuchengxu avatar montekki avatar nazar-pc avatar niklasad1 avatar robbepop avatar seanyoung avatar shaunxw avatar skymanone avatar tadeohepperle avatar tdimitrov avatar tovarishfin avatar wigy-opensource-developer avatar xanewok avatar xermicus avatar xlc avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

scale-info's Issues

no method named `field_of` found for struct `FieldsBuilder` in the current scope

I was using scale_info 0.6.0 and now upgraded to 1.0.0

It fails with the following error

error[E0599]: no method named `field_of` found for struct `FieldsBuilder` in the current scope
  --> /flipper/models.rs:75:42
   |
75 |             .composite(Fields::unnamed().field_of::<[u8; 20]>("[u8; 20]"))
   |                                          ^^^^^^^^ method not found in `FieldsBuilder<UnnamedFields>`
impl TypeInfo for TokenAddress {
    type Identity = Self;

    fn type_info() -> Type {
        Type::builder()
            .path(Path::new("TokenAddress", "pool"))
            .composite(Fields::unnamed().field_of::<[u8; 20]>("[u8; 20]"))
    }
}

How can I fix this error? what can I use instead of field_of?

Please help me. thanks

Empty tuple is encoded as null in the polkadot SDK

After upgrading our parachain to use metadata v14, we now have no way to distinguish whether an element is present or not in a StorageMap<AccountId32 ()>, as the api.query.pallet.storage(key) returns null in both cases. I can see also from polkadotjs apps that the map is a map from AccountId32 to Option<Null>.
So far, the only workaround I found is to fetch ALL the keys via api.query.pallet.keys and then filter, but that can get quite expensive if the map has a lot of keys. Is there a more efficient way to check for existence in such a map or, if there is not, would it be possible to fix this issue by encoding empty tuples differently? I am not sure if this issue pertains the SDK side or the chain side.

I also opened an issue in the @polkadot/api repo.

On the chain side, this is what I get when printing type info for the () type:

Type {
    path: Path {
        segments: [],
    },
    type_params: [],
    type_def: Tuple(
        TypeDefTuple {
            fields: [],
        },
    ),
    docs: [],
}

Tracking issue for 1.0

Tracking issue for what we definitely need to cut a 1.0 release for substrate, which we should aim for before this makes its way into any mainnet chains. Note that we may do an intermediate release for the substrate PR paritytech/substrate#8615, leaving the door open for any breaking changes after that PR is merged, but before it is deployed anywhere.

  • Enum indices #80
  • Doc comments #87
  • Compact type bounds #83
  • 0 based type lookup ids #90
  • Handle Box types #82 Edit: can go in a 1.x release
  • Skip type params #95
  • Potentially reconsider TypeInfo bound on the T in PhantomData to remove the requirement for some unnecessary(?) TypeInfo bounds in paritytech/substrate#8615
  • Naming of things, are we happy with naming of types/concepts etc. before we lock down for 1.0? Edit: naming stays the same for 1.0
    • Rename Composite and Variant type defs #93

Don't enable bitvec when "std" feature is enabled

Currently we have:

std = [
    "bitvec/std",
    "scale/std",
]

As the std feature in Cargo.toml. Enabling it enables bitvec too.

Let's swap to:

std = [
    "bitvec?/std",
    "scale/std",
]

To avoid enabling bitvec unless it's also asked for?

Handle `#[codec(index = $int)]` on regular enums

PR #44 handles the index attribute on C-like enums, but does not address regular enums with some variants marked #[codec(index = $int)].

All enum variants have a discriminant. C-style enums allow explicitly setting the discriminant. Regular enums do not.
The #[codec(index = 123)] attribute sets the index byte for the variant it decorates when the enum is SCALE encoded.

"index" and "discriminant" do not mean the same thing, but the current implementation conflates them somewhat and the default "index" for a variant is, in order of priority, the #[codec(index = …)] attribute, the explicitly set discriminant, and lastly the implicit index from the order of appearance.

I believe parity-scale-codec is slightly broken today, in that the following will lead to bad things happening when encoding/decoding:

#[derive(Encode)]
enum Foo {
	A,
	#[codec(index = 2)]
	B = 8,
	C
}

The Variant struct in scale-info has a discriminant member, an Option<u64>. Today it is None for all non C-style enums.

Picking the right solution here can be a bit tricky. We can use the current discriminant field to store the index. This is a questionable solution though, as we perpetuate the mixing of index and discriminant. Open to alternative ideas.

Allow removing requirement for `TypeInfo` bounds on type params

Polkadot has the type Xcm<Call> which has an Order<Call> and an Order<()> which in turn has an Xcm<Call>. We get a overflow evaluating the requirement Vec<order::Order<()>>: TypeInfo

Since Call is not directly encoded, see double_encoded, this type parameter does not need to implement Encode/Decode/TypeInfo

To get around this for other derives they are just removing the bounds:

#[derivative(Clone(bound = ""), Eq(bound = ""), PartialEq(bound = ""), Debug(bound = ""))]
#[codec(encode_bound())]
#[codec(decode_bound())]

On the surface we should be able to add #[scale_info(bounds()))] from #88 to erase the bounds. Though that is not enough on its own because there is still TypeInfo requirement for adding to type_params.

So what we need to be able to do is control which type params are included in the type_params list, in addition to (or even instead of) the custom bounds.

Consider making "built-in" complex types more strongly typed

Currently builtin types like Option, Result, Range etc. are defined as normal Composite or Variant types in the prelude. One downside to this is that it requires string matching in downstream consumers in order to identify the types.

One option to make it more strongly typed would be to introduce another top level variant e.g. BuiltIn or Prelude which contains a sub enum which would cover all the possible prelude types.

This would make it easier to differentiate between user defined types and built-in scale-info types.

Related #130

Support Compact types

I just had a chat with @jacogr yesterday and we came to the conclusion that we also need support for Compact types. We could implement those into our format in 3 different ways and I do not know which one is the best.

  • We could provide a flag that states whether some type is compact encoded, similar to how it has been done for displayName in ink!.
  • We could provide another built-in type alongside the others like Slice and Array that is Compact and has a single generic parameter.
  • We could make it a custom type that has special meaning and a prelude namespace.

Originally posted by @Robbepop in #3

Improve `docs` feature

Currently the docs feature only excludes docs for derived TypeInfo. We should also exclude docs added from manual impls via the docs builder methods.

As suggested by @thiolliere paritytech/substrate#8615 (comment) the docs methods could be inlined and made a no-op if the docs feature is disabled. We would need to double check that this works as intended by not including the static strings in the resulting Wasm.

A similar technique could also be used together with this in frame-metadata, to remove all docs. See paritytech/frame-metadata#6.

Refactor the derive crate

As raised in #61 (review), it is less than ideal we have to pass around the crate names to all the functions.

Originally the proc macro was very small so could justify being a few functions, but now it is becoming unwieldy and there is the above mentioned problem of shared global state.

One approach would be to separate the process into 2 steps parse -> Intermediate Representation -> expand which is a common pattern in proc macros.

Impossible to implement `TypeInfo` for enums with more than 256 variants

The whole ecosystem's derives (Encode, Decode, TypeInfo) don't support big enums well, but in the case of Encode and Decode, it's not much of an issue because it's easy to implement them by hand, with an index that can exceed an u8 bound (with an u16 for example). However, in the case of TypeInfo, as far as I understand it, it's impossible to implement the trait for a big enum with the current API since a variant index is typed as an u8: https://docs.rs/scale-info/1.0.0/scale_info/build/struct.VariantBuilder.html#method.index.

Is there any reason why it's an u8? I guess it could be an usize (for example) without breaking any functionality, and it would allow people to include those big enums in the SDK.

What do you think about that?

For reference, this is what we do in our code for now:

pub enum Ledger {
    // bunch of variants
}

impl TypeInfo for Ledger {
    type Identity = Self;

    fn type_info() -> Type {
        Type::builder()
            .path(Path::new(stringify!(Ledger), module_path!()))
            .composite(Fields::unnamed().field(|f| f.compact::<u16>().type_name("u16")))
    }
}

It's inconvenient because we see the enum as a plain u16.

I found a little flaws in `scale-info` doc V2.1.2

It's about the example of scale_info::TypeInfo implemented for enum in the document:

1652934199(1)

I think these docs are owned by Paritytech.

I think the default index should start with 0. I met this problem when I'm developing one of my user-defined struct libs in the ink! smart contract.

Derive does not add bounds for wrapped associated types correctly

The following struct contains an associated type that needs TypeInfo + 'static bounds. The derived code does not provide the required bound.

    trait Colorizer {
        type Property;
    }
    struct Felt {}
    impl Colorizer for Felt {
        type Property = String;
    }
    #[allow(unused)]
    #[derive(TypeInfo)]
    struct Pen<P: Colorizer = Felt> {
        usage: Docs<P>,
        properties: Vec<P::Property>
    }

Pen needs the following bounds:

Docs<P>: TypeInfo + 'static,
P: Colorizer + TypeInfo + 'static,
P::Property: TypeInfo + 'static,

The derive macro however does not provide the last one, P::Property: TypeInfo + 'static.

Macro expansion:

    const _: () = {
        impl<P: Colorizer> ::scale_info::TypeInfo for Pen<P>
        where
            Docs<P>: ::scale_info::TypeInfo + 'static,
            Vec<P::Property>: ::scale_info::TypeInfo + 'static,
            P: Colorizer + ::scale_info::TypeInfo + 'static,
        {
            type Identity = Self;
            fn type_info() -> ::scale_info::Type {
                ::scale_info::Type::builder()
                    .path(::scale_info::Path::new("Pen", "derive"))
                    .type_params(<[_]>::into_vec(box [::scale_info::meta_type::<P>()]))
                    .composite(
                        ::scale_info::build::Fields::named()
                            .field_of::<Docs<P>>("usage", "Docs<P>")
                            .field_of::<Vec<P::Property>>("properties", "Vec<P::Property>"),
                    )
            }
        };
    };

A concrete example of where we need this is https://github.com/paritytech/scale-info/blob/master/src/ty/mod.rs#L65-L81

Notice the odd Vec<P::Property>: ::scale_info::TypeInfo + 'static, in the expansion.

String Representation

  • Either we go full way and represent String the same as str or we do the same as I proposed for HashMap where we make it like a tuple struct to be able to drop the vec name for the field.

  • Maybe we actually can do better than this? What we actually want is like a type alias, or what do you think? So we could say that a String is really just a str but still something different.

A good idea because this way we are not losing information i.e. that the original type was String and not str.

  • Yes, but actually I am asking myself if this is useful information with scale-info being a SCALE codec protocol, not a type protocol.

Originally posted by @Robbepop in type-metadata/type-metadata#42

trait bound `H256: TypeInfo` is not satisfied; `TypeInfo` is not implemented for `H256, H160, H64, ethereum_types::U256`

Frontier(pallet-ethereum) project failed to compile after updating from polkadot-v0.9.17 to v0.9.18

   Compiling polkadot-runtime v0.9.18 (https://github.com/paritytech/polkadot?branch=release-v0.9.18#9ed0c982)
   Compiling kusama-runtime v0.9.18 (https://github.com/paritytech/polkadot?branch=release-v0.9.18#9ed0c982)
   Compiling p-chain-runtime v0.0.1 (/mnt/sda3/polkadot/1pchain/runtime)
    Checking ethereum v0.12.0
error[E0277]: the trait bound `H256: TypeInfo` is not satisfied
   --> /home/user2038/.cargo/registry/src/github.com-1ecc6299db9ec823/ethereum-0.12.0/src/header.rs:10:39
    |
10  |     derive(codec::Encode, codec::Decode, scale_info::TypeInfo)
    |                                          ^^^^^^^^^^^^^^^^^^^^ the trait `TypeInfo` is not implemented for `H256`
    |
note: required by a bound in `FieldBuilder::<N>::ty`
   --> /home/user2038/.cargo/registry/src/github.com-1ecc6299db9ec823/scale-info-2.1.1/src/build.rs:382:13
    |
382 |         TY: TypeInfo + 'static + ?Sized,
    |             ^^^^^^^^ required by this bound in `FieldBuilder::<N>::ty`
    = note: this error originates in the derive macro `scale_info::TypeInfo` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0277]: the trait bound `H160: TypeInfo` is not satisfied
   --> /home/user2038/.cargo/registry/src/github.com-1ecc6299db9ec823/ethereum-0.12.0/src/header.rs:10:39
    |
10  |     derive(codec::Encode, codec::Decode, scale_info::TypeInfo)
    |                                          ^^^^^^^^^^^^^^^^^^^^ the trait `TypeInfo` is not implemented for `H160`
    |
note: required by a bound in `FieldBuilder::<N>::ty`
   --> /home/user2038/.cargo/registry/src/github.com-1ecc6299db9ec823/scale-info-2.1.1/src/build.rs:382:13
    |
382 |         TY: TypeInfo + 'static + ?Sized,
    |             ^^^^^^^^ required by this bound in `FieldBuilder::<N>::ty`
    = note: this error originates in the derive macro `scale_info::TypeInfo` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0277]: the trait bound `Bloom: TypeInfo` is not satisfied
   --> /home/user2038/.cargo/registry/src/github.com-1ecc6299db9ec823/ethereum-0.12.0/src/header.rs:10:39
    |
10  |     derive(codec::Encode, codec::Decode, scale_info::TypeInfo)
    |                                          ^^^^^^^^^^^^^^^^^^^^ the trait `TypeInfo` is not implemented for `Bloom`
    |
note: required by a bound in `FieldBuilder::<N>::ty`
   --> /home/user2038/.cargo/registry/src/github.com-1ecc6299db9ec823/scale-info-2.1.1/src/build.rs:382:13
    |
382 |         TY: TypeInfo + 'static + ?Sized,
    |             ^^^^^^^^ required by this bound in `FieldBuilder::<N>::ty`
    = note: this error originates in the derive macro `scale_info::TypeInfo` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0277]: the trait bound `ethereum_types::U256: TypeInfo` is not satisfied
   --> /home/user2038/.cargo/registry/src/github.com-1ecc6299db9ec823/ethereum-0.12.0/src/header.

My Rust tool-chain

Default host: x86_64-unknown-linux-gnu
rustup home:  /home/user2038/.rustup
installed toolchains
stable-x86_64-unknown-linux-gnu
nightly-2021-09-10-x86_64-unknown-linux-gnu
nightly-2021-11-12-x86_64-unknown-linux-gnu
nightly-x86_64-unknown-linux-gnu
bpf
1.59.0-x86_64-unknown-linux-gnu (default)

active toolchain
1.59.0-x86_64-unknown-linux-gnu (default)
rustc 1.59.0 (9d1b2106e 2022-02-23)

Derive requires direct dependency on `parity-scale-codec`

Discussed here and here.

Using the derive crate assumes a direct dependency on parity-scale-codec from the user's crate. Brainstorming ideas:

  • as Basti suggests do a reexport of the required types (see comment)
  • better error message if the crate is not present
  • add a feature "compact" or so to enable that functionality for which the codec types are required.
  • make the macro determine whether any compact fields are present and only import the lib.

This also feeds into the larger discussion about how these two libraries coexist, and to what extent they should be coupled.

/cc @Robbepop @dvdplm @ordian

Allow excluding docs

We should be able to either disable the inclusion of docs at compile time via a feature or allow removing of the docs from the metadata at runtime (or both). This is to reduce the size of the metadata if the docs are not required.

Also see #87 (comment)

Originally posted by @ascjones in #87 (comment)

Regression from latest release with crate resolution

This is likely a regression from #145, my hunch is that :: is likely no longer always prepended to scale_info.

We have a test in ink! that started to fail since the new version of scale-info was published.

To reproduce:

Create a Cargo.toml with this content:

[package]
name = "foobar"
version = "0.1.0"
authors = ["[your_name] <[your_email]>"]
edition = "2021"

[dependencies]
ink_primitives = { version = "3.0", default-features = false }
ink_metadata = { version = "3.0", default-features = false, features = ["derive"], optional = true }
ink_env = { version = "3.0", default-features = false }
ink_storage = { version = "3.0", default-features = false }
ink_lang = { version = "3.0", default-features = false }

scale = { package = "parity-scale-codec", version = "3", default-features = false, features = ["derive"] }
scale-info = { version = "2", default-features = false, features = ["derive"], optional = true }

[lib]
name = "foobar"
path = "lib.rs"
crate-type = [
	# Used for normal contract Wasm blobs.
	"cdylib",
]

[features]
default = ["std"]
std = [
    "ink_metadata/std",
    "ink_env/std",
    "ink_storage/std",
    "ink_primitives/std",
    "scale/std",
    "scale-info/std",
]
ink-as-dependency = []

Crate a lib.rs with this content:

#![no_implicit_prelude]

#[::ink_lang::contract]
mod contract {
    #[ink(storage)]
    pub struct Contract {}

    impl Contract {
        #[ink(constructor)]
        pub fn constructor() -> Self {
            Self {}
        }

        #[ink(message)]
        pub fn message(&self) {}
    }
}

Then execute cargo +nightly build:

$ cargo +nightly build                                                                                                                               
   Compiling foobar v0.1.0 (/tmp/foobar)                                       
error[E0433]: failed to resolve: use of undeclared crate or module `scale_info`                                                                                
 --> lib.rs:6:5                                                                
  |                                                                            
6 |     pub struct Contract {}         
  |     ^^^^^^^^^^^^^^^^^^^^^^ use of undeclared crate or module `scale_info`  
  |                                    
  = note: this error originates in the derive macro `::scale_info::TypeInfo` (in Nightly builds, run with -Z macro-backtrace for more info)                    
                                                                               
error[E0433]: failed to resolve: use of undeclared crate or module `scale_info` 
 --> lib.rs:6:5                        
  |                                                                                                                                                            
6 |     pub struct Contract {}                                                                                                                                 
  |     ^^^^^^^^^^^^^^^^^^^^^^ not found in `scale_info`                                                                                                       
  |
  = note: consider importing this struct:
          scale_info::Type
  = note: this error originates in the derive macro `::scale_info::TypeInfo` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0433]: failed to resolve: use of undeclared crate or module `scale_info` 
 --> lib.rs:6:5
  |
6 |     pub struct Contract {}
  |     ^^^^^^^^^^^^^^^^^^^^^^ not found in `scale_info`
  |
  = note: consider importing one of these items:
          scale_info::Path
          std::path::Path
  = note: this error originates in the derive macro `::scale_info::TypeInfo` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0433]: failed to resolve: use of undeclared crate or module `scale_info` 
 --> lib.rs:6:5
  |
6 |     pub struct Contract {}
  |     ^^^^^^^^^^^^^^^^^^^^^^ not found in `scale_info::build`
  |
  = note: consider importing this enum:
          scale_info::build::Fields
  = note: this error originates in the derive macro `::scale_info::TypeInfo` (in Nightly builds, run with -Z macro-backtrace for more info)

For more information about this error, try `rustc --explain E0433`.
error: could not compile `foobar` due to 8 previous errors

Update CI

I think that we should:

  • Run cargo clippy and rustfmt on a standard toolchain (default target, not nightly); I've noticed intermittent issues running those as they are on nightly wasm.
  • Have a separate wasm32 test job using that target
  • Use the aarch64-unknown-none target for the no_std test (this seemed to work reasonably well in scale-bits at least) to ensure no std paths creep in.
  • Use stable checks rather than nightly so that things like rustfmt and clippy don't run the risk of breaking or changing day by day.

Are nested generics supported?

We are using substrate-fixed in our pallets, and have some problems with the scale-info.

We have a type defined as follows:

#[cfg_attr(feature = "derive-scale-info", derive(scale_info::TypeInfo))]
#[derive(Eq, PartialEq, Ord, PartialOrd, Clone, Copy, Hash, Debug, Default)]
pub struct UInt<U, B> {
    /// The more significant bits of `Self`.
    pub(crate) msb: U,
    /// The least significant bit of `Self`.
    pub(crate) lsb: B,
}

However, we get this error when using this type:

 the trait bound `typenum::uint::UInt<typenum::uint::UInt<typenum::uint::UInt<typenum::uint::UInt<typenum::uint::UInt<typenum::uint::UInt<typenum::uint::UInt<typenum::uint
::UTerm, typenum::bit::B1>, typenum::bit::B0>, typenum::bit::B0>, typenum::bit::B0>, typenum::bit::B0>, typenum::bit::B0>, typenum::bit::B0>: TypeInfo` is not satisfied
  --> primitives/src/balances.rs:29:62
   |
29 | #[derive(Encode, Decode, Default, RuntimeDebug, Clone, Copy, TypeInfo)]
   |                                                              ^^^^^^^^ the trait `TypeInfo` is not implemented for `typenum::uint::UInt<typenum::uint::UInt<typenum::uint::UInt<type
num::uint::UInt<typenum::uint::UInt<typenum::uint::UInt<typenum::uint::UInt<typenum::uint::UTerm, typenum::bit::B1>, typenum::bit::B0>, typenum::bit::B0>, typenum::bit::B0>, typenum::b
it::B0>, typenum::bit::B0>, typenum::bit::B0>`
   |
   = note: required because of the requirements on the impl of `TypeInfo` for `substrate_fixed::FixedI128<typenum::uint::UInt<typenum::uint::UInt<typenum::uint::UInt<typenum::uint::UIn
t<typenum::uint::UInt<typenum::uint::UInt<typenum::uint::UInt<typenum::uint::UTerm, typenum::bit::B1>, typenum::bit::B0>, typenum::bit::B0>, typenum::bit::B0>, typenum::bit::B0>, typen
um::bit::B0>, typenum::bit::B0>>`
   = note: this error originates in the derive macro `TypeInfo` (in Nightly builds, run with -Z macro-backtrace for more info)

The contained types: B0, B1, UTerm have scale-info derived (although, they probably don't need to because they are all unit structs?)

Implement stable type id

Something similar to TypeId but that's stable. i.e. It never changes as long as the shape of the type renames the same.

This will be useful to detect unexpected scale codec breaking change, implement safe any type map, etc.

Basically this should be a deterministic secure hash of the Type without path.

How to represent `BitFlags`?

In substrate we have the following type:

#[repr(u64)]
#[derive(Encode, Decode, Clone, Copy, PartialEq, Eq, BitFlags, RuntimeDebug, TypeInfo)]
pub enum IdentityField {
	Display = 0b0000000000000000000000000000000000000000000000000000000000000001,
	Legal = 0b0000000000000000000000000000000000000000000000000000000000000010,
	Web = 0b0000000000000000000000000000000000000000000000000000000000000100,
	Riot = 0b0000000000000000000000000000000000000000000000000000000000001000,
	Email = 0b0000000000000000000000000000000000000000000000000000000000010000,
	PgpFingerprint = 0b0000000000000000000000000000000000000000000000000000000000100000,
	Image = 0b0000000000000000000000000000000000000000000000000000000001000000,
	Twitter = 0b0000000000000000000000000000000000000000000000000000000010000000,
}

It is wrapped in a BitFlags in the following wrapper type to allow custom encoding/decoding.

/// Wrapper type for `BitFlags<IdentityField>` that implements `Codec`.
#[derive(Clone, Copy, PartialEq, Default, RuntimeDebug)]
pub struct IdentityFields(pub(crate) BitFlags<IdentityField>);

Question is how to represent this with scale_info: it is encoded as a raw u64 but we need to also have access to the IdentityField enum definition which maps the discriminants reduced to u8 indices.

A temporary workaround is to provide the following TypeInfo implementation

impl TypeInfo for IdentityFields {
	type Identity = Self;

	fn type_info() -> Type {
		Type::builder()
			.path(Path::new("BitFlags", module_path!()))
			.type_params(vec![ TypeParameter::new("T", Some(meta_type::<IdentityField>())) ])
			.composite(
			Fields::unnamed()
				.field(|f| f.ty::<u64>().type_name("IdentityField")),
		)
	}
}

See how it represents BitFlags<IdentityField> including the IdentityField as a type parameter, and the field itself as type u64.

However this requires some string matching for downstream type generators, so a preferable solution might be to add fist class support for a BitFlags like TypeDef variant.

On the other we might not want to be adding a new TypeDef variant for each corner case, be curious to see what others think.

/cc @jacogr

Trait bounds: derive as much as is reasonable, `#[scale_info(bounds())]` for the rest

In a meeting it was decided that we put forward with scale-info in the following way:

  • We will go with the "hacks" that have already been implemented for it and for parity-scale-codec crates to cover the most common use cases with 80/20 rule.
  • For the other 20% of cases that are not covered by the scale-info and parity-scale-codec derive macros we will provide a feature for users to define custom trait bounds similar to the design serde chose. Those custom trait bounds will be replacing the generated ones instead of being additive.
  • We might consider in the future to remove the "cruft" of "hacks" for common cases and consider removing them altogether with a major release for both derives.

Originally posted by @Robbepop in #76 (comment)

Reconsider handling of parameterized strings

Further to #35, we should explore ways to either improve this API or even completely change approach.

The goal is to avoid the confusing API where users need to supply String as a parameter to PortableForm when decoding/deserializing.

Note that this is a constraint introduced by the requirement of integrating scale-info into substrate runtimes.

Ideas discussed with @Robbepop

  • Add a crate feature which defines the type of the String at compile time.
  • Remove the metadata definitions from the runtime wasm in substrate and generate it as a separate process similar to ink!. Note that this is something I will explore anyway, and if we do it we will be able to generate metadata in the std environment which could just use Strings

Rename `Composite` and `Variant` type defs

@Robbepop and I went back and forth over this originally and settled on Composite and Variant as non-rust specific names for struct and enum types.

However I think this naming is slightly awkward in places, especially when writing code for a "variant" type which itself contains "variants".

So I think we should just use the enum and struct terminology, people understand what they are and the code will be easier to read/write. Also that is the terminology used in the SCALE defitions: https://substrate.dev/docs/en/knowledgebase/advanced/codec#enumerations-tagged-unions

Refactor Composite and Variant type builder duplication

The Composite and Variant type builders both create a Path and type_params before finalizing with either fields or variants. Consider ways to unify the first stage of these builders, could be done in conjunction with #6 (which deals with type_params)

Generics

As a follow up to #3, from @Robbepop: #3 (comment). Should be included in a 0.1 release.

Seeing this we still have the problem that our encoding is not generics-aware which it should imo. So we end up having a different type definition for each Option<T> with a different T instead of re-using the one generic definition. I have made some thoughts about this and came to the conclusion that it should even be fairly simple to actually implement this with some additional built-ins.

There are actually 2 ways we could implement this:

By introducing two new built-in type classes generic_type and generic_param:

{
    "strings": {
        "Option", # ID 1
        "Some",   # ID 2
        "None",   # ID 3
        "T",      # ID 4
        "Foo",    # ID 5
        "Bar",    # ID 6
        "Baz",    # ID 7
    }
    "types": {
        { # ID 1
            "variant": {
                "path" [ 1 ],
                "params": [ 4 ]
                "variants": [
                    {
                        "name": [ 2 ],
                        "type": [ 2 ],
                    },
                    {
                        "name": [ 3 ],
                    }
                ],
            }
        },
        { # ID 2
            # Generic parameter with name `"T"`.
            "generic_param": {
                "name": 4
            }
        },
        { # ID 3
            "primitive": "u32"
        },
        { # ID 4
            "primitive": "bool"
        },
        { # ID 5
            # Forwards to the `Option<u32>` type.
            "generic_type": {
                "type": 1,
                "params": 3,
            }
        },
        { # ID 6
            # Demonstrates usage of a type that contains a Option<u32> type.
            "composite": {
                "path": [ 5 ],
                "fields": [
                    {
                        "type": 5
                    }
                ],
            }
        },
        { # ID 7
            # Forwards to the `Option<bool>` type.
            "generic_type": {
                "type": 1,
                "params": 4,
            }
        },
        { # ID 8
            # Demonstrates usage of a type that contains a Option<bool> type.
            "composite": {
                "path": [ 6 ],
                "fields": [
                    {
                        "type": 7
                    }
                ],
            }
        },
        { # ID 9
            # Demonstrates usage of a non-generic type.
            "composite": {
                "path": [ 7 ],
                "field": [
                    {
                        "type": 3,
                    },
                    {
                        "type": 4,
                    },
                ]
            }
        }
    }
}

By introducing flags for all type fields:

{
    "strings": {
        "Option", # ID 1
        "Some",   # ID 2
        "None",   # ID 3
        "T",      # ID 4
        "Foo",    # ID 5
        "Bar",    # ID 6
        "Baz",    # ID 7
    }
    "types": {
        { # ID 1
            "variant": {
                "path" [ 1 ],
                "params": [ 4 ]
                "variants": [
                    {
                        "name": [ 2 ],
                        "type": [
                            {
                                # 4 indexes into the string table and must have
                                # the same index as one of the identifier in `params`.
                                "parameter": 4
                            }
                        ],
                    },
                    {
                        "name": [ 3 ],
                    }
                ],
            }
        },
        { # ID 3
            "primitive": "u32"
        },
        { # ID 4
            "primitive": "bool"
        },
        { # ID 5
            # Demonstrates usage of a type that contains a Option<u32> type.
            "composite": {
                "path": [ 5 ],
                "fields": [
                    {
                        # Demonstrates usage of the generic type at index 1 (`Option<T>`)
                        # with a parameter of 4 where 4 indexes into the type table as `u32`.
                        "type": {
                            "generic": {
                                "type": 1,
                                "params": [ 3 ],
                            },
                        }
                    }
                ],
            }
        },
        { # ID 6
            # Demonstrates usage of a type that contains a Option<bool> type.
            "composite": {
                "path": [ 6 ],
                "fields": [
                    {
                        "type": {
                            # Demonstrates usage of the generic type at index 1 (`Option<T>`)
                            # with a parameter of 4 where 4 indexes into the type table as `bool`.
                            "generic": {
                                "type": 1,
                                "params": [ 4 ],
                            },
                        }
                    }
                ],
            }
        },
        { # ID 7
            # Demonstrates usage of a non-generic type.
            "composite": {
                "path": [ 7 ],
                "field": [
                    {
                        "type": {
                            # 3 indexes into the type table and refers to a non
                            # generic (aka concrete) type, in this case `u32`.
                            "concrete": 3
                        },
                    },
                    {
                        "type": {
                            # 4 indexes into the type table and refers to a non
                            # generic (aka concrete) type, in this case `bool`.
                            "concrete": 4
                        }
                    },
                ]
            }
        }
    }
}

Missing Clone implementations

I just came across a limitation that most *Spec types don't derive Clone, and was wondering wether there was a specific reason for that? I couldn't think of any, especially because the Form trait already requires Clone for its associated types.

if there is no reason not to add this, I'm happy to prepare a PR.

Add UI tests for crate renaming

#145 adds the ability to tell scale-info where the crate lives, which is super handy in some circumstances.

let's add a couple of UI tests for this feature.

Potentially we can rename scale-info in the test Cargo.toml and then use foo::{self as scale_info}; at the top of most tests to expose it in the "usual place", and then we can fully test that #[scale_info(crate = "foo")] works when it is only present under eg foo::.

Can scale-info support `skip` or `ignore` for some struct field?

for serde and parity-scale-codec, the macro provides skip to ignore some field to avoid encode/decode.

so is there the TypeInfo macro can also provide a sub macro to do the same thing? avoid this field into the type info.

this is useful when some field is just a cached field.

bitvec 1.0

Should we be targeting bitvec 1.0 now? (I know that the generic param order has switched around fyi)

Compact trait bounds overflow

When integrating the latest scale-info master into substrate (see https://github.com/paritytech/substrate/compare/aj-metadata-vnext) I receive the following error:

image

It appears to have been introduced in #53.

Removing the TypeInfo derive on MultiAddress makes it compile:

diff --git a/primitives/runtime/src/multiaddress.rs b/primitives/runtime/src/multiaddress.rs
index a47201008..d09cd7aca 100644
--- a/primitives/runtime/src/multiaddress.rs
+++ b/primitives/runtime/src/multiaddress.rs
@@ -21,7 +21,7 @@ use codec::{Encode, Decode};
 use sp_std::vec::Vec;
 
 /// A multi-format address wrapper for on-chain accounts.
-#[derive(Encode, Decode, PartialEq, Eq, Clone, crate::RuntimeDebug, scale_info::TypeInfo)]
+#[derive(Encode, Decode, PartialEq, Eq, Clone, crate::RuntimeDebug)]
 #[cfg_attr(feature = "std", derive(Hash))]
 pub enum MultiAddress<AccountId, AccountIndex> {
 	/// It's an account ID (pubkey).
diff --git a/primitives/runtime/src/traits.rs b/primitives/runtime/src/traits.rs
index 00fae6fe6..0fdf6d55b 100644
--- a/primitives/runtime/src/traits.rs
+++ b/primitives/runtime/src/traits.rs
@@ -203,7 +203,7 @@ pub trait Lookup {
 /// context.
 pub trait StaticLookup {
 	/// Type to lookup from.
-	type Source: Codec + Clone + PartialEq + Debug + scale_info::TypeInfo;
+	type Source: Codec + Clone + PartialEq + Debug;
 	/// Type to lookup into.
 	type Target;
 	/// Attempt a lookup.

Here is the expanded derived TypeInfo impl for MultiAddress:

impl<AccountId, AccountIndex> ::scale_info::TypeInfo for MultiAddress<AccountId, AccountIndex>
        where
            AccountId: ::scale_info::TypeInfo + 'static,
            AccountIndex: ::codec::HasCompact,
            <AccountIndex as ::codec::HasCompact>::Type: ::scale_info::TypeInfo + 'static,
            AccountId: ::scale_info::TypeInfo + 'static,
            AccountIndex: ::scale_info::TypeInfo + 'static,
        {
            type Identity = Self;
            fn type_info() -> ::scale_info::Type {
                ::scale_info::Type::builder()
                    .path(::scale_info::Path::new(
                        "MultiAddress",
                        "sp_runtime::multiaddress",
                    ))
                    .type_params(<[_]>::into_vec(box [
                        ::scale_info::meta_type::<AccountId>(),
                        ::scale_info::meta_type::<AccountIndex>(),
                    ]))
                    .variant(
                        ::scale_info::build::Variants::with_fields()
                            .variant(
                                "Id",
                                ::scale_info::build::Fields::unnamed()
                                    .field_of::<AccountId>("AccountId"),
                            )
                            .variant(
                                "Index",
                                ::scale_info::build::Fields::unnamed()
                                    .compact_of::<AccountIndex>("AccountIndex"),
                            )
                            .variant(
                                "Raw",
                                ::scale_info::build::Fields::unnamed()
                                    .field_of::<Vec<u8>>("Vec<u8>"),
                            )
                            .variant(
                                "Address32",
                                ::scale_info::build::Fields::unnamed()
                                    .field_of::<[u8; 32]>("[u8; 32]"),
                            )
                            .variant(
                                "Address20",
                                ::scale_info::build::Fields::unnamed()
                                    .field_of::<[u8; 20]>("[u8; 20]"),
                            ),
                    )
            }
        };

TypeInfo for types with 'static lifetime

Make this work:

impl TypeInfo for Me<'static> {
    type Identity = Self;
    fn type_info() -> scale_info::Type {
        // todo: .field_of<&'static Me<'static>>,
        unimplemented!()
    }
}

See the closed generics PR here.

Support "Voldemort" types for common structures (e.g. key/value mappings)

Following up on #3, required before cutting a 0.1 release. From @Robbepop #3 (comment)

I was having a chat with @jacogr and we had the idea to conceptualize certain common structures in the generated JSON SCALE info.
For example we conceptualize the structure of mapping types.
A mapping type encodes as a sequence over some other type that has a key and value field. That's it. Below you can see the JSON that could conceptualize such a scenario concretely. Note that it might be nice to have so-called Voldemord composites that do not have a name since it isn't needed here. We just generate the type at index 1 upon serialization and will never need its name.

{
    names: [
        "BTreeMap", # ID: 1
        "KeyValue", # ID: 2
        "K",        # ID: 3,
        "V",        # ID: 4,
        "key",      # ID: 5,
        "value",    # ID: 6,
    ]
    types: [
        { # ID: 1
            composite: {
                path: [ 2 ], # We actually don't need this.
                             # Could we define composite types without names?
                             # Does that make sense at all? It would be similar
                             # to the concept of Voldemord types.
                fields: [
                    { name: 5, type: 4 },
                    { name: 6, type: 5 },
                ]
            }
        },
        { # ID: 2
            sequence: {
                type: 1
            }
        },
        { # ID: 3
            composite: {
                path: [ 1 ],
                params: [ 4, 5 ],
                fields: [
                    type: 2
                ]
            }
        },
        ...
    ],
}

Substrate compatibility

As part of a proof-of-concept to utilize scale-info in substrate, this issue will track all the changes in this crate in order to make it work.

  • Add informational field type name #30
  • Consolidate common prelude for std and no_std and usage #33
  • Derive TypeInfo for fields with associated types without bounds #20
  • Optional serde feature #34
  • Parameterize CompactForm String for optional SCALE impl #35
  • Use PortableRegistry for encoding and serializing #40
  • Add a compact flag to Field #42
  • Handle more SCALE attributes: skip, index #44

For reference, all the above mentioned PRs are integrated into the https://github.com/paritytech/scale-info/compare/aj-substrate branch, which is the working branch I am using from my substrate branch PoC https://github.com/paritytech/substrate/compare/aj-metadata-vnext

Implement `TypeInfo` for `Duration`

Something in Substrate was merged recently, probably paritytech/substrate#13302, which made our code no longer compile with this error:

error[E0277]: the trait bound `std::time::Duration: scale_info::TypeInfo` is not satisfied
   --> crates/sp-consensus-subspace/src/lib.rs:479:1
    |
479 | / sp_api::decl_runtime_apis! {
480 | |     /// API necessary for block authorship with Subspace.
481 | |     pub trait SubspaceApi<RewardAddress: Encode + Decode> {
482 | |         /// The slot duration in milliseconds for Subspace.
...   |
534 | |     }
535 | | }
    | |_^ the trait `scale_info::TypeInfo` is not implemented for `std::time::Duration`

Encoding and decoding is supported for Duration, so TypeInfo should be too.

Support `r#` in the module path

scale-info now complains that All path segments should be valid Rust identifiers if a module path contains r#, uncovered in autonomys/subspace#330.

A quick solution could be simply ignoring r# in my opinion, not sure whether it has other side effects though.

@@ -97,7 +84,16 @@ impl Path {
     ///
     /// - If the type identifier or module path contain invalid Rust identifiers
     pub fn new(ident: &'static str, module_path: &'static str) -> Path {
-        let mut segments = module_path.split("::").collect::<Vec<_>>();
+        let mut segments = module_path
+            .split("::")
+            .map(|s| {
+                if let Some(stripped) = s.strip_prefix("r#") {
+                    stripped
+                } else {
+                    s
+                }
+            })
+            .collect::<Vec<_>>();

CC @nazar-pc

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.