dfinity / candid Goto Github PK
View Code? Open in Web Editor NEWCandid Library for the Internet Computer
License: Apache License 2.0
Candid Library for the Internet Computer
License: Apache License 2.0
Candid applies some logic to every element of a vector, which is unnecessary for blobs since they're a single slice of memory.
This is going to consume a lot of gas for users so it should be made more efficient.
Encoding could also use a memcpy improvement in this use case.
The usage of blob
is frequent enough (e.g. Wallet installs and call forwarding) that it should be high priority.
#[derive(CandidType)]
struct Empty {};
#[derive(Deserialize)]
struct A { foo: Option<String> }
let bytes = Encode(&Empty {})?;
Decode!(&bytes, A)?;
The last line should fail, but it deserialized to None
. If foo
is a non-Option type, it correctly reports missing field foo
.
can interpret its own output
Meaning that given an output representing a diff, the tool can perform this diff as an edit
(new subcommand, joining echo
and diff
). The edit
subcommand takes a value and a diff and it produces a second value, related to the first by the diff.
Create a C API for Candid in Rust and compile it down to WASM to distribute with C SDK.
Probably something along the lines of impl<T: CandidType + ToOwned> CandidType for Cow<T> { ... }
.
The obvious advantage is to be able to serialize and deserialize without copying the data if not needed.
I suspect it's due to having r#
at the start of the field name, which should be ignored by Candid in the proc macro.
The spec currently says
Completeness: Subtyping covers all cases of successful deserialisation.
and I wonder what that precisely means, and if we really want it.
(Previous discussion at #128 (comment) )
An actor reference inherently is represented by a principal.
This also unblocks dfinity/motoko#2264
The new user’s guide, added in #158, does not fully address the JS-specific workflow and type mappings.
As a JS frontend developer, I want JS-specific Candid documentation, so that I can build JS frontends that use Candid well.
Is your feature request related to a problem? Please describe.
I want to write an actor/service like this in Motoko, with a corresponding Candid service type:
actor MapThing<X,Y> {
put : (X, ?Y) -> async ();
get : X -> async ?Y;
}
Leaving out some inessential details, this MapThing
actor provides a service whose API is parametric in two types, X
and Y
, for a generic key and value type, respectively. It stores a dynamic map from X
s to Y
s.
More generally, I want to author Candid services that use generic type arguments, and as a user, instantiate services by providing these types over the network.
Describe the solution you'd like
The solution here involves extending Candid, Motoko, as well as (perhaps) the Internet Computer itself.
Describe alternatives you've considered
Many systems and languages lack polymorphic types. The usual workarounds map a polymorphic design into a monomorphic one, somehow. For instance, we could assume a single choice of type for X
and Y
above, and try to provide a way to map into and out of these single choices (e.g., use [nat8]
for all types and do extra layers of serialization, the current solution).
Additional context
None.
Minimal repro:
use candid::{CandidType, Decode, Deserialize, Encode};
fn main() {
#[derive(CandidType, Deserialize)]
struct Wrapper(Vec<usize>);
let w = Wrapper(vec![1]);
let b = candid::Encode!(&w).expect("failed to encode");
candid::Decode!(&b[..], Wrapper).expect("failed to decode");
}
fails with:
thread 'X' panicked at 'failed to decode:
Message: "Type mismatch. Type on the wire: Vec; Provided type: Nat64"
States:
Trailing type: [I(-8)]
Trailing value: [01, 01, 00, 00, 00, 00, 00, 00, 00]
Type table: [[I(-20), U(1), U(0), I(1)], [I(-19), I(-8)]]
Remaining value types: []
Example: in a .did
file I have:
type ProposalId = nat64;
service : {
"change_proposal_deadline_for_tests": (proposal_id: ProposalId, timestamp_unix_epoch_nanos: nat64) -> ();
}
In the web UI, I get 2 input fields both labelled nat64
--- so it's hard to tell apart the 2 arguments. It'd be nice if the Web UI would also display the arg name, here proposal_id
and timestamp_unix_epoch_nanos
.
Is your feature request related to a problem? Please describe.
When building CI workflows, we need a human-readable representation of how two Candid values relate when they are not equal.
This human-readable representation would be included in reports of CI failures (e.g., lists of "expected versus observed" test outputs).
To compute that representation meaningfully, we need some constructive math to define it.
Describe the solution you'd like
This issue proposes a new Rust crate, which would augment the existing candid
crate with higher-level implementations of various relations over candid values and types.
Let's call it candid_relate
.
It gives semantic operations over ASTs of Candid values and types, including:
mo:base/Order.Order
)This crate is not essential for building candid services, but would be essential for any candid tool that tries to debug expected versus observed values of candid data.
Here's an example of some related uses, demonstrating diff
, delta
and equal
, and how they inter-relate:
// suppose we have two Candid ASTs, and a type that classifies each:
let (x : candid::Ast, y : candid::Ast, t : candid::Type) = ... ;
// (note: candid values x and y each have a common candid super-type, t)
// At the common type, we can edit one value into the other
let d : candid_relate::Delta = candid_relate::diff(x, y, t);
// difference/delta is represented by d, and can be applied as follows:
let z = candid_relate::delta(x, d);
// equal at the common supertype t, but perhaps not generally:
assert_eq!( candid::equal(z, y, t) );
When two values are not equal, a CI script could use this crate to help pretty-print a report that summarizes ("usefully") how the two values differ, e.g., by implementing the Show
trait, or something similar:
println!("{}", d)
Describe alternatives you've considered
Additional context
This came up while trying to build useful tools (in Rust) for CI to run to generate reports, and reports about failures of IC services written in Motoko/Rust.
Caching and cache repair: The proposed "incremental edits representation" is (potentially) related to a future effort that would make query results systematically incremental across the Internet Computer. Just as we prescribe a representation of data by designing Candid, the representation of edits to these values could also be shared/systematic/general.
The Candid repo currently contains two versions of the spec, one in .adoc the other in .md format. And the former seems out of sync with the latter.
We should either generate one from the other, or get rid of one of the copies.
Is your feature request related to a problem? Please describe.
An example candid deserialization error message is:
"No more values to deserialize".
That can be hard to debug: what I want to know is:
Describe the solution you'd like
I'd like something like:
"No more value to deserialize. While trying to deserialize into a , the end of the byte stream was reached, but a was expected."
Describe alternatives you've considered
I have none.
Additional context
Can you please help me to understand a couple of things?
candid/IDL.md
section Serialisation -> Types says:
We assume that the fields in a record or function type are sorted by increasing id and the methods in a service are sorted by name.
What is "sorted by increasing id" in these terms? Sorted in an order they appear in source .did
file?
How exactly does the type definition table works (one which assigns uint to a type)?
Is this correct that expressions of kind type Example = opt record { nat; nat; }
(in other words - type definition expressions) are added to the type definition table? And every other appearance of type "Example" (when used as datatype id) should be serialized as the index of "Example" in the type definition table?
Is this correct that the type in expression like
service A {
foo : (vec opt text) -> () oneway;
^------^ this complete type
}
is also added to the type definition table if it didn't appear earlier? Even if there was no expression like type OptText = opt text;
met before.
After parse of a .did
file one can end up having a "complete" type definition table - which contains every type that appear in a file. Should she use this type definition table when serializing a method call or she should somehow filter this table so it would contain only types that are really needed for the serialization of this method call? Is the order of types in the type definition table matters? How do one assert the correct order?
The new user’s guide, added in #158, does not fully address the Rust-specific workflow and type mappings.
As a rust canister developer, I want Rust-specific Candid documentation, so that I can build Rust canisters that use Candid.
Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Currently, all the functions that do the serialization construct a new buffer from scratch (e.g., https://docs.rs/candid/0.6.17/candid/ser/fn.encode_one.html).
This might lead to unnecessary work when serializing large primitives (e.g., large blobs).
It would be nice if the API could work with any implementation of Write
.
This way one would be able to write to the message reply buffer directly using something like
pub struct ReplyAppender;
impl std::io::Write for ReplyAppender {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
// We can be a bit smarter and have a buffer for very small writes
// and call append directly for big writes, depending on the cost model.
ic0::msg_reply_data_append(buf.as_ptr() as u32, buf.len() as u32);
Ok(buf.len())
}
fn flush(&mut self) -> std::io::Result<()> { Ok(()) }
}
This will eliminate one extra copy when sending the data to the clients (which might easily be the main part of the work performed by the canister).
serde serializes SystemTime
as Duration
from std::time::UNIX_EPOCH
. Having a CandidType
implementation for it that does the same would be useful when sending SystemTime
in a message.
I actually tried to implement this myself but failed because I don't understand how to generate named fields in candid. The serde implementation for SystemTime
wants a record with secs_since_epoch
and nanos_since_epoch
fields, for the second and nanosecond parts, respectively. I don't know if that's possible to do in candid.
#[serde(rename_all = "snake_case")]
pub enum CanisterStatusType {
Running,
Stopping,
Stopped,
}
The Query/Call buttons are not type=submit
, so they don't trigger when pressing Enter.
Right now we have to use an IDLValue
which isn't ideal (as then we have to match everywhere). There should be a native encapsulation, similar to how we do Nat
and Int
.
Finished by matthewhammer/motoko-bigtest#10
Originally:
I would like to show examples of using the Candid tools within Github Actions:
dfx canister call MyCanister getFullLog
) let bytes = Encode!(&1023)?;
let x = Decode!(&bytes, i8)?;
assert_eq!(x, -1);
We have two structures;
#[derive(Deserialize)]
struct CallCanisterArgs1 {
canister: Principal,
method_name: String,
args: Vec<u8>,
cycles: u64,
}
// RUNNING OUT OF CYCLE
#[derive(Deserialize)]
struct CallCanisterArgs2 {
canister: Principal,
method_name: String,
cycles: u64,
}
The CallCanisterArgs2
runs out of cycle if we try to deserialize the same value CallCanisterArgs1
would deserialize, which points to a performance regression when skipping fields.
We keep getting user questions about Candid, e.g., about its syntax. But all documentation we currently have is the "spec", which is not really targeted at users and is overly technical.
We should have some primary "user guide" that is accessible for programmers, not implementers.
when dfx call
takes an opt
type, the opt subtyping rule essentially turns off the type checking. We need to disable or at least warn about the use of opt subtyping rules in the parser.
Looks like we still haven’t found the grail yet…
TL;DR: Our relaxed coercion rules for opt (#110 and following) may lead to “hard” deserialization errors later on.
I seems that the following can’t all hold:
Here is the counter example:
service A1 : { bar : () -> () }
service B1 : { baz : () -> () }
service C : {
init : (service A1, service B1) -> (); // stores (a1, b1)
run : () -> () ; // runs a1.bar(b1.baz())
}
If we pass A1
and B1
to C.init
, all is well. C.run
works fine.
Now we upgrade A1
to A2
and B1
to B2
, as follows:
type WantsBool = service : { foo : (bool) -> () }
type WantsInt = service : { foo : (int) -> () }
service wantsInt : WantsInt
service A2 : { bar : (opt WantsBool) -> () } // calls `arg.foo(true)` if `arg` is not `null`
service B2 : { baz : () -> (opt WantsInt) } // returns (opt wantsInt)
These upgrades are permitted by our :<
, and actually desirable (i.e. restricting <:
to avoid “obviously bogus” upgrades doesn’t help)
But now C.run
will cause B2 to pass opt wantsInt
to A2
. This will succeed at deserialization (because “Function and services references coerce unconditionally”). Now A2 invokes wantsInt.foo(true)
, and the deserialization of (true)
at expected type (bool)
will fail in wantsInt
.
In https://github.com/dfinity/candid/blob/master/spec/IDL-Soundness.md#proof-for-canonical-subtyping we have a generic soundness proof for “Canonical subtyping”, that says “A solution with canonical subtyping (transitive, contravariant) is sound.”. And our <:
is that!
But it’s not “decompositional” any more. In particular,
If
t1 <: t2
ands1 in t1 <: s2 in t2
thens1 <: s2
.
(as required in the generic proof) doesn’t hold .
Concretely in our example above, we have opt WantsInt <: opt WantsBool
(because all opt
types related). But because the reference value is passed through, this also means WantsInt in opt WantsInt <: WantsInt <: opt WantsBool
, but we don’t have WantsInt <: WantsBool
.
Dunno. No concrete suggestions. It’s sunday morning, and I just wanted to get this insight out of my head.
I'm building a custom IDL parser using this repository's spec and tests, but it seems like there is an error.
parse_type.rs
type broker = service {
find : (name: text) ->
(service {up:() -> (); current:() -> (nat32)}); <<< missing semicolon here, but compiles
};
IDL.md
<actortype> ::= { <methtype>;* } <<< semicolon is not optional
From the point of view of the spec, this field's type does not matter; whatever the canister returns we send back in the same shape. But from the Candid file documenting the HTTP Query spec, it becomes a simple record {}
because there's no way to communicate this properly in Candid.
Either Candid could implement a generic (in which case the token becomes T
) or the any
type, as the token could change type between two responses (and the spec does not actually care).
I often need a little more than the signature definitions to get the right syntax for specifying Candid values on the command line (for example, do I need to use equal sign or colon, comma or semi-colon?).
Adding.a usage example for each type would make it easy to look up this information.
When a user mistakenly uses and invalid record field name, the following error message is observed.
Invalid data: Unable to convert IDL to bytes: field 2468089002 not found
This appears to reference the hash of the invalid record field name and not the invalid record field name itself, which is somewhat ambiguous. Would awesome if we could display the unhashed value so that the user sees:
Invalid data: Unable to convert IDL to bytes: field 'name' not found
Wasm interface types are a way for wasm modules to have a type-rich interface without any inherent serialization overhead. They're mostly intended to support low-cost calls between multiple wasm modules, but I suspect that they could be used for dfinity as well. They also don't require the module to include any serialization/deserialization code internally, all that gets generated at instantiation-time at the boundary automatically.
Interface types aren't ready for prime time, but if my understanding of candid is correct, there appears to be a lot of overlap.
Plus, languages like Rust will eventually support generating exported functions with interface types automatically without additional tooling.
enum E {
A(i8),
}
When encoding E
, Candid thinks field A
is of type record { 0: int8 }
, but decoding is expecting i8
directly.
Right now, if you pass in 0
as a number as is, the textual parser turns it into an Int
.
A better experience would be to have the textual parser uses the smallest type that can represent the number (e.g. 0-255
uses Nat8
, -257 - -65536
use Int16
, etc.
Then, have the candid values officially support Nat8 <: Nat16 <: Nat32 <: Nat64 <: Nat
and Int8 <: Int16 <: Int32 <: Int64 <: Int
then NatN <: IntN
and Nat <: Int
, thus removing any forced typings when entering numbers through the textual parser.
We currently have this rule:
<nat> not in <fieldtype>;*
record { <fieldtype>;* } <: record { <fieldtype'>;* } ~> f
------------------------------------------------------------------------------
record { <fieldtype>;* } <: record { <nat> : opt <datatype'>; <fieldtype'>;* }
~> \x.{f x with <nat> = null}
This gives record {} <: record { foo : opt bool }
.
We also have record { foo : opt bool } <: record { foo : reserved }
(by virtue of opt bool <: reserved
).
By transitivity, we should have record {} <: record { foo : reserved }
.
So it seems we need
<nat> not in <fieldtype>;*
record { <fieldtype>;* } <: record { <fieldtype'>;* } ~> f
------------------------------------------------------------------------------
record { <fieldtype>;* } <: record { <nat> : reserved; <fieldtype'>;* }
~> \x.{f x with <nat> = null}
as well.
(This uses null
as an arbitrary representation for a reserved value. Woudn’t mind adding an explicit reserved
to the textual representation and data model, that would be handy in other contexts as well.)
We could combine the two into a single rule by saying
<nat> not in <fieldtype>;*
record { <fieldtype>;* } <: record { <fieldtype'>;* } ~> f
opt empty <: <datatype'>
------------------------------------------------------------------------------
record { <fieldtype>;* } <: record { <nat> : <datatype'>; <fieldtype'>;* }
~> \x.{f x with <nat> = null}
If you try and compile the following code in a rust project without serde
as a dependency:
use candid::Deserialize;
#[derive(Deserialize)]
pub struct A();
You will get the following error
error[E0658]: use of unstable library feature 'rustc_private': this crate is being loaded from the sysroot, an unstable location; did you mean to load this crate from crates.io via `Cargo.toml` instead?
--> rust_canisters/dfn_http/src/lib.rs:3:10
|
3 | #[derive(Deserialize)]
| ^^^^^^^^^^^
|
= note: see issue #27812 <https://github.com/rust-lang/rust/issues/27812> for more information
= note: this error originates in a derive macro (in Nightly builds, run with -Z macro-backtrace for more info)
error: aborting due to previous error
For more information about this error, try `rustc --explain E0658`.
This is the case for any struct, but it is resolved as soon as you add serde as a dependency.
This is true for 0.25 and master at the time of writing.
The “option type can be specialised to its constituent type” still worries me…
Here is one problem, maybe fixable, but maybe indication that this rule just doesn't fit well with the system.
With this rule, we certainly want
true ~> opt true : opt bool
But also
true ~> opt true : opt bool
true ~> opt opt true : opt opt bool
true ~> opt opt opt true : opt opt opt bool
…
So what does this mean for
type FixOpt = opt FixOpt
true ~> ? : FixOpt
Let’s try to apply the rules (with a logical variable ?X
for the result).
true ~> ?X : FixOpt
⇐ opt true ~> ?X : FixOpt
⇐ opt true ~> opt ?Y : FixOpt -- ?X = opt Y
⇐ true ~> ?Y : FixOpt
and we are in a loop. The solution ?X = opt ?X
is not allowed (our values are inductive, not coinductive – I hope!). And since we understand _ ~> _ : _
to be inductive, we should get
not (∃x. true ~> x : FixOpt)
But by the rule about “failed parses at type opt
turn into null`”, this implies
opt true ~> null : opt FixOpt
which (because opt FixOpt = Opt
) implies
opt true ~> null : FixOpt
which implies
true ~> null : FixOpt
by the “subtype to constituent type” rules.
This is a blatant contradiction! This means our _ ~> _ : _
relation is inconsistent!
How can that happen? Don’t all inductively defined relations well-defined? No, they are only well-defined when the relation appears only in strictly positive positoins in the rules. And our rule
not (<v> ~> _ : <t>)
-------------------------
opt <v> ~> null : opt <t>
breaks that. This would also be a major headache when trying to put this into a theorem prover.
The way to deal with that there (e.g. Coq) would be to define the relation as a well-founded function on the v
(we can’t use t
because the types themselves are coinductive). In all rules, the antecents only mention subterm of v
. And it also (nicely) matches a real implementation which would traverse v
.
Well, all rules but his one:
<v> ≠ null
<v> ≠ (null : reserved)
<v> ≠ opt _
opt <v> ~> <v'> : opt <t>
-------------------------
<v> ~> <v'> : opt <t>
Maybe we can patch around this issue (e.g. requiring <t> ≠ opt <t>
) … but I am leaning towards just dropping it. It makes the formalism cleaner (a good sign) and it makes the implementation easier to get right and secure (they otherwise have to guard themselves against an infinite loop upon true ~> ? : FixOpt
).
Do we have any compelling reason/use case for this rule?
not (null <: <datatype>)
<datatype> <: <datatype'>
-----------------------------
<datatype> <: opt <datatype'>
Currently specifically says only Rust implementation, is that still accurate?
I started to do some baby steps in formalizing a “mini candid” in Coq, starting with the rules that seem safe.
But adding the “specialize to constituent type” rule is causing problems:
If the rule is
not (null <: t2) t1 <: t2
--------------------------------
t1 <: opt t2
as it is now, then this is not transitive. We have int <: opt int
(by this rule), opt int <: opt opt int
(by the normal rule), but not int <: opt opt int
(because of the not (null <: t2)
restriction).
This restriction was added in #135, upon my suggestion. Without that restriction, we have the problems described in #135, i.e. bad interaction with the “catch decoding rule”, and the problem that
t1 <: t2
------------------
t1 <: opt t2
doesn't make progress if t2 = opt t2
.
Or, to respond to @rossberg says in #135 (comment)
Interesting suggestion. I suppose options of options are not something you would see often in an interface, so limiting their evolution might be okay. How confident can we be that this has no problems?
I wish we had spare cycles to mechanise this, and prevent further holes.
I am no longer confident that this has no problems.
One way to maybe! taper over the problem is to restrict the other rule as well:
not (null <: t2) t1 <: t2
--------------------------------
opt t1 <: opt t2
so that you can’t go from opt int
to opt opt int
… but that essentially disallows evolution under opt opt
and might be better served by disallowing opt opt
… probably not nice. Or maybe this rule:
(null <: t1) iff (null <: 2) t1 <: t2
---------------------------------------
opt t1 <: opt t2
which rules out opt int <: opt opt int
without ruling out opt nat <: opt int
nor opt opt nat <: opt opt int
?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.