filecoin-project / lotus Goto Github PK
View Code? Open in Web Editor NEWReference implementation of the Filecoin protocol, written in Go
Home Page: https://lotus.filecoin.io/
License: Other
Reference implementation of the Filecoin protocol, written in Go
Home Page: https://lotus.filecoin.io/
License: Other
To make starting test networks, it would be really nice to have a single command that creates a new genesis block filled with everything needed to initialize a network, starts mining on that genesis block (using keys and actors generated in that block) and then writes out both that genesis block and the bootstrap address of its libp2p node for others to connect to.
Got this while playing around with pond
runtime error: index out of range
goroutine 1523 [running]:
github.com/filecoin-project/go-lotus/chain.(*ChainStore).NearestCommonAncestor(0xc00021a910, 0xc009602e80, 0xc0095b3dc0, 0x0, 0x0, 0x0)
/Users/why/go/src/github.com/filecoin-project/go-lotus/chain/chain.go:428 +0x10d
github.com/filecoin-project/go-lotus/chain.(*Syncer).selectHead(0xc0003cd740, 0xc0003ee6c0, 0xc009607d70, 0x1, 0x1)
/Users/why/go/src/github.com/filecoin-project/go-lotus/chain/sync.go:416 +0x257
github.com/filecoin-project/go-lotus/chain.(*Syncer).SyncBootstrap(0xc0003cd740)
/Users/why/go/src/github.com/filecoin-project/go-lotus/chain/sync.go:195 +0x134
github.com/filecoin-project/go-lotus/chain.(*Syncer).InformNewHead.func1(0xc0003cd740, 0xc0095b3d40)
/Users/why/go/src/github.com/filecoin-project/go-lotus/chain/sync.go:154 +0x13b
created by github.com/filecoin-project/go-lotus/chain.(*Syncer).InformNewHead
/Users/why/go/src/github.com/filecoin-project/go-lotus/chain/sync.go:148 +0x143
Implement the multisig wallet.
This involves first writing and testing the actor (docs here ). Then, some API and CLI interfaces so the user can interact with it and use it.
Should be easy with libp2p eventbus
Theres a few new test cases we should add to the syncer. I'll just descibe them here briefly, using a notation of 'we want to sync from A to B, and the chain progresses to the right, and [ ... ]
represents 'any number of blocks'
[ A ] - [ ... ] - [ B ]
[ ] - [ ... ] - [ A ] - [ B ]
[ ] - [ ... ] - [ A ] - [ ... ] - [ B ]
[ ] - [ ...] - [ ] - [ ] - [ A ]
|
[ ] - [ ] - [ ] - [ B ]
definitely more to come after this, but if we make testing scenarios like this easy, then we can get some pretty good coverage.
In order to program around certain things happening (or not happening) on chain, we need a robust api that makes this easy to think about from the point of view of the person writing the logic.
Users should be able to use their hardware wallet to sign transactions. We should investigate how much effort it would be to add support for this directly in lotus (given some past experiments, I think its pretty straightforward).
(We also should store api token with limited permissions)
Reference the actors doc and the payments doc
First implement then actor, and add some tests. Then move on to API and CLI interface to the payment channels. We want to encourage people to use payment channels whenever possible for paying others, so making these APIs nice and usable is important.
quick sketch of cli thoughts:
lotus paych create <target> <total amount>
lotus paych voucher <channel> <amount> # sequence IDs could be tracked internally
lotus paych redeem <voucher>
lotus paych status <channel>
lotus paych list # list open channels to and from us
suspect that this is resolved in #93, but filing an issue just in case
Running this command creates the repo immediately, but if it fails, leaves it in place. We should clean up after ourselves.
Followup from #37, the actors code is done, now we just need a CLI interface.
Copied from the sketch in the previous issue:
lotus paych create <target> <total amount>
lotus paych voucher <channel> <amount> # sequence IDs could be tracked internally
lotus paych redeem <voucher>
lotus paych status <channel>
lotus paych list # list open channels to and from us
We're currently using FCS for serialization. The protocol working group has since decided that that wasnt the best idea, and now we're just using tuple serialized cbor-ipld everywhere. The net change is to stop using a custom CID, and remove the tag prefix on all the objects.
Additionally all the other objects in the system need to be serialized this way as well, including actor state, and parameters. We should even investigate using this for the HAMT's internal serialization.
This was mentioned on Slack, opening an issue so this doesn't get lost
I wrote gengen for go-filecoin. We probably want to do the same thing here, just with the spec compliant format.
We should be doing some amount of block validation before rebroadcasting blocks over pubsub, not the full validation, but some structural validation.
This is described in the block propagation spec: https://github.com/filecoin-project/specs/blob/master/data-propagation.md#block-propagation
right now the block reward is a fixed number, need to change this to the correct values from the spec.
If the current state of the chain says the nonce is too low, we should error out immediately (failing out the call the MpoolPush
)
In a recent go-filecoin PR, @laser extracted a bunch of the sector builder binding code into a separate package. Now we need to pull that into a separate repo so we can import it and use it.
Additionally, it will be generally useful to be able to easily write standalone tools that utilize that code.
In order to make testing easier, and also to generally improve usability, lotus should make it easy to import and export chains to a car file. Additionally, lotus should be able to verify a car file in place (without importing).
The immediate value here is that it makes cross-implementation testing much easier to perform, simply run lotus for a while, do a bunch of things, export the chain, and make sure another implementation validates it.
$ lotus chain import chain.car
$ lotus chain export chain.car
$ lotus chain verify chain.car
We should construct a set of synthetic benchmarks for chain processing.
Some initial ones i'd like to see:
More later for sure, but this is a good start.
with a daemon running, some commands work, chain head doesnt:
why@WhyNet ~/g/s/g/f/go-lotus> ./lotus wallet list
t1hzvnuhscaz7vgekajur7ipqrqrwgmdv56ne3nna
t3rv34lpcpxoqx6scl767zzdnsuiipfoej3owelsmluyx4z6cz5eenxdtxhrrttmz5e7quwshwqb4v7nvtsejq
why@WhyNet ~/g/s/g/f/go-lotus> ./lotus chain head
2019/07/17 22:39:14 RPC client error: unknown address network
# base level necessary things
lotus wallet sign-msg HEXENCODEDMESSAGE
lotus mpool push HEXENCODEDSIGNEDMESSAGE
lotus mpool pending
# calls method on an actor, does not send a message, really only for 'read
# only' things as a way to query chain state
# NOTE: ethereum has an explicit "estimate gas" api. We could also have that, but having this call return a receipt with the gas it would have used might be a nice api
lotus ??? call ADDRESS METHOD PARAMS...
# sends a message to the network invoking the given method with params on the
# specified actor. under the hood, this creates, signs, and pushes a message to
# the network
# To consider: should this block until its included in the chain? (at least multiple minutes)
lotus ??? invoke ADDRESS METHOD PARAMS...
# TODO: wallet commands to create and manage keys (take hardware wallets into account)
# go-filecoin wallet code (by dig) was inspired by ethereum, we should
# investigate, its likely good to use
lotus wallet ??
lotus wallet list
lotus wallet balance ADDRESS
lotus chain head
lotus chain getactor ADDRRESS # shows all actor info, like balance, code cid, state root, ID address and resolved normal address
lotus chain getblock <by height | by hash>
lotus chain getmsg <hash>
lotus chain pushblock <serialized block + cid list> # for a miner to submit a block they made
lotus node status # maybe too general?
# alternatively... just get the specific thing we want
lotus node issyncing
lotus node gasprice
lotus swarm # libp2p funtimes
lotus config # you know the drill
# TODO: Mining api!
# if this is a separate process, it needs an api, if its not, it needs api entries here...
# we want to be able to support multiple different miner actors in a single process
# do we separate block production from proving? if so, how?
lotus-miner status
lotus-miner set-price <miner> <price> # assuming theres only one discriminator for whether or not to accept deals
lotus-miner list-miners
????
Ethereum API for reference: https://github.com/ethereum/wiki/wiki/JSON-RPC
Bitcoin core api: https://en.bitcoin.it/wiki/Original_Bitcoin_client/API_Calls_list
Zcash api docs: https://zcash.readthedocs.io/en/latest/rtd_pages/payment_api.html (they extended bitcoins apis a bit to add stuff for sheilded addresses, including a sort of 'background jobs' api for snark proving)
This is a meta issue looking at what is still needed for complete deal flow:
PieceInclusionProof
to clientAddPiece
does, is it?We need a more optimized datastructure for storing sector metadata in the state tree. The HAMT (probably) won't cut it in the long term.
Research has been done in this direction here: filecoin-project/specs#116
We should implement the described datastructure and start using it.
When referencing messages by their CID, we need to take into account that signatures are stripped from BLS signed messages before inclusion into a block. So if we take the Cid of a BLS 'SignedMessage' (where the hash includes the message) then someone looking into the block for that message will not be able to find it. We need to make sure that the Cid of a BLS signed message is always just the Cid() of the unsigned 'Message'.
thoughts @Kubuxu @magik6k @dignifiedquire ?
require.go:157:
Error Trace: sync_test.go:167
sync_test.go:239
Error: Not equal:
expected: 0x32
actual : 0x0
Test: TestSyncMining
This test is time based and occasionally fails on CI. We should integrate the eventbus at some point to consume sync events and use that to signal when the test should move on
i'm working on moving the block miner to be spec compliant. Part of this involves several signatures with the miners worker key.
This means that the process creating blocks on behalf of a given storage miner needs access to that storage miners key.
I think the most straightforward way of doing this would be to just have the full node keep the key for the storage miner, instead of the current situation where the storage-miner has its own wallet.
Title says it all. obviously the daemon shouldnt crash, but we could probably detect the connection close and exit.
See if existing jsonrpc clients actually work with our lib
2019/08/29 20:21:08 Error when uploading spans to Jaeger: write udp 127.0.0.1:33423->127.0.0.1:6831: write: connection refused
2019-08-29T20:22:32.043-0700 ERROR rpc jsonrpc/handler.go:122 panic in rpc method: runtime error: invalid memory address or nil pointer dereference
github.com/filecoin-project/go-lotus/lib/jsonrpc.doCall.func1
/home/why/code/go-lotus/lib/jsonrpc/handler.go:122
runtime.gopanic
/home/why/go/src/runtime/panic.go:522
runtime.panicmem
/home/why/go/src/runtime/panic.go:82
runtime.sigpanic
/home/why/go/src/runtime/signal_unix.go:390
github.com/filecoin-project/go-lotus/chain/store.(*ChainStore).WaitForMessage
/home/why/code/go-lotus/chain/store/store.go:579
github.com/filecoin-project/go-lotus/node/impl/full.(*ChainAPI).ChainWaitMsg
/home/why/code/go-lotus/node/impl/full/chain.go:60
reflect.Value.call
/home/why/go/src/reflect/value.go:447
reflect.Value.Call
/home/why/go/src/reflect/value.go:308
github.com/filecoin-project/go-lotus/api.permissionedAny.func1
/home/why/code/go-lotus/api/permissioned.go:78
reflect.callReflect
/home/why/go/src/reflect/value.go:536
reflect.makeFuncStub
/home/why/go/src/reflect/asm_amd64.s:20
github.com/filecoin-project/go-lotus/api.(*FullNodeStruct).ChainWaitMsg
/home/why/code/go-lotus/api/struct.go:194
reflect.Value.call
/home/why/go/src/reflect/value.go:447
reflect.Value.Call
/home/why/go/src/reflect/value.go:308
github.com/filecoin-project/go-lotus/lib/jsonrpc.doCall
/home/why/code/go-lotus/lib/jsonrpc/handler.go:126
github.com/filecoin-project/go-lotus/lib/jsonrpc.handlers.handle
/home/why/code/go-lotus/lib/jsonrpc/handler.go:196
2019-08-29T20:22:32.044-0700 ERROR rpc jsonrpc/server.go:76 RPC Error: fatal error calling 'Filecoin.ChainWaitMsg': panic in rpc method: runtime error: invalid memory address or nil pointer dereference
github.com/filecoin-project/go-lotus/lib/jsonrpc.rpcError
/home/why/code/go-lotus/lib/jsonrpc/server.go:76
github.com/filecoin-project/go-lotus/lib/jsonrpc.handlers.handle
/home/why/code/go-lotus/lib/jsonrpc/handler.go:198
2019-08-29T20:22:32.044-0700 WARN rpc jsonrpc/server.go:82 rpc error: fatal error calling 'Filecoin.ChainWaitMsg': panic in rpc method: runtime error: invalid memory address or nil pointer dereference
Currently inokation interface is:
Send(to address.Address, method uint64, value types.BigInt, params []byte) ([]byte, uint8, error)
It is hard to differentiate which errors should be reported as errors and which should be return codes.
There are two ideas I have, either we change the interface to:
Send(...) ([]byte, actorError)
type aError struct {
msg string
err error
frame xerrors.Frame
retCode uint8
fatal bool
}
type actorError *aError
func NewFatalErr(msg string, err error) actorError {...}
fund NewRetErr(msg string, code uint8, err error) actorError {...}
// Fatal marks an error that should abort execution of message
func (actorError) IsFatal() bool {
return actorError.fatal
}
func (actorError) RetCode() uint8 {
return actorError.retCode
}
func IsFatal(err actorError) bool {
return err != nil && err.IsFatal()
}
func RetCode(err actorError) uint8 {
if err == nil {
return 0
} else { return err.RetCode() }
}
And create wrappers around all apis to return actorError.
Then handling of api call would be:
// If we want to recover
result, err := vmctx.Something()
if IsFatal(err) {
return nil, err // or: nil, Wrap(err, "optional msg")
}
if RetCode(err) != 0 {
// Recover
}
// If we don't want to try to recover
// If we want to recover
result, err := vmctx.Something()
if err != nil {
return nil, err// or: nil, Wrap(err, "optional msg")
}
It will allow us to understand why it takes so long (if it does) and where is the performance going.
The api needs a few things from a security standpoint:
Blocks are persisted right now, but the latest head is not, so when i restart my chain everything is brand new (unless i sync from some other peer)
We need to charge gas for the execution of messages. This has to be something that ends up in the spec, but good values will have to be driven by development.
The storage miner should be a separate process that interacts with a 'full node' process.
The storage miner is responsible for:
In order to make this all work, we need to expose a number of APIs:
Implement the retrieval protocol.
This is a very basic "send the miner money and hope they are kind enough to actually send back the data" sort of thing. We will also need some basic API/CLI interface to it, i sketched some out in the linked doc a while ago, but its likely not ideal.
To simplify basic things like getting a storage miner peerid, something like:
act := apiactor.StorageMiner(api) // Could also take options like tipset
peerid := act.GetPeerID()
We may also want a variant, which can commit messages to chain, but that can be done later.
Currently when the client disconnects the behaviour isn't clearly defined.
When the client loses connection, we should block all calls and try to reconnect in background.
This will be a change in the codegen lib, but we need to make sure it happens
We should have benchmarks for the serializations of our types, include the basic ones like block and message, and the more complex ones like the HAMT.
We should be able to run make bench
to run all the benchmarks and produce a common output format. Something like:
{
"commit": {
"hash": "12fde71ba723bce7",
"parent": "dbb8912efa73b3c3",
"date": "03-07-2019 4:17 UTC"
},
"tags": ["something"],
"submitter": "whyrusleeping",
"runDate": "Tue Jul 30 11:55:58 PDT 2019",
"system": {
"cpu": "i7-9900k",
"os": "linux 4.21"
},
"benchmarks": {
"chain-process-10000-m10": {
"timeperop": {"value": 12501492, "type":"ns", "stdev": 12662},
"allocsperop": {"value": 125, "type": "count"},
"bytesperop": {"value": 150000, "type": "bytes"},
},
"serialize-block": {
"timeperop": {"value": 129853, "type": "ns"},
"allocsperop": {"value": 4, "type": "count"},
"bytesperop": {"value": 126520000, "type":"bytes"},
},
.... many more ...
}
Once its easy to run benchmarks and produce nice output like this, we can write a dumb little web server that collects these (maybe with some auth?) and displays them on a graph. The engine working group can probably help us out there.
Gas charging of them is a bit weird right now and IDK what should happen with them in general.
What I mean by NOOP is not invoking a method nor transferring funds.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.