application-research / filclient

A minimal filecoin client library

filclient

A standalone client library that enables users to interact with the Filecoin storage network. The types of interactions available are listed below, under "Features".

This functionality is accomplished by allowing users to interact with the API of any Lotus Node on the Filecoin storage network. One such node is api.chain.love, which is a node hosted by the Outercore Engineering team.

Features

• Make storage deals with miners
  • Query storage ask prices
  • Construct and sign deal proposals
  • Send deal proposals over the network to miners
  • Send data to miners via data transfer and graphsync
  • Check status of a deal with a miner
• Make retrieval deals with miners
  • Query retrieval ask prices
  • Run retrieval protocol, get data
• Data transfer management
  • Ability to query data transfer (in or out) status
• Market funds management
  • Check and add to market escrow balance
• Local message signing capabilities and wallet management

Roadmap

• Cleanup and organization of functions
• Godocs on main methods
• Direct mempool integration (to avoid relying on a lotus gateway node)
• Cleanup of dependency tree
  • Remove dependency on filecoin-ffi
  • Remove dependency on go-fil-markets
  • Remove dependency on main lotus repo
• Good usage examples
  • Sample application using filclient
• Integration testing

License

MIT

filclient's Issues

State metrics into filclient

segfault on RetrieveContent with nil miner

this should error cleanly

Should be able to do free filc retrievals without a wallet address

We should skip payment channel step on zero cost retrievals (WIP)

Stop using the Lotus `itest` framework for integration tests

What

I want to raise awareness that this repo's use of Lotus itest framework for integration tests makes updates of modules dependencies quite difficult, or at minimum tied to lotus also updating sudependencies. These tests work by spinning up simulation nodes of lotus and lotus miner as well as filclient in a single process within a go test. Because these nodes are all in the same process, they are all essentially locked to the same go module dependency graph in order to compile. This essentially means that not only is filclient locked to its Lotus version, but it's locked to all of Lotus's sub dependencies. This means for example the entire libp2p and transfer stack must be on the same version as Lotus

Proposed solution

Bedrock has the same problem in Boost. I've filed an issue here -- filecoin-project/boost#990 -- my proposed solution is something based on docker, though the specifics aren't ironed out. I think once we solve it in Boost we can try to port a similar situation to filclient, for feel free to try your own approaches.

Automatically fill `--announce` flag on filc `deal` subcommand

When the boost PR merges, filc users will need to fill the --announce flag with /ip4/[public ip]/tcp/6755 when making deals. This is pretty un-discoverable for general users, and I don't think there's any reason not to just abstract this away in most cases.

It's fine just to do this at the filc level. This feature is mainly for CLI accessibility. The main filclient library can continue to require this field.

Return retrieval stats even after failure

If a retrieval starts and then fails, we still want to know how long we spent, how much FIL we paid, how much was received, etc. Right now, we just return an error that says uh oh something went wrong!

We will probably want to rename RetrievalStats to RetrievalResult, and if a retrieval fails partway through, instead of returning a non-nil error, we will fill out the RetrievalResult and include an Outcome value or Err error inside of the struct.

Planned like this because returning valid data along with a non-nil error in the return tuple is weird and unexpected.

Protect main branch

Given that there are a lot of contributors to this project, should the main branch be protected?

Missing the ability to do offline deals

Currently when I try filc deal, the option is limited.

OPTIONS:
   --miner value, -m value  
   --verified               (default: false)
   --announce value         the public multi-address from which to download the data (for deals with protocol v120)
   --help, -h               show help (default: false)

I'd like to see it can do fil+ verified deals to both lotus and boost market nodes. That means more options are needed, such as commP, dataCid, price etc, and the ability to switch between wallet addresses containing the datacap.

Data programs version of Filclient (forked)

@dkkapur needs the following from filc:

• track retrieval failures to a central database
  • client address
  • piece CID
  • payload CID
  • timestamp of retrieval finished
  • storage provider ID
  • success/fail
    • if fail, reason
• this will be in a fork of filclient, no merge master - so as to not increase scope of filclient

Should do normal stdout printing for general command output, and logging library should be silent by default

Using logging library for default command output is presenting a lot of problems. For one, it has to be either enabled manually through env, or enabled automatically in the cli code itself (which overrides env and makes logger unconfigurable). Also, most users probably do not need timestamps. If timestamps are required, that should be handled by filc itself.

Cannot build Filclient Error : cannot use t.gsReqQueuedHook

Hi,

I cannot build the last master, get the following error :

github.com/filecoin-project/go-data-transfer/transport/graphsync

/opt/lotus/go/pkg/mod/github.com/filecoin-project/[email protected]/transport/graphsync/graphsync.go:362:88: cannot use t.gsReqQueuedHook (type func(peer.ID, graphsync.RequestData)) as type graphsync.OnIncomingRequestQueuedHook in argument to t.gs.RegisterIncomingRequestQueuedHook

Complete log :

DM lotus@lamia:/usr/src/filclient$ make

go build
go: downloading github.com/filecoin-project/lotus v1.13.2-0.20211214230829-0e2278cc76d0
go: downloading github.com/filecoin-project/go-data-transfer v1.12.0
go: downloading github.com/filecoin-project/go-fil-markets v1.13.5
go: downloading github.com/ipfs/go-datastore v0.5.1
go: downloading github.com/ipfs/go-ipfs-blockstore v1.1.2
go: downloading github.com/ipfs/go-graphsync v0.11.4
go: downloading github.com/filecoin-project/go-state-types v0.1.1
go: downloading go.opentelemetry.io/otel v1.2.0
go: downloading github.com/ipld/go-car v0.3.3
go: downloading github.com/libp2p/go-libp2p-core v0.13.0
go: downloading github.com/ipfs/go-log/v2 v2.4.0
go: downloading github.com/filecoin-project/go-padreader v0.0.1
go: downloading github.com/ipfs/go-ipld-cbor v0.0.6
go: downloading github.com/multiformats/go-multihash v0.1.0
go: downloading github.com/filecoin-project/specs-actors/v2 v2.3.6
go: downloading github.com/multiformats/go-base32 v0.0.4
go: downloading go.opentelemetry.io/otel/trace v1.2.0
go: downloading github.com/mattn/go-isatty v0.0.14
go: downloading github.com/ipfs/go-ipfs-ds-help v1.1.0
go: downloading github.com/ipfs/go-merkledag v0.5.1
go: downloading github.com/ipld/go-ipld-prime v0.14.3
go: downloading golang.org/x/sys v0.0.0-20211209171907-798191bca915
go: downloading golang.org/x/crypto v0.0.0-20211209193657-4570a0811e8b
go: downloading lukechampine.com/blake3 v1.1.7
go: downloading github.com/ipfs/go-peertaskqueue v0.7.1
go: downloading github.com/libp2p/go-msgio v0.1.0
go: downloading github.com/filecoin-project/go-ds-versioning v0.1.1
go: downloading github.com/ipfs/go-blockservice v0.2.1
go: downloading github.com/ipfs/go-ipld-legacy v0.1.1
go: downloading github.com/filecoin-project/go-statestore v0.2.0
go: downloading github.com/benbjohnson/clock v1.2.0
go: downloading github.com/ipfs/go-ipfs-exchange-interface v0.1.0
go: downloading github.com/filecoin-project/go-crypto v0.0.1
go: downloading github.com/filecoin-project/go-jsonrpc v0.1.5
go: downloading github.com/libp2p/go-libp2p-pubsub v0.6.0
go: downloading github.com/ipld/go-ipld-selector-text-lite v0.0.1
go: downloading github.com/BurntSushi/toml v0.4.1
go: downloading github.com/ipfs/go-ds-leveldb v0.5.0
go: downloading github.com/dgraph-io/badger/v2 v2.2007.3
go: downloading github.com/ipfs/go-ds-measure v0.2.0
go: downloading github.com/ipfs/go-ds-badger2 v0.1.2
go: downloading github.com/gbrlsnchs/jwt/v3 v3.0.1
go: downloading google.golang.org/genproto v0.0.0-20210917145530-b395a37504d4
go: downloading github.com/filecoin-project/dagstore v0.4.4
go: downloading github.com/elastic/go-sysinfo v1.7.0
go: downloading github.com/go-kit/log v0.2.0
go: downloading golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2
go: downloading github.com/cilium/ebpf v0.2.0
go: downloading github.com/sirupsen/logrus v1.8.1
go: downloading github.com/coreos/go-systemd/v22 v22.3.2
go: downloading github.com/cespare/xxhash/v2 v2.1.2
go: downloading github.com/godbus/dbus/v5 v5.0.4
go: downloading github.com/go-logfmt/logfmt v0.5.1
go: downloading github.com/golang/snappy v0.0.3
go: downloading github.com/libp2p/go-libp2p-discovery v0.6.0
go: downloading github.com/dgraph-io/ristretto v0.1.0
go: downloading github.com/libp2p/go-libp2p-peerstore v0.6.0
go: downloading github.com/golang/glog v1.0.0
# github.com/filecoin-project/go-data-transfer/transport/graphsync
/opt/lotus/go/pkg/mod/github.com/filecoin-project/[email protected]/transport/graphsync/graphsync.go:362:88: cannot use t.gsReqQueuedHook (type func(peer.ID, graphsync.RequestData)) as type graphsync.OnIncomingRequestQueuedHook in argument to t.gs.RegisterIncomingRequestQueuedHook
make: *** [Makefile:66: filclient] Error 2```

Revisit datatransfer event handling in retrieval

I've been noticing some retrievals finishing instantly without error, without actually ever receiving any data. I think some important events from datatransfer are not being handled properly in some cases.

It's a little confusing to me which datatransfer events are finalizing events, and which ones are repeated / non-final, and even if multiple "finalizing" events can happen in the same run. filclient should properly handle all exit paths from datatransfer but this is not happening right now.

@hannahhoward some input from you on this would be appreciated!! would you mind looking over the datatransfer callback in the retrieval function?

`RetrieveContent`'s `CloseDataTransferChannel` returns an error which is unhandled

Handle `DealStatusFundsNeededLastPayment`

Could be causing issues potentially maybe?

Support input of transfer params

When calling filclient, there should be the ability to specify transfer params (use cases exist where trasnfer may not happen over libp2p, may be http etc)

Remove filecoin-ffi dependency

Increases build complexity for users

Need an option to enable detailed filc logging (timestamps, line numbers, etc)

I've been trying to determine how to make logs look nice while also keeping them simple. I realized those are two contradicting goals that should both be present, but separately. It'd be nice to print simple, plain output by default, and have a flag to enable all the extra timestamps/components/modules/etc. for debugging.

RetrieveContent failed

My version of filclient: github.com/application-research/filclient v0.0.0-20220504204501-ca86574e3ee5

I store the file to miner successfully, then retrieve the file using filclient RetrieveContent, the first retrieval successfully retrieves the file, but more calls fail to retrieve it. I think the method blocks at the location in screenshot 1 and cannot return the data. The logs are shown in screenshot 2

Add helper functions for error checking

Why's example --

instead of:

var clientErr *filclient.Error
		if !xerrors.As(err, &clientErr) && clientErr.Code == filclient.ErrLotusError {

maybe something like:

if filclient.ErrIs(err, filclient.ErrLotusError) {

`filc` - allow CLI flags to be passed in any order

Right now the commands will fail if you put flags after args. We should fix this so flags & args can be passed in any order.

Move filecoin retrieval progress logging out of filclient

This information should be provided through a channel or callback, and it should be up to the user of the library to print out this information if desired.

There is a flag in the FilClient struct relevant to this that will need to be removed.

On this topic, we should write a small abstraction for printing byte retrieval progress, since this process needs to be done for both FIL and IPFS downloads. It'd be nice to have rate limiting progress printing as well, so terminal writes don't possibly try to bottleneck someone's epic insane download speed.

Protect connections for retrieval queries

What

Autoretrieve is getting a ton of "stream reset" messages when making retrieval query-asks, suggesting that the connection is not protected and as a result, it's getting cleaned up by libp2p's connection manager while waiting for a response.

I think we should put connection protection around the overall process of making the query. (i.e. before opening the stream and until you get the response). Since there are two retrieval query methods (not sure why -- one should call the other probably) -- we'll have to put it in both.

FILC cancels retrievals while a download with blocks already cached is catching up

FILC automatically tries a different miner for retrievals if a download stalls for too long.

In the case that a download was started, stopped, and then restarted, many blocks will already be present in the blockstore and will be reported as retrieved very quickly. Meanwhile, the download will progress (generally a lot more slowly) also starting from the root node, and since the first downloaded blocks were already marked as retrieved, no progress will be shown for a while after that initial burst. This ends up appearing as a timeout, even though an active download is in fact running.

One way of fixing this could be to only report explicitly network-retrieved blocks to the retrieval progress callback.

Windows binary

Its possible to build'n'release windows binary?

`doRpc` occasionally hangs and does not respect context cancellation

doRpc sometimes hangs temporarily, followed by a stream reset several seconds later. during this time, context cancellations do nothing.

doRpc takes a context but all it does with it is copy the deadline to the inet stream, so a hanging read or write has no way to close out when the context is canceled.

a quick fix for this would be to start a goroutine that sets the stream deadline to zero if context cancellation is detected, but not sure how much overhead that'll introduce