Look into using boxo gateway module instead of our own. This will ensure that our gateway configuration / set up is consistent with boxo

• docker-compose.yaml
• service yml (for endpoints)

TBD

application-research/delta-dm#78

In order to better communicate the connection between Delta and Edge, rebrand EdgeUR to Delta Edge

Stats

Global and API key specific stats:

• total number and size of uploads
• total number and size of uploads with deal_ids
• day to day and month to month stats
• number of edge-ur peers
• number of CIDs hosted by this edge-node

Instead of using the myriad of legacy ipfs libraries, EngRes has produced the boxo initiative. This library should be used to replace blockstore, bitswap and all other direct calls to underlying libraries for edge-ur.

Update:
blocked by: ipfs/boxo#291
blocked by: ipfs/go-merkledag#105

Moving to boxo 0.9.0 yeilds the following:

~/Documents/code/edge-ur boxo-wip*
❯ make       
go generate
go build -tags netgo -ldflags="-s -w -X main.Commit=3b0eb73 -X main.Version=v1.0.2-53-g3b0eb73-dirty" -o edge
# github.com/ipfs/go-merkledag
../../../go/pkg/mod/github.com/ipfs/[email protected]/merkledag.go:26:9: undefined: format.Register
../../../go/pkg/mod/github.com/ipfs/[email protected]/merkledag.go:27:9: undefined: format.Register
../../../go/pkg/mod/github.com/ipfs/[email protected]/merkledag.go:28:9: undefined: format.Register
../../../go/pkg/mod/github.com/ipfs/[email protected]/merkledag.go:30:9: undefined: ipldlegacy.RegisterCodec
../../../go/pkg/mod/github.com/ipfs/[email protected]/merkledag.go:31:9: undefined: ipldlegacy.RegisterCodec
../../../go/pkg/mod/github.com/ipfs/[email protected]/merkledag.go:86:16: undefined: ipldlegacy.DecodeNode
../../../go/pkg/mod/github.com/ipfs/[email protected]/merkledag.go:146:16: undefined: ipldlegacy.DecodeNode
../../../go/pkg/mod/github.com/ipfs/[email protected]/merkledag.go:254:23: undefined: ipldlegacy.DecodeNode
make: *** [build] Error 2

Edge node should still be able to consolidate files even without an network connection or "immediate" connection with delta node.

• Edge should consolidate the request into a dispatch queue
• Dispatch queue monitors the network and dispatch the upload jobs when it's ready.

Generate the swagger file for all the edge-ur endpoints available

Quick wireframe

wireframe sample 1

wireframe sample 2

Steps to reproduce:

1. Attempt to upload a 835 GiB file to https://edge.estuary.tech/.
2. Results:

Uploading 835 GiB file to edge.
{"error":{"code":500,"reason":"Internal Server Error","details":"multipart: NextPart: bufio: buffer full"}}

Users should be able to retrieve a cid from the network using lassie.

• support for expiring api keys
• acl (download and upload control access)
• Flag to remove bootstrap nodes

Quick wireframe

wireframe sample 1

wireframe sample 2

Proposal

• Edge uses HTTP to transfer files over to delta.
• We need to add support for RPC and Bitswap transfers using delta /existing endpoint to connect directly with the edge node.

We would like to have S3 integration added to our overall architecture diagrams. Should demonstrate how Amazon/Azure/GCP S3 storage fits into the Delta stack.

cc: @schreck23 @jimmylee

A job the generates a car for every defined threshold (1GB by default).

Multiple users in the Enterprise space are looking to interface directly with S3. As an edge, delta user I want to be able to push data up to our services through the S3 api, and have it automatically flow the data through the backend into Filecoin per usual.