textileio / broker-core Goto Github PK
View Code? Open in Web Editor NEWBroker for the Filecoin network
License: MIT License
broker-core's People
Contributors
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
Stargazers
Watchers
Forkers
avichalpbroker-core's Issues
Enable deal cancelling post auction by the client
There may be some cases where an auction deal hasn't been created yet but the client needs to cancel the auction. Do we want to support this feature? It may be inevitable once we support remote signing
Auctioneer CI fails sometimes
--- FAIL: TestClient_RunAuction (98.64s)
auctioneer_test.go:153:
Error Trace: auctioneer_test.go:153
Error: "map[01fc6kmxh2f88rcjq5t2cqgvtq:{foo bar QmbwNQUCZwQxCnzbnPQKp3dCk9yk7gSuinXZKzEoeKbAMd %!s(int64=100000000000) %!s(int64=100000000000) %!s(uint64=2880) %!s(bool=true) 2021-08-03 18:15:40.32292096 +0000 UTC} 01fc6kmy81f4c01fcjmecjaks0:{foo bar QmcvLVh7WkTGcxBosDL2xz7Sa6d9QbXxfeMwWy3XGXNJ4P %!s(int64=100000000000) %!s(int64=100000000000) %!s(uint64=2880) %!s(bool=true) 2021-08-03 18:15:41.057535288 +0000 UTC} 01fc6kmycfpy0b4an8fspfra62:{foo bar QmWEVcvAjHTQGgjNWHNVzhwB5Tbc6SqNddCSWyd7HyYXQy %!s(int64=100000000000) %!s(int64=100000000000) %!s(uint64=2880) %!s(bool=true) 2021-08-03 18:15:41.199314981 +0000 UTC} 01fc6kmyqpxb55sfc8mfsctf17:{foo bar Qmdj1JQjLsTrKV2xhJaZTf8pQtKDdMgiL4cRSEyUo7fRJK %!s(int64=100000000000) %!s(int64=100000000000) %!s(uint64=2880) %!s(bool=true) 2021-08-03 18:15:41.558467831 +0000 UTC} 01fc6kmyyj1t3k7ps772g7w7jc:{foo bar QmYsZ4g3Kq4Qbx33Af3FxwiydYneQHgkDqCn92ZS5PZcHa %!s(int64=100000000000) %!s(int64=100000000000) %!s(uint64=2880) %!s(bool=true) 2021-08-03 18:15:41.778787028 +0000 UTC} 01fc6kmzj79d3jmjd72vf912sf:{foo bar Qmf34hHz9nBrgC6rjXbtEgBw1QnHoQEhoWEgsmnxvtwGnq %!s(int64=100000000000) %!s(int64=100000000000) %!s(uint64=2880) %!s(bool=true) 2021-08-03 18:15:42.40769367 +0000 UTC} 01fc6kmznsrxm85e7x4y29b5nn:{foo bar QmePfZYvd3Na3qNmDsRucTJEn5jrYqArifDSiZgi32ubre %!s(int64=100000000000) %!s(int64=100000000000) %!s(uint64=2880) %!s(bool=true) 2021-08-03 18:15:42.521151565 +0000 UTC} 01fc6kmzqhfjtkn3d1at305pw4:{foo bar QmWY83CSA6LVZnQrXbzEwwN8fRNMG6LxuVF6xNVuNpewww %!s(int64=100000000000) %!s(int64=100000000000) %!s(uint64=2880) %!s(bool=true) 2021-08-03 18:15:42.577516757 +0000 UTC} 01fc6kn0077bd7jvmgs8xym04g:{foo bar QmfA5w3oiLS2anhFENRB7uZs484yCfuVxEK1NniASrvRCX %!s(int64=100000000000) %!s(int64=100000000000) %!s(uint64=2880) %!s(bool=true) 2021-08-03 18:15:42.85512746 +0000 UTC}]" should have 10 item(s), but has 9
Test: TestClient_RunAuction
FAIL
FAIL github.com/textileio/broker-core/cmd/auctioneerd/auctioneer 117.597s
Enable batching of CIDs (needs evaluation)
The way @jsign had originally engineered the storage API was that a "store" method needed to be called after the upload method.
one benefit of going back to this method would be to
- use the
pushendpoint for a single session to build a user's dag in the IPFS cache. that would require some session state and the ability to keep pushing new files to a directory path. perhaps there is some usability issues to figure out too. e.g. if i push a single file, do i still get back a dir node with the file in it or just my file's cid directly? - use the
storeendpoint to then move that entire dag to filecoin (and longer-term ipfs pinning.
experimenting with that may get us less batching down the road. it would also help ensure that all a user's data (from within a single session) remains in the same deal... where as now, data i upload in 1 minute may end up in a different auction than data i upload a minute later.
SQL syntax error
➜ make sql-assets
for daemon in broker piecer dealer packer api; do cd cmd/${daemon}d/store && /Users/aaron/go/bin/go-bindata-v3.1.3 -pkg migrations -prefix migrations/ -o migrations/migrations.go -ignore=migrations.go migrations && /Users/aaron/go/bin/sqlc-v1.8.0 generate; cd -; done;
# package db
queries/unpin_job.sql:1:1: syntax error at or near "asc"
Enable status by CID
Right now we are using the unique request id for checking status. This has the downside of forcing dapps to maintain some state about data stored on top of the data stored.
I was wondering, why not change it to just check by CID. perhaps it could return the life-time information about deals containing a CID? a dapp could then even verify that, if a cid is stored, the expiration is sufficiently far in the future to maybe not even store again.
Enable d2a user to set deal deadline as a parameter
Probably a lot of ways we could do this. Raw number, or maybe we have categories, fast, regular, slow that we can then give guidance what they'll mean and have a bit more control still.
Auctioneer CI hangs
715 2021-08-06T15:12:39.1845476Z goroutine 2581 [semacquire, 9 minutes]:
716 2021-08-06T15:12:39.1846117Z sync.runtime_Semacquire(0xc0001a2d78)
717 2021-08-06T15:12:39.1846826Z /opt/hostedtoolcache/go/1.16.6/x64/src/runtime/sema.go:56 +0x45
718 2021-08-06T15:12:39.1847507Z sync.(*WaitGroup).Wait(0xc0001a2d70)
719 2021-08-06T15:12:39.1848237Z /opt/hostedtoolcache/go/1.16.6/x64/src/sync/waitgroup.go:130 +0xe5
720 2021-08-06T15:12:39.1849337Z github.com/textileio/broker-core/cmd/auctioneerd/auctioneer/queue.(*Queue).Close(0xc0001a2d20, 0x0, 0x0)
721 2021-08-06T15:12:39.1850645Z /home/runner/work/broker-core/broker-core/cmd/auctioneerd/auctioneer/queue/queue.go:115 +0x5f
722 2021-08-06T15:12:39.1852023Z github.com/textileio/go-libp2p-pubsub-rpc/finalizer.(*Finalizer).Cleanup(0xc001d6be78, 0x0, 0x0, 0x0, 0xc001387478)
723 2021-08-06T15:12:39.1853557Z /home/runner/go/pkg/mod/github.com/textileio/[email protected]/finalizer/finalizer.go:47 +0x10c
724 2021-08-06T15:12:39.1854922Z github.com/textileio/broker-core/cmd/auctioneerd/auctioneer.(*Auctioneer).Close(0xc0000f2820, 0x39, 0x37f)
725 2021-08-06T15:12:39.1856251Z /home/runner/work/broker-core/broker-core/cmd/auctioneerd/auctioneer/auctioneer.go:123 +0xd9
726 2021-08-06T15:12:39.1857633Z github.com/textileio/go-libp2p-pubsub-rpc/finalizer.(*Finalizer).Cleanup(0xc001e1d308, 0x0, 0x0, 0x5a43a0, 0x38e3118)
727 2021-08-06T15:12:39.1859042Z /home/runner/go/pkg/mod/github.com/textileio/[email protected]/finalizer/finalizer.go:47 +0x10c
728 2021-08-06T15:12:39.1860353Z github.com/textileio/broker-core/cmd/auctioneerd/service.(*Service).Close(0xc005f5dbf0, 0x0, 0x0)
729 2021-08-06T15:12:39.1861601Z /home/runner/work/broker-core/broker-core/cmd/auctioneerd/service/service.go:78 +0x113
730 2021-08-06T15:12:39.1862886Z github.com/textileio/go-libp2p-pubsub-rpc/finalizer.(*Finalizer).Cleanup(0xc001e1d200, 0x0, 0x0, 0xc001608900, 0xc040000000)
731 2021-08-06T15:12:39.1864368Z /home/runner/go/pkg/mod/github.com/textileio/[email protected]/finalizer/finalizer.go:47 +0x10c
732 2021-08-06T15:12:39.1865773Z github.com/textileio/broker-core/cmd/auctioneerd/auctioneer_test.newClient.func1()
733 2021-08-06T15:12:39.1867209Z /home/runner/work/broker-core/broker-core/cmd/auctioneerd/auctioneer/auctioneer_test.go:213 +0x57
734 2021-08-06T15:12:39.1868080Z testing.(*common).Cleanup.func1()
735 2021-08-06T15:12:39.1868983Z /opt/hostedtoolcache/go/1.16.6/x64/src/testing/testing.go:901 +0x1bf
736 2021-08-06T15:12:39.1869706Z testing.(*common).runCleanup(0xc001608900, 0x0, 0x0, 0x0)
737 2021-08-06T15:12:39.1870490Z /opt/hostedtoolcache/go/1.16.6/x64/src/testing/testing.go:1007 +0xaf
738 2021-08-06T15:12:39.1871235Z testing.tRunner.func2(0xc001608900)
739 2021-08-06T15:12:39.1871974Z /opt/hostedtoolcache/go/1.16.6/x64/src/testing/testing.go:1187 +0x75
740 2021-08-06T15:12:39.1872650Z runtime.Goexit()
741 2021-08-06T15:12:39.1873352Z /opt/hostedtoolcache/go/1.16.6/x64/src/runtime/panic.go:632 +0x146
742 2021-08-06T15:12:39.1873978Z testing.(*common).FailNow(0xc001608900)
743 2021-08-06T15:12:39.1875052Z /opt/hostedtoolcache/go/1.16.6/x64/src/testing/testing.go:741 +0x65
744 2021-08-06T15:12:39.1875922Z github.com/stretchr/testify/require.Len(0x3030a68, 0xc001608900, 0x213f000, 0x0, 0x2, 0x0, 0x0, 0x0)
745 2021-08-06T15:12:39.1876822Z /home/runner/go/pkg/mod/github.com/stretchr/[email protected]/require/require.go:1036 +0x113
746 2021-08-06T15:12:39.1878146Z github.com/textileio/broker-core/cmd/auctioneerd/auctioneer_test.TestClient_RunAuction(0xc001608900)
747 2021-08-06T15:12:39.1879520Z /home/runner/work/broker-core/broker-core/cmd/auctioneerd/auctioneer/auctioneer_test.go:156 +0x69c
748 2021-08-06T15:12:39.1880337Z testing.tRunner(0xc001608900, 0x2603fb0)
749 2021-08-06T15:12:39.1881133Z /opt/hostedtoolcache/go/1.16.6/x64/src/testing/testing.go:1193 +0x203
750 2021-08-06T15:12:39.1881874Z created by testing.(*T).Run
751 2021-08-06T15:12:39.1882523Z /opt/hostedtoolcache/go/1.16.6/x64/src/testing/testing.go:1238 +0x5d8
752 2021-08-06T15:12:39.1883155Z
Occasional gpubsub failed tests
2021-07-22T16:03:51.738Z WARN gpubsub gpubsub/gpubsub.go:209 creating topic test--topic-1
2021-07-22T16:03:51.739Z WARN gpubsub gpubsub/gpubsub.go:209 creating topic test--topic-1
2021-07-22T16:03:51.747Z INFO gpubsub gpubsub/gpubsub.go:223 closing pubsub msg broker...
2021-07-22T16:03:51.747Z INFO gpubsub gpubsub/gpubsub.go:227 closing topics...
2021-07-22T16:03:51.747Z INFO gpubsub gpubsub/gpubsub.go:233 topics closed
2021-07-22T16:03:51.747Z INFO gpubsub gpubsub/gpubsub.go:235 waiting for receiving handlers to close...
2021-07-22T16:03:51.747Z INFO gpubsub gpubsub/gpubsub.go:237 all receiving handlers to closed
2021-07-22T16:03:51.747Z INFO gpubsub gpubsub/gpubsub.go:239 pubsub msg broker closed
--- FAIL: TestE2E (0.00s)
--- FAIL: TestE2E/emulator (28.42s)
gpubsub_test.go:88:
Error Trace: gpubsub_test.go:88
gpubsub_test.go:51
Error: Received unexpected error:
get topic: creating topic test--topic-1: rpc error: code = AlreadyExists desc = Topic already exists
Test: TestE2E/emulator
--- FAIL: TestTwoSubscriptions (38.41s)
gpubsub_test.go:132:
Error Trace: gpubsub_test.go:132
Error: Received unexpected error:
looking for subscription: context deadline exceeded
Test: TestTwoSubscriptions
FAIL
FAIL github.com/textileio/broker-core/msgbroker/gpubsub 38.452s
Disable pinata pinning for Testnets
Now that we have all the testnets live, we should stop paying for longterm pins for testnets.
An ideal scenario might be
- A user on testnet pushes some bytes. Those bytes get
addto our own IPFS node but not pinned - The deals could still go through to Filecoin (for now, or we start mocking this in the same work).
- Normal gc or system operations would allow that added data to get cleared out from our system
cc @carsonfarmer for thoughts here, but yeah, paying for long-term storage of testnet data seems unnecessary
Error updating database
Saw this from make up locally.
brokerd_1 | 2021-08-10T22:00:18.162Z DEBUG broker broker/broker.go:412 auction 01fcs18b087mprak1pjct3491v closed batch 01fcs189fvc5tjqbmpqawfyt2g with 1 winning bids, errorCause
postgres_1 | 2021-08-10 22:00:18.197 UTC [60] ERROR: could not serialize access due to read/write dependencies among transactions
postgres_1 | 2021-08-10 22:00:18.197 UTC [60] DETAIL: Reason code: Canceled on identification as a pivot, during write.
postgres_1 | 2021-08-10 22:00:18.197 UTC [60] HINT: The transaction might succeed if retried.
postgres_1 | 2021-08-10 22:00:18.197 UTC [60] STATEMENT: -- name: UpdateStorageRequestsStatus :exec
postgres_1 | UPDATE storage_requests
postgres_1 | SET status = $2,
postgres_1 | updated_at = CURRENT_TIMESTAMP
postgres_1 | WHERE batch_id = $1
postgres_1 |
brokerd_1 | 2021-08-10T22:00:18.199Z ERROR gpubsub gpubsub/gpubsub.go:152 calling auction-closed handler: processing closed auction: adding storage-provider deals: saving storage request: ERROR: could not serialize access due to read/write dependencies among transactions (SQLSTATE 40001)
brokerd_1 | 2021-08-10T22:00:18.283Z DEBUG broker broker/broker.go:412 auction 01fcs18b087mprak1pjct3491v closed batch 01fcs189fvc5tjqbmpqawfyt2g with 1 winning bids, errorCause
brokerd_1 | 2021-08-10T22:00:18.288Z DEBUG broker broker/broker.go:495 publishing ready to create deals for auction 01fcs18b087mprak1pjct3491v, batch%!(EXTRA broker.BatchID=01fcs189fvc5tjqbmpqawfyt2g)
batch isn't in expected status: error
Saw from local docker setup. BatchAuctioned does handle idempotency, but maybe the same message can be sent several times so with different operation ID?
brokerd_1 | 2021-08-10T22:22:22.421Z DEBUG broker broker/broker.go:507 finalized deal from auction (01fcs2gq92jyx5rzhqfec431sh, 01fcs2gqs5jcjxkqxb6980j2a6), batch 01fcs2gnbk936f7d6q3thtrvr2, storage-provider f0001
brokerd_1 | 2021-08-10T22:22:22.426Z DEBUG broker broker/broker.go:579 auction 01fcs2gq92jyx5rzhqfec431sh, batch 01fcs2gnbk936f7d6q3thtrvr2 finalized deal-id 2933568871211445515: 1/1
brokerd_1 | 2021-08-10T22:22:22.431Z DEBUG broker broker/broker.go:586 batch 01fcs2gnbk936f7d6q3thtrvr2 success
brokerd_1 | 2021-08-10T22:22:22.603Z DEBUG broker broker/broker.go:507 finalized deal from auction (01fcs2gqdyaenmd2g0y69fxdgc, 01fcs2gqtqtbsvdw99hc75hdr2), batch 01fcs2gng9w2hc1tymf5kp4da6, storage-provider f0001
brokerd_1 | 2021-08-10T22:22:22.608Z DEBUG broker broker/broker.go:579 auction 01fcs2gqdyaenmd2g0y69fxdgc, batch 01fcs2gng9w2hc1tymf5kp4da6 finalized deal-id 2610529275472644968: 1/1
brokerd_1 | 2021-08-10T22:22:22.612Z DEBUG broker broker/broker.go:586 batch 01fcs2gng9w2hc1tymf5kp4da6 success
brokerd_1 | 2021-08-10T22:22:22.927Z DEBUG broker broker/broker.go:412 auction 01fcs2a16bhvq903q8dhsx54fv closed batch 01fcs29zmjyh03bg5vj8q82mx1 with 1 winning bids, errorCause
brokerd_1 | 2021-08-10T22:22:22.931Z ERROR gpubsub gpubsub/gpubsub.go:152 calling auction-closed handler: processing closed auction: batch isn't in expected status: error
...
brokerd_1 | 2021-08-11T01:24:36.317Z DEBUG broker broker/broker.go:412 auction 01fcs2a16bhvq903q8dhsx54fv closed batch 01fcs29zmjyh03bg5vj8q82mx1 with 1 winning bids, errorCause
brokerd_1 | 2021-08-11T01:24:36.348Z ERROR gpubsub gpubsub/gpubsub.go:152 calling auction-closed handler: processing closed auction: batch isn't in expected status: error
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.