web3-storage / dealer Goto Github PK

We need to get the following information from Spade state

• poll state of an aggregate (did a previous offered aggregate landed into a SP? did it fail?)
• get details from the deals that exist for each aggregate previously submitted

Schema can be seen https://raw.githubusercontent.com/ribasushi/fil-naive-marketwatch/master/misc/pg_schema_diagram.svg

Open questions:

• is it a PostgresDB? can we have credentials for a replica RO?
• Is there any roadmap to expose this as an API rather than requiring all clients to connect to DB and do queries themselves?
• Are there docs about queries we should make?
  • for instance, there is pieceCid in the schema above, but how can we query for an aggregate commD and see its state?

Context #2 (comment)

Details are in https://www.notion.so/w3-spade-deal-flow-bd479485bbf64e85aa0dee0ad6fc3019

We previously aligned that for Spade to Polling spade-proxy on aggregates to pass to SPs, Spade would poll from a S3 bucket where we would write the details for an aggregate. This polling will happen on an agreed timeframe (TBD - @ribasushi is there any recommendation from Spade or based on our previous usage of dagcargo?).

During the given timeframe for polling we can have more than one aggregates ready to go. Therefore, we need to make sure that all of them are ready to be polled. S3 client sadly does not support append to a already existing file, which would make our lives much easier. We are currently writing temporary files keyed as YYYY-MM-DD HH:MM:00 commPcid and have a PR with a cron job that kicks in and merges all these files on a single file keyed as YYYY-MM-DD HH:MM:00 that can be polled. This solution has some potential issues:

• If we kick the cron before the timeframe is over another aggregate can be in the process of being written.
• Spade would need to poll with some delay so that thing is already there

Alternatively, we could allow Spade to perform a Bucket list with prefix YYYY-MM-DD HH:MM:00 once time is ready and then perform a get in each key. This makes Spade job a tiny bit more complex on read, but is less susceptible to weirdness with times. This could also happen via an API call by Spade, where spade-proxy exposes an endpoint and does the merge of available aggregates.

For the content format, we are looking at a JSON file with format and I will create a separate issue to discuss that

Thoughts @ribasushi @alanshaw @xinaxu

Context: #15 (comment)

I think we should embrace the composed principals exposed by ucanto instead of assuming which key types are at play. Specifically we have SignerImporter interface that import Signer from '@ucanto/principal' implements.

That is to say I think it would make a lot more sense if we wrote JSON.serialize(signer.toArchive()) into secrets and then read loaded it with Signer.from(JSON.parse(secret)).

Also note that this would remove need for signer.withDID(did) dance as long as what we stored in secrets is ed25519.parse('...').withDID(did)

P.S. Only reason toArchive does not return bytes or a string today is because in browsers we won't be able to serialize non-extractible keys, yet we could store object containing them into indexedDB.

An aggregate/offer can fail when SPs try to accept an aggregate. It can be for a few reasons, like when a commP for a CAR file that is part of the aggregate is wrong.

spade-proxy should issue a receipt for refused aggregate with the details of what failed to be added. This will enable web3.storage actor to create a new aggregate without the "bad content" and flag the bad content.

There is still open question on how we can get such details in spade-proxy:

• Can this info be queried from Spade Oracle?
  • Or do we have to expose an API for getting such details?
• Are there any docs on types of failures and how they look like?

We currently default to a random value in queue client