api3dao / airnode Goto Github PK
View Code? Open in Web Editor NEWAirnode monorepo
Home Page: https://docs.api3.org/
License: MIT License
Airnode monorepo
Home Page: https://docs.api3.org/
License: MIT License
More contracts using the feed = Less secure
We only want the contracts that have paid for the feed to be able to use it. See:
https://github.com/smartcontractkit/chainlink/blob/46d4a77faae2904b6b15b747edd5ba2955337034/evm-contracts/src/v0.6/dev/AccessControlledAggregator.sol#L40
This tends to be a lower than "fast" gas price, but it may be a better idea than having a constant minimum value of 40 gwei
https://docs.ethers.io/v4/api-providers.html?highlight=gasprice#blockchain-status
There used to be a method to error a call
airnode/packages/protocol/contracts/interfaces/IAirnode.sol
Lines 119 to 129 in fd1fdf2
errorCode
is returned in statusCode
of fulfill
:airnode/packages/protocol/contracts/interfaces/IAirnode.sol
Lines 93 to 105 in 696ebad
All code about erroring needs to be removed. The node should try to call fulfill()
and if that reverts, it should call fail()
directly.
A requester no longer needs to make a request to have a wallet designated and the node doesn't need to fulfill these.
All code related to wallet designations should be removed.
Just use the requesterId
as the walletInd
Errored
(wrong)
ClientRequestCreated
sounds good. The other names would go likeClientShortRequestCreated
etc.?and how about
ClientRequestFulfilled
/ClientRequestErrored
/ClientRequestFailed
?
Leaving this here as a to do
#5 (comment)
A user should be able to configure Ethereum providers. This will allow the node to be more resilient if a single provider is down and will also better protection if a node starts reporting bad (or even malicious) data. e.g. The current block number is 99999999.
For each configured Ethereum provider, we need to find all of the relevant details at the start of the serverless invocation. This should be done in parallel. We'll need the following information for each provider:
And possibly a few other things. This information should be store against each provider in the state.
For unique request, the node should make a single API call. Each of these calls needs to happen in separate serverless functions. The response value should also be extracted in the separate function invocation using the _path
parameter. API responses can be very large and we could easily hit the memory limit if multiple large responses get returned to the main function.
For each configured Ethereum provider and each response, we make transaction. This means duplicate transactions. I don't think this step needs to be in separate serverless functions, but that is a possibility.
This script seems to have been removed
airnode/packages/node/README.md
Lines 35 to 36 in fd1fdf2
Request IDs are derived from request parameters, for example:
airnode/packages/protocol/contracts/Airnode.sol
Lines 51 to 55 in 696ebad
Here, we only need to test for requestId
:
The node should attempt to connect to multiple providers on function run in case one or more are down. It should then choose between responding nodes and stick with that provider for the duration of the function.
Maybe it should check for the latest block among responding nodes to determine which provider to use. Not sure what should happen if multiple respond with the same block. Maybe the user should be allowed to rank them in terms of preference?
i.e.
ETHEREUM_PROVIDER_URL=xxx
ETHEREUM_BACKUP_PROVIDER_URL=xxx
ETHEREUM_BACKUP_PROVIDER_2_URL=xxx
(There are probably better names we could use)
Implement something like the below RetryProvider. I'm not sure that it's exported anywhere in ethers
https://github.com/ethers-io/ethers.js/blob/master/packages/experimental/src.ts/retry-provider.ts
Instead of getting the block number while initalizing the providers, the node should call Convenience to get the block number + provider data:
airnode/packages/protocol/contracts/interfaces/IConvenience.sol
Lines 9 to 16 in 696ebad
providerId
off-chain as keccak256(address of path 'm')
. If this method returns xpub
as ''
, it means that the record for this provider hasn't been created on this chain yet. Then, the node should call this:airnode/packages/protocol/contracts/interfaces/IProviderStore.sol
Lines 36 to 42 in 696ebad
m
, where admin is
from config.json
and xpub
is derived from the private key.
Note that this method is payable
. This is because the provider will have to send some ETH to the wallet with path m
for the node to be able to do this. While doing this, the node will send the remaining funds to admin
.
All requests pass designatedWallet
and requesterInd
.
When the node receives a request, it should verify that designatedWallet
is the address of m/0/${requesterInd}
and ignore the request if this is not the case.
I realized that I made a mistake with the OIS. The security
field should be a SecurityRequirement
, not a SecurityRequirement[]
. See:
https://github.com/clc-group/airnode/wiki/OIS-v1.0.0#43-security
Currently, we require a transaction to be made for an endpoint to be created on-chain, essentially to only create an endpoint ID. Instead, we can use a providerId
-endpointInd
pair to refer to endpoints. This would also be preferable as we would have identical endpointInd
s across chains (which can be thought together with mirroring the providerId
across chains, see #55 ).
The authorization check shouldn't call the contract if authorizers
is []
(return false
automatically) or [ethers.constants.AddressZero]
(return true
automatically)
We want to provide a set of generic Authorizer contracts for the providers so that they can combine them to implement custom authorization policies.
The tests commented out here spend inconsistent amounts of gas
f0ac82c#diff-55b6470a81b9afa68182a49553eedb12
Resetting ganache as below doesn't help
beforeEach(async () => {
jest.resetModules();
const ganache = require('ganache-core');
...
These tests are not important in particular, but ganache not resetting properly between tests is concerning (assuming the inconsistency is caused by the tests running in changing order).
Request events used to be
airnode/packages/protocol/contracts/interfaces/IAirnode.sol
Lines 9 to 19 in fd1fdf2
airnode/packages/protocol/contracts/interfaces/IAirnode.sol
Lines 21 to 27 in fd1fdf2
airnode/packages/protocol/contracts/interfaces/IAirnode.sol
Lines 29 to 39 in fd1fdf2
airnode/packages/protocol/contracts/interfaces/IAirnode.sol
Lines 9 to 20 in 696ebad
airnode/packages/protocol/contracts/interfaces/IAirnode.sol
Lines 22 to 29 in 696ebad
airnode/packages/protocol/contracts/interfaces/IAirnode.sol
Lines 31 to 42 in 696ebad
Using .solhintignore
to ignore node_modules
doesn't work
https://github.com/clc-group/airnode/blob/f7fd762dba2e1bd79459c140e1edbc3f827d10e9/package.json#L7
It should be a two-step procedure where the new owner accepts the transfer. See
https://github.com/smartcontractkit/chainlink/blob/46d4a77faae2904b6b15b747edd5ba2955337034/evm-contracts/src/v0.6/dev/FluxAggregator.sol#L544
https://github.com/clc-group/airnode/blob/d5ceae3936da859eff69f80dc5d7e13412e6e1a0/packages/contracts/package.json#L5
prestart
doesn't refresh the ganache-cli instance if one already exists, but keeps the old one around. It would be ideal to refresh it to avoid persisting effects between runs.
One can create a template with a very large parameters
field, which would cause the Convenience
method to revert due to the exceeding the gas limit, resulting in not being able to retrieve other 9 templates. I'll investigate further.
convenience.getTemplates
(#184)convenience.getAuthorizationStatuses
During tests, ganache-core is giving this error message with no apparent problem
(node:1467) V8: /home/burak/git/airnode/packages/contracts/node_modules/ganache-core/node_modules/rustbn.js/lib/index.asm.js:2 Linking failure in asm.js: Unexpected stdlib member
It should return early
The contracts should derive request IDs (both API and withdrawal) from request-time parameters. For example the short request ID is derived as
requestId = keccak256(abi.encodePacked(
noRequests,
templateId,
parameters
));
and emits
emit ShortRequestMade(
providerId,
requestId,
msg.sender,
noRequests
templateId,
parameters
);
Then, the node can check if templateId
or parameters
is tampered with by the Ethereum provider by checking if their hash matches the requestId
.
The node should do the same thing with templates, i.e., when it fetches a template, it should calculate its hash and check if it matches the template ID.
This should also be done for withdrawal requests
There are two withdrawals:
In both cases, we need the node to send balance - txCost
to the destination. Here, we should estimate the gas cost to estimate the transaction cost
airnode/packages/protocol/test/UserFlow.js
Lines 131 to 147 in 696ebad
airnode/packages/protocol/test/UserFlow.js
Lines 297 to 321 in 696ebad
This (i.e. not hardcoding the gas cost) good for three reasons:
payable
contract instead of a regular wallet, in which case the gas cost would be variableand withdrawals won't be common, so the additional Ethereum call is not significant.
Fulfillment and failure arguments used to be
airnode/packages/protocol/contracts/interfaces/IAirnode.sol
Lines 95 to 105 in fd1fdf2
airnode/packages/protocol/contracts/interfaces/IAirnode.sol
Lines 131 to 132 in fd1fdf2
airnode/packages/protocol/contracts/interfaces/IAirnode.sol
Lines 93 to 105 in 696ebad
airnode/packages/protocol/contracts/interfaces/IAirnode.sol
Lines 121 to 127 in 696ebad
fail()
still doesn't call back fulfillAddress
/fulfillFunctionId
, it just needs these arguments to check designatedWallet
)
This is done for the fulfillment parameters to be checked on-chain. In the previous state, the provider was able to fulfill an existing request using providerId
, designatedWallet
, fulfillAddress
and fulfillFunctionId
that are different than the ones defined by the requester.
Calls to these methods need to be updated.
The node should not be able to use any fulfillment parameter (fulfillAddress
etc.) they want for a specific request. Their hash should be recorded at request time and checked for during fulfillment.
This would require both fulfill
and error
to be called with fullfillment and error parameters.
Authorization checks used to be for endpoint-client pairs:
airnode/packages/protocol/contracts/interfaces/IConvenience.sol
Lines 48 to 54 in fd1fdf2
Now they are request specific and have a bunch of other arguments (providerId
is not an array!):
airnode/packages/protocol/contracts/interfaces/IConvenience.sol
Lines 43 to 53 in 696ebad
Also note that the non-batch version of this method is also in Convenience (will be used as fallback if the batch version reverts):
airnode/packages/protocol/contracts/interfaces/IConvenience.sol
Lines 31 to 41 in 696ebad
Request IDs are derived from request parameters. The derivation changes with type:
Regular:
airnode/packages/protocol/contracts/Airnode.sol
Lines 51 to 55 in 696ebad
airnode/packages/protocol/contracts/Airnode.sol
Lines 100 to 104 in 696ebad
airnode/packages/protocol/contracts/Airnode.sol
Lines 156 to 161 in 696ebad
When the node receives a request, it should verify this derivation to ensure that the Ethereum provider didn't tamper with any of the request parameters.
Template ID is derived from its contents:
airnode/packages/protocol/contracts/TemplateStore.sol
Lines 70 to 78 in 696ebad
When the node receives a template, it should verify this to ensure that the Ethereum provider didn't tamper with any of the template fields.
If a requester reserves makes a transaction to reserve a wallet and the node does not see that event, the requester loses the authorizationDeposit
and also gets locked out of being able to have a wallet reserved. The requester should be able to rebroadcast the reservation event without making any deposit for the node to be able to see and handle it.
Note that the node shouldn't serve duplicate events separately.
Assuming the provider will use a single private key for all chains, we can derive the providerId
from that so that they would have identical providerId
s in all chains.
Right now, the client contracts announce their (potential) endorser under the public variable requesterId
. Wallet addresses can't do the same, so they can't be endorsed. If these are kept it RequesterStore
, wallet addresses can also be endorsed and make requests that can be fulfilled with a reserved wallet.
Having a gas price data feed on Ropsten (https://github.com/clc-group/airnode/blob/master/packages/node/src/core/ethereum/contracts/gas-price-feed.ts#L6) doesn't make sense because apparently miners don't care about gas prices while mining transactions. Similarly, people on permissioned chains etc. won't be running gas price feeds. Therefore, we can simplify how that works at the node.
The easiest solution I can think of is that the node assumes it's on mainnet, calls the data feed to get the gas price and only use it if it makes sense. This is safe because:
1 - You can't try to deploy a contract on Ropsten to have the exact same address as the mainnet gas price feed address
2 - Even if you did, it's Ropsten
I'll do some research on how this would work and report back.
Currently, withdrawal requests are prioritized and cause other requests made to the respective designated wallet to be ignored.
A malicious provider can't make an Airnode make a withdrawal by faking an event because the fulfillment transaction checks if such a withdrawal request exists on-chain
https://github.com/clc-group/airnode/blob/c1250cbc94f7c3e7c36ffa05edacede0d10eb699/packages/protocol/contracts/ProviderStore.sol#L344
so even if the node attempts to fulfill the faked withdrawal request, the fulfillment will revert.
However, with this scheme, the fake withdrawal event can still be used to put a specific designated wallet of an Airnode out of operation for 1 hour because it will ignore all other requests.
Therefore, withdrawal requests should not cause other requests to be ignored, and should be sent to the back of the nonce-queue.
No need to validate that walletInd
is not 0
(we don't have walletInd
anymore in the first place) or that its balance is more than minBalance
.
All code related to walllet checks need to be removed.
Request fulfillment events used to be
airnode/packages/protocol/contracts/interfaces/IAirnode.sol
Lines 41 to 45 in fd1fdf2
airnode/packages/protocol/contracts/interfaces/IAirnode.sol
Lines 47 to 51 in fd1fdf2
airnode/packages/protocol/contracts/interfaces/IAirnode.sol
Lines 53 to 57 in fd1fdf2
airnode/packages/protocol/contracts/interfaces/IAirnode.sol
Lines 59 to 62 in fd1fdf2
airnode/packages/protocol/contracts/interfaces/IAirnode.sol
Lines 44 to 49 in 696ebad
airnode/packages/protocol/contracts/interfaces/IAirnode.sol
Lines 51 to 56 in 696ebad
airnode/packages/protocol/contracts/interfaces/IAirnode.sol
Lines 58 to 61 in 696ebad
The ABI needs to be updated
Template format used to be
airnode/packages/protocol/contracts/TemplateStore.sol
Lines 13 to 21 in fd1fdf2
Now it is
airnode/packages/protocol/contracts/TemplateStore.sol
Lines 23 to 31 in 696ebad
getTemplates()
from Convenience
changed accordingly
airnode/packages/protocol/contracts/interfaces/IConvenience.sol
Lines 18 to 29 in 696ebad
Types and contract calls need to be updated.
The node used to get requester data from Convenience:
https://github.com/api3dao/airnode/blob/566fc7b72e1b22486cdc6e87d1e86c27d93ecacd/packages/node/src/core/evm/requests/requester-data.ts
It got requesterId
, walletAddress
, walletInd
, walletBalance
, minBalance
.
Now, all requests include requesterInd
and designatedWallet
, there is no separate walletInd
, and the minimum balance check is done at an Authorizer.
All code related to fetching requester data should be removed.
API calls may or may not be idempotent so the API provider might not want to make duplicate API calls between serverless function invocations while the transaction is pending. The API response needs to be optionally cached.
The cache should store the following information against the requestId
:
_path
or the errorCache keys need to have an expiration mechanism too.
Before making an API call, the cache should be checked for the requestId
. If a value is found, skip making that call and attempt another transaction with that value.
API calls should default to cached if a cache is configured. Otherwise, a new call should be made each serverless invocation.
requestid
exists in the cache, but no unfulfilled request is found?When idempotence is important, a client request should only be attempted to be fulfilled when the node is sure that the request exists, which requires a configurable parameter for how many blocks it should wait for. I think it's perfectly fine to not have this for now.
Tracked by https://api3dao.atlassian.net/browse/AN-160
The normal mode of operation doesn't give any guarantees for not calling the API more than once for a single request:
https://api3dao.github.io/api3-docs/pre-alpha/airnode/implementation.html#non-idempotent-operations
The normal mode of operation also doesn't give any guarantees about being absolutely sure that an incoming request is real (and not spoofed for example).
We need to have a mode and guidelines for when the user wants to use non-idempotent operations, and thus only want to call the API when there is an actual request, and make that call only once.
and not sure if it's worth it
The initial plan was to use getGasPrice()
and also get the gas price from a feed, then use the maximum. The assumption was that getGasPrice()
may (will) be below what we want to use, so we can use the gas price feed to ramp up the gas prices temporarily.
This is both not a very elegant solution, and with the increased focus on serving non-mainnet chains, we should move towards a more chain-agnostic solution. My proposed solution is to only depend on getGasPrice()
, but use a dead reckoning approach to bump gas prices as needed.
The main idea is to use k * getGasPrice()
as the gas price, where k > 1
and increases with currentBlock - requestBlock
. The exact coefficient (for example, what is k
if the request is 20
blocks old and still not fulfilled?) will be determined based on off-chain statistical analysis of gas prices. This coefficient will be a lot more static than the gas price itself (so it doesn't matter if the gas price is 50 gwei or 500 gwei, k
would be 1.5
if the request is 20
blocks old). Then we can hardcode this into config.json
. There's also the possibility of keeping this coefficient on-chain (update every week for example), possibly by the API3 DAO or one of its teams.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.