emschwartz / ilp3

An implementation of Interledger V3

ILPv3 - A new take on Interledger

This is a prototype. Nothing is finished, everything could change and feedback is welcome.

Overview

ILPv3 is a different take on Interledger that aims to simplify the protocol stack and implementation even further and get the open Interledger started faster. It begins with the premise that all payments will be small and larger ones will be sent as Chunked Payments. Building for small payments makes connectors simpler, lowers their risk, and should make the system more competitive.

ILPv3 also adopts a number of other simplifications, including making quoting an end-to-end concern and making a standard ILP packet encoding optional (i.e. no OER if you don't want it). The ledger layer protocol used here is just a couple of ILP headers (ILP-Destination, ILP-Condition, ILP-Expiry) attached to a standard HTTP request (the ILP packet data is the HTTP body).

This implementation uses a "middleware function"-based architecture, inspired by Koa.js, instead of the Ledger Plugin design. Requests are passed through a stack of functions that handle behavior such as parsing ILP details, checking a user's balance, and sending outgoing transfers using specific ledger protocols. Middleware functions can be composed into senders, connectors, and receivers (see example.js), enabling greater code reuse and easier extensibility.

Trying It Out

Clone the repo, npm install, and run DEBUG=* node example.js to see end-to-end quoting and chunked payments in action over XRP Payment Channels.

Building for Small Payments

"Streaming payments change everything." - @justmoon

Building an Interledger implementation designed exclusively for small payments enables us to make a number of important simplifications to the protocols and stack. This not only makes Interledger fit better with our target use case of micropayments, but there are reasons to believe that all payments may become micropayments (if you're saying "whaat??", read on).

Streaming Payments are payments sent in many little increments in exchange for an ongoing or streaming service, such as paying for a movie 1 Mb at a time. We used to use this to also refer to what we now call Chunked Payments.

Chunked Payments are larger payments split into smaller chunks for sending over the Interledger. Now why would you want to do that?

Advantages of Chunked Payments

1. Accommodating Low Maximum Payment Sizes

Every payment path through the Interledger will have some Maximum Payment Size (MPS, like the Internet's Maximum Transmission Unit or how large packets can be). The average MPS is likely to be low because of connector risk and liquidity factors. Chunked payments will be required if senders ever want to send larger amounts than the path MPS.

2. Enabling Smaller Connectors

There are simply more parties in the world that could facilitate a $0.10 payment than a $10,000 payment. Splitting larger payments into many small ones allows connectors with less liquidity to participate in the Interledger.

3. Increasing Competition

This is the main reason we expect Interledger payments to be significantly cheaper than traditional payments in the medium to long term. The more connectors can compete to facilitate each payment, the cheaper they should be.

4. Reducing Connector Risk

One of the main risks for Interledger connectors is failing to fulfill an incoming payment in time, and thus losing the principal of a payment. The smaller each payment, the lower the risk. If connectors are sending many tiny payments and some of them start failing, mitigation strategies can be used to avoid losing more money. Smaller payments also enable shorter timeouts (because the downside of payments failing is lower) and reduce the free option problem that comes with conditional payments.

5. Works with Simple Payment Channels

Interledger can be used with any ledger using different types of Hashed Timelock Agreements. Simple (unconditional) payment channels are arguably a good balance between ledger requirement complexity (XRP, Ethereum, and Bitcoin, even without SegWit, support these), speed, and cost. If all payments are small, the risk to connectors posed by peers running off before sending a payment channel update can be reduced to an acceptable amount (note that senders still do not need to trust connectors).

6. Protocol Simplicity

If we can assume all payments are small, we can simplify the Interledger protocol stack further. For example, liquidity curves are only necessary if payments vary greatly in size. A simple exchange rate suffices for small payments.

7. Hiding Test Payments

If all payments across the Interledger are small, it should be harder for connectors to identify test payments sent by other connectors to probe routes. This should enable connectors to keep better statistics on the real rates and connectivity provided by their peers.

Disadvantages of Chunked Payments

Note: These are potentially serious issues, but it is important to note that they are inevitable if anyone ever wants to send larger payments than the payment path can support. One of the main questions we have to answer now is whether we expect the average Maximum Payment Size to be on the order of $1, $10, $100, $1000 or more. The higher this number, the greater the risk and liquidity requirements will be for connectors, which in turn limits the pool of potential connectors.

1. Partial Payments

If the whole payment is no longer delivered atomically, there is a possibility that the path could run out of liquidity mid-payment and the receiver would end up with only some of the money they requested. In most cases, the sender should be able to keep retrying chunks until the whole payment is delivered. However, if the path completely runs out of liquidity (which should happen extremely rarely), receivers might need to send back payments that cannot be completed.

2. Fluctuating Exchange Rates

The rate for a payment can change due to legitimate or illegitimate reasons after the first chunk is sent but before the last one is received. If the exchange rate changes dramatically and unpredictably over the course of a payment, it would make for a bad sender experience.

Differences from ILPv1

Interledger Layer

1. Forwarding Only

Connectors only "forward" payments, applying their own rate to the incoming transfer to get the outgoing transfer amount. There is no "delivery" so connectors do not need to know the exact and up-to-date exchange rates of all other connectors.

2. No Prefix Restrictions on ILP Addresses

In order to support the "delivery" feature, connectors needed to know whether they were the last hop in a payment path and thus whether an ILP address was "local". This required restrictions on ILP addresses (not being able to use a ledger's address in another ledger's prefix unless the exchange rate is 1:1) that are unnecessary in a forwarding-only system.

3. No Destination Amount in the ILP Packet

The amount field in the ILP packet was intended for transport layer protocols and for connectors to be able to deliver the exact destination amount. Since there is no delivery, there is no need for a destination amount field to be part of the standard ILP packet understood by every connector. Transport layer protocols can include the amount in the data field if they so choose.

4. End-to-End Quoting

The requirements for quoting are inextricably linked to the transport protocol being used. For example, a static, non-binding ILQP quote is not very helpful for a streaming or chunked payment because the rate may change over time. ILPv3 makes quoting functionality part of the transport protocol and removes the need for an Interledger Quoting Protocol that must be understood by all connectors. Instead, senders and receivers use test payments (which can be fulfilled or rejected depending on the use case) to determine how much money arrives when a certain source amount is sent. (I believe @justmoon came up with the idea of using test payments for quoting)

5. No Standard Packet Encoding

The main reasons for having a canonical ILP packet format were a) to have a consistent encoding for ILP and other Interledger layer protocols such as ILQP b) to distinguish the Interledger layer details from the ledger layer details and c) for transport layer protocols to be able to hash the packet into the condition as IPR and PSK do. Point a) is no longer applicable because there is only ILP, there are no other protocols on the Interledger layer. After the "Interledger Enlightenment", b) is no longer necessary because we make less of a distinction between ledgers and connectors, and connectors need to see the destination address alongside the incoming transfer amount. Finally, transport layer protocols can hash the data by itself instead of the "packet" as a whole. Given all of these, having a standard encoding becomes only a nice-to-have that can prevent the few unchanging fields in an ILP payment (destination address, data, and condition) from being decoded and reencoded at every hop. However, this is not strictly necessary and does not need to be standardized up-front. (@adrianhopebailie was the one that first took issue with the distinction between the Interledger and ledger layers)

6. Simple Exchange Rates Instead of Liquidity Curves

Liquidity curves were necessary to express how exchange rates varied with payment size. If all payments are assumed to be small, this complex feature can be replaced by a single number representing the exchange rate.

7. (Potentially) Higher Data Limit

In this implementation, the ILP data is simply the body of an HTTP request, which connectors can stream from the incoming request to the outgoing request, rather than buffering it all into memory. If that becomes standard practice, connectors could allow larger amounts of data to travel with ILP payments, because the impact on the connector would be minimal.

Ledger Layer

1. Fulfillments and Errors are Responses, Not Requests

In most cases, there is nothing a connector can do if they try to pass back a fulfillment or an error and it is not accepted.

2. Transfers Only

Since there is no ILQP, there is no need for the ledger layer protocol to do messaging or anything other than sending transfers. Other protocols such as those related to routing can be built either on top of TCP/IP or Interledger payments.

3. Recommend Using HTTP

Since the ledger layer protocol only needs to handle a single request/response call with a couple of structured fields and some opaque data, HTTP is a perfect protocol for this. The ILP-related fields can be sent as headers and the ILP data can be sent as the HTTP body. Nearly all programming languages support HTTP and no additional encoding library would be needed to implement ILP. HTTP Keepalive or HTTP2 can be used to avoid the extra round trips for TCP and TLS when multiple payments are being sent between the same two peers. Implementations may still abstract away the communication protocol used to enable using alternatives such as RPC over Websockets.

4. Fast HTLAs Only

Interledger can theoretically support a wide variety of ledger integrations (see Hashed Timelock Agreements (HTLAs)), but today, most ledgers available are too slow or expensive for on-ledger escrow to provide a good experience. Payment channels and trustlines should be the only recommended HTLA types for now and other protocols should be built to assume that transfers can be executed in milliseconds, as opposed to seconds or longer. This recommendation would change once ledgers are capable of processing large volumes of payments with negligible costs and latency.

Transport Layer

1. Only Hash Data

Unlike PSK 1.0, this implementation of PSK only hashes the ILP data, rather than the whole ILP packet.

2. Include End-to-End Quoting and Chunked Payments

In addition to encrypting the end-to-end data and generating the fulfillment and condition, this version of PSK also handles end-to-end quoting and chunked payments. This allows all details associated with these use cases to be encrypted within the PSK data, and allows senders to assume that any receiver that supports PSK will support end-to-end quoting and chunked payments. @sentientwaffle raised the question of whether E2E quoting and chunked payments should be part of PSK or implemented as a separate "layer" built on top of it (so PSK would only handle encryption and condition generation), and this should be discussed further.

3. Out of Band Cipher Negotiation

PSK 1.0 includes the cipher suite in the "public headers" outside of the encrypted data. Instead of putting plaintext data into the ILP data, the cipher negotiation should be part of the PSK details exchanged between the sender and receiver (which currently consist of the receiver's address and the shared secret). This version of PSK assumes the cipher is AES-256-GCM.

4. No Set Destination Amount

Currently, this implementation of PSK does not include a destination amount in the encrypted data. The receiver fulfills every incoming chunk they see and they use the fulfillment data to communicate back to the sender how much arrived (this is encrypted and authenticated). It is then up to the sender whether they want to continue sending more chunks, the chunk size to use, and whether they should switch to a different connector. @michielbdejong has raised a number of potential issues with this approach, which should be explored in greater depth.

One alternative would be for the sender to use the first payment chunk as a kind of quote and then to inform the receiver how much to expect on each successive chunk. This would provide the same properties as requesting a quote using ILQP and then sending multiple payments using ILPv1. However, connectors could still play with their rates on the first chunk.

Implementation

1. Middleware Instead of Plugins

The idea of Ledger Plugins was to enable the same ILP client and connector software to be used with different ledgers. However, we found that we needed additional ways to reuse code across plugins (see the Payment Channel Plugin Framework), which suggests that plugins are not the best way to structure the code internally. Inspired by Koa.js and levelup, this implementation separates tasks into a stack of middleware functions that handle various checks or transformations and then pass control to the next function. This design makes it easier to add new functionality into the flow, such as a balance checker that uses a specific database, a transfer logger, or a new ledger layer protocol. See the Middleware API below for more details.

2. Single Process Per Account

Individual account balances must be kept consistent, so those are likely to always be a performance bottleneck. As pioneered by @justmoon in the ilp-connector-shard, this implementation assumes that each account will be managed by a single process. This allows the balance to be cached and updated in memory and enables using fast, single-process databases such as LevelDB or RocksDB.

3. Don't Persist Prepared Transfers

If an account is managed by a single process and transfer timeouts are short, the balance change from a prepared transfer can be kept in memory instead of on disk. If the connector server crashes before the transfer was finalized, it is unlikely that the server will come back online in time for the transfer to be fulfilled, so there is little point in persisting the prepared transfer. This makes the speed of preparing a payment end-to-end equal to the network latency plus a small number of in-memory operations, which should be extremely fast. Even when transfers are fulfilled, the balance changes can be persisted after the fulfillment is passed on.

4. Stream ILP Data

ILP data is end-to-end, so connectors can stream the data from the incoming request to the outgoing request without buffering it into memory. This should enable connectors to have higher data limits with less impact on their server performance.

Middleware API

ILP3 middleware functions use the following properties on the context (ctx) object:

• ctx.incoming.transfer
• ctx.incoming.account
• ctx.outgoing.transfer
• ctx.outgoing.account

Transfer Schema

Property	Type	Required?	Description
amount	(Positive) Integer String	Y	Transfer amount, denominated in the ledger's minimum units
destination	ILP Address	Y	Destination address the payment is for
condition	Buffer or Base64 String	Y	Hashlock condition used to secure the transfer
expiry	ISO 8601 Timestamp	Y	Expiration date for the transfer
data	Buffer or Readable Stream	N	End-to-end data
to	ILP Address	N	Local account the transfer is for
from	ILP Address	N	Local account the transfer is from
extensions	Object	N	Additional key-value pairs attached to the transfer (for example, payment channel claims)

Account Schema

Property	Type	Required?	Description
uri	URI	N	URI used to communicate with this account-holder
currencyCode	String	N	Currency code (such as "USD") the account is denominated in
currencyScale	Number	N	Integer (..., -2, -1, 0, 1, 2, ...), such that one of the ledger's base units equals 10^-<currencyScale> <currencyCode>
minBalance	Integer String	N	The minimum balance the account-holder is allowed to have
adjustBalance	Integer String	N	Used to instruct a balance-tracking middleware to adjust the account's balance by the given amount (for example, '1000' would mean the account balance should be credited 1000 and '-1000' would mean the account balance should be debited 1000)

TODOs

• Connector exchange rates
• Connector streams data from incoming to outgoing request
• Send authorization in HTTP header
• Sender automatically caveats macaroon token
• Unified middleware API for senders, receivers, and connectors
• Connector keeps balances for multiple senders (and adjusts balance on incoming and outgoing transfers)
• Connector uses ILP addresses to determine where transfers are going to / coming from
• Quoting
• Chunked payments
• XRP payment channel claim support
• Sender submits claims after receiving fulfillment
• Save payment channel claims to disk
• Standalone XRP payment channel claim submitter
• Store balances in DB
• Connector dynamically adjusts users' minimum balance
• Use a single db for all of the connector middleware
• Receiver can request that the connector create a payment channel to them
• Connector stores routes in db
• Connector uses IP-based rate limiting and tracks accounts that don't submit claims
• Auto-connect to connectors and save config (env file or db?)
• Bitcoin payment channel support
• Ethereum payment channel support (ideally including ERC 20 tokens)
• Configurable congestion avoidance algorithm for chunked payments
• Separate chunked payments from PSK
• User data on chunked payments
• Error handler that produces machine-readable error objects
• Compatibility API (that mimicks the ilp module for ILPv1)
• Bundle recommended set of middleware for senders, receivers, connectors
• Auto-fund payment channel when balance is too low
• Data collection
• Use normal Passportjs for auth instead of Macaroons

ilp3's Issues

Last line of example transcript

  ilp3-psk:receiver got last chunk of amount 5 for payment: a88f7f6a59fe3003be9656aeab775569. total received: 500 +19ms
receiver got payment for 500
  ilp3-http:parser responding to sender with fulfillment +1ms
  ilp3-http:client got fulfillment: V3dGwN4PEi4RiPRW9KEFJ7Alg05YN0BHvO68gclGXZQ= and data: [Stream] +4ms
  ilp3-fulfillments:validator fulfillment matches original transfer condition +17ms
  ilp3-balance-tracker crediting account: test3.crypto.xrp.pc.rw3PbBm3HJGXtJUxstWWDtu1i3U7ss9T2T.rnpQy9QfeYMmXudBQfUfRHTcBiKYyNaSEM by: 5, balance is now: 1490 +4ms
  ilp3-connector responding to the sender with the fulfillment: V3dGwN4PEi4RiPRW9KEFJ7Alg05YN0BHvO68gclGXZQ= +4ms
  ilp3-bandwidth-adjuster lowering minBalance to -2513 for account test3.crypto.xrp.pc.rw3PbBm3HJGXtJUxstWWDtu1i3U7ss9T2T.rMRyYByxGS48tfu5Qvy9n9G7mqQT6HvKcg +18ms
  ilp3-http:parser responding to sender with fulfillment +9ms
  ilp3-http:client got fulfillment: V3dGwN4PEi4RiPRW9KEFJ7Alg05YN0BHvO68gclGXZQ= and data: XKTnFZ3uNKUYaOl9QQbpHCHx+RizgxJvzClpRZMBZKjmNT5C9WTGA6C0 +13ms
  ilp3-fulfillments:validator fulfillment matches original transfer condition +2ms
  ilp3-xrp:outgoing transfer succeeded, increasing channel amount to 1506 +19ms
  ilp3-psk:sendChunkedPayment transfer was successful, increasing chunk size to: 7 +18ms
sender delivered 500 by sending 506

That sounds like those 6 were used up as commission, I think it would be clearer to also specify the channel balances at the end, to show they add up?

[XRP] Consider renaming channel.amount to channel.balance

The amount secured by the best claim is stored in channel.amount. However, in the context of XRP paychans, amount refers to the total funding in a channel and balance denotes how much was claimed out of the channel (see rippled and ripple-lib). To avoid confusions, it would be better to stick with the same terminology as rippled and ripple-lib.

[XRP] Don't accept incoming transfers on a paychan that is about to close

There is a TODO for this in the code, but I am documenting this here as well since it is pretty important and allows to steal from a connector.

The connector accepts incoming transfers on paychans without checking if the channel is about to close. An attacker could 1) open a chan to the connector, 2) right away send a channel close tx, 3) wait until the settle delay is almost over and finally 4) send a transfer, which does leave the claim submitter with no time to send the claim.

Solution: If the channel watcher detects that a channel is about to close, he writes this info back into the DB. The middleware that handles incoming transfers could then read the channel details from the DB and reject incoming transfers on closing channels.

psk implementation incompatible with psk spec

This is obviously on purpose, but I'm wondering what the reason behind it is; https://github.com/emschwartz/ilp3/blob/master/lib/psk.js#L10-L11 uses different strings than interledger/rfcs#336 - why not stick to the strings that are used in PSK/1.0?

use linter

[XRP] Lookup channels in parallel

checkIfChannelsNeedToBeClaimed() checks sequentially if any of the connector's paychans need to be claimed. .getPaymentChannel() and .submit() are both awaited. I wonder how long these functions take to complete if RippleNet is under load. Are there any worst-case estimates?

Assuming that getPaymentChannel lookups take some time to complete, an attacker could open many paychans to the connector and make the connector forward a payment on the channel the connector looks up last. If the time to look up all the channels exceeds the settle delay, the connector will miss submitting the claim before the channel closes.

Solution: Lookup channels and submit claims in parallel. Also, a real connector probably wants to closely monitor RippleNet load and adapt accordingly.

Multi-process access to DB

Claim writer and channel watcher run in separate processes but access the same db. Did you find a solution for accessing the db from multiple processes? From the levelup docs:

Stores like LevelDB are thread-safe but they are not suitable for accessing with multiple processes. You should only ever have a store open from a single Node.js process. Node.js clusters are made up of multiple processes so a levelup instance cannot be shared between them either.
See the aformentioned wiki for modules like multilevel, that may help if you require a single store to be shared across processes.

Did you check multilevel?

[XRP] detect if the channel amount was increased

Related to #18

The incoming xrp paychan middleware does not detect if the sender increases the channel amount. As proposed in #18, the channel watcher should detect modified channel details and write them back to the DB.

Review devDependencies

koa does not seem to be a dev dependency. What about leveldown?

Does ilp3 invalidate the existing ilp architecture?

Hi!

ilp3 looks very promising and hopeful. Does ilp3 invalidate the existing ilp architecture?

Regards,

Nathan Aw Ming Kun

Why is the repository private?

You could also just put an 'Experimental' warning in the readme, maybe?

Call crypto.randomBytes() asynchronically

In edge cases, crypto.randomBytes() will take longer to complete, which would make your connector unresponsive. Excerpt from the node docs:

The crypto.randomBytes() method will not complete until there is sufficient entropy available. This should normally never take longer than a few milliseconds. The only time when generating the random bytes may conceivably block for a longer period of time is right after boot, when the whole system is still low on entropy.

It's better to be on the safe side and call .randomBytes() asynchronically. Here is a snippet:

  return new Promise((resolve, reject) => {
    crypto.randomBytes(4, (err, buf) => {
      if (err) reject(err)
      resolve(buf.readUInt32BE(0))
    })
})

[XRP] Don't initialize the channel balance with 0

Instead of initializing the channel balance with 0, use paychan.balance. If an attacker gets the connector to somehow forget the cached channel details, the attacker could reuse an old channel with balance > 0.

Option for HTTP proxy

I added some code for using an HTTP proxy, thought you might want to include something similar. Makes it easier to see (and mess) with the HTTP requests sent on the wire.

In http.js

const HttpsProxyAgent = require('https-proxy-agent')
...
      response = await fetch(url.format(parsedUri, { auth: false }), {
        method: 'POST',
        headers,
        body: transfer.data,
        compress: false,
        agent: new HttpsProxyAgent('http://127.0.0.1:8080')
      })

Fulfillments and Errors are Responses, Not Requests

The design principal Fulfillments and Errors are Responses, Not Requests in combination with Recommend Using HTTP limits payment timeouts in ilp3 to that of HTTP timeouts. In consequence, only very fast ledgers will be compatible with ilp3. Also, if a single TCP connection along the path drops the payment fails and the connector that had his connection dropped potentially losing funds. This seems to be very fragile.

Double-check that channel watcher interval is less than the settle delay

watchChannels() should double-check that CHANNEL_WATCHER_INTERVAL < DEFAULT_SETTLE_DELAY. There is nothing wrong with these constants in the current code, but if they are changed carelessly it will be easy to steal from the connector.

Race Condition when adjusting balances

balance.js caches account balances. This could result in a race condition. For example:

For two requests arriving at the same time and with the sametransfer.from, the check if an account is cached (line 21) evaluates to false for both. Whichever db read returns first will initialize the cached balance and move on to adjust the balance (line 58). However, when the second db read returns it overwrites the adjusted balance of the first request.

Solution: Make sure that the check if an account exists and the DB read is atomic.

Also, make sure there is no race condition when you write back the balance to the DB (line 83).

Clarify which ILP v1 problems are solved by ILP v3

One thing I'm missing is a more explicit problem statement. What's wrong with ILP v1 that makes you propose to throw away a perfectly usable protocol stack and start almost from scratch? Is ILP v1 not as usable as we thought it was?

ILP v3 [...] aims to simplify the protocol stack and implementation even further and get the open Interledger started faster

That implicitly states that one reason that the open Interledger isn't starting as fast as we would wish is that the protocol stack and implementation aren't simple enough?

I personally think the three things that are stopping the open interledger from growing faster are uncertainty about legal risks, chicken-and-egg problem of "there are no apps -> there are no users -> there are no connectors", and the fact that we have not automated route switching yet. I wouldn't list protocol stack complexities as one of the main problems to solve today, so I think it's worth describing that argument in a bit more detail.

Is it your personal experience that there are complexities in the protocol stack that make adoption unattractive, or did you maybe meet people who told you that the reason they're not yet running a connector is that the protocol stack and/or its implementation is too complex for them? If so, can you be more specific about something that is currently complex and that desperately needs to be simplified?

connectors do not need to know the exact and up-to-date exchange rates of all other connectors.

That's something you can use - explain how in ILP v1, for instance the memory requirements are too high and that that's something that needs fixing.

restrictions on ILP addresses (not being able to use a ledger's address in another ledger's prefix unless the exchange rate is 1:1) that are unnecessary in a forwarding-only system

That's another one, but what problems did you run into due to these restrictions? Can you give an example of a downside of this restriction?

Simple Exchange Rates Instead of Liquidity Curves

That would be an advantage of chunked payments, regardless of whether you do them with ILP v1 or ILP v3.