c-otto / rebalance-lnd

A script that can be used to balance lightning channels of a lnd node

rebalance-lnd's Issues

Make script faster

Currently, nothing is cached. For some requests that makes the script run slow.

Some debug output I created for a typical rebalance job which shows the total time for the given functions (in seconds):

{'generate_invoice': 0.011147022247314453, 'get_routes': 16.460828065872192, 'send_payment': 13.990042924880981, 'get_policy': 19.16523838043213, 'get_payment_hash': 0.0009760856628417969, 'get_graph': 18.756810426712036, 'init': 9.5367431640625e-07, 'get_info': 1.6338262557983398, 'get_expiry': 0.017612218856811523, 'get_own_pubkey': 0.8776724338531494, 'get_edges': 18.761753797531128, 'get_current_height': 0.7577559947967529, 'get_channels': 1.1018741130828857}

some route work, some doesn't...

I have and issue where the script would work on some channels but not all, and the one that doesn't, here's the error message:

Traceback (most recent call last):
File ".\rebalance.py", line 227, in
main()
File ".\rebalance.py", line 55, in main
amount = get_amount(arguments, first_hop_channel_id, last_hop_channel)
File ".\rebalance.py", line 70, in get_amount
amount = get_rebalance_amount(last_hop_channel)
File ".\rebalance.py", line 179, in get_rebalance_amount
return abs(int(math.ceil(float(get_remote_surplus(channel)) / 2)))
File ".\rebalance.py", line 201, in get_remote_surplus
return channel.remote_balance - channel.local_balance
AttributeError: 'NoneType' object has no attribute 'remote_balance'

Cleanup/extend error handling

The error handling in logic.py is ugly and incomplete. I'd like the list of handled errors to be more complete (see https://github.com/lightningnetwork/lightning-rfc/blob/master/04-onion-routing.md). Furthermore, it might make sense to ignore nodes instead of (unidirectional) channels. As a bonus, I'd appreciate someone else checking the current implementation. I'm not sure if the correct edge is ignored (off by one?).

Limit fee relative to rebalance amount

Instead of having a fixed limit, I'd prefer to only rebalance if the fees are at most x% of the amount that I need to transfer.

Feature Request: Pick entry and exit channels

Balancing needs to be able to happen from both sides.
One needs to know the channels where too much is on the remote side, and during the actual rebalance, be able to pick both the incoming and outgoing channels.

Daemon mode

Add a daemon mode where the tool auto rebalances every x hours or whenever an imbalance worth correcting presents itself (what this exactly means TBD)

What about the other way around?

Say I've just opened two new channels. Both of these channels will have 100% of their funds on the local side. Now, if I want to be able to route other people's transactions, I need to get some funds on the remote sides of these channels. Could the script be updated to work in this direction as well?

Specify absolute fee max instead of average fee max

Right now it seems that it's only possible to specify the maximum average fee per hop for a rebalancing route, when really I'd be more interested in specifying a maximum absolute fee (still in terms of the lnd default for a single hop). Basically, I'd rather pay 100 satoshis for 2 hops than 120 satoshis for 3 hops, even though the latter has a lower per-hop average.

This could also allow for some shortcuts to prevent searching for routes when the fees for the first or last hop (specified by the -f and -t arguments) exceed the maximum fee. If I don't want to pay more than 10 times the lnd default fee, and I'm trying to rebalance a channel which itself charges 20 times the lnd default fee, then the rebalancing should not even be attempted potentially saving time and headache. I guess this might change if negative fees ever become a thing, however.

Last minor request, it'd be nice to have a single character alias for this argument, like -m or -f instead of --max-fee-factor.

Require fewer permissions

Currently the documentation states we need admin access. With lnd 0.9.0 it is possible to create macaroons with fewer permissions. See https://github.com/lightningnetwork/lnd/tree/master/macaroons#bakery

UNKNOWN_PAYMENT_HASH: Temporary channel failure, Could not find any suitable route

I am getting Temporary channel failure, Could not find any suitable route, whenever I try to rebalance my channels. What could be the cause?

Use correct amount in QueryRoutes (including fees from added last hop)

Currently we run QueryRoutes for the given amount, ignoring the fees we have to pay by using the channel back to our own node. This currently isn't too important, but in the future we could run into issues when using QueryRoutes with mission control (see lightningnetwork/lnd#3209).

Add unit tests

unable to send, no routes provided (but route is there)

Hello,

I have:

• rebalance-lnd
• LND: 0.6.1-beta

list of channels works... finding routes works... but when i try to pay the route i get the following error:

$ python rebalance.py -t 17 -a 1000
Sending 1,000 satoshis to rebalance to channel with ID 63xxxxxxxxxxxxxxxx
requesting 15 routes from lnd, please wait.
lnd returned 1 routes that will now be tested
Trying route #1
63xxxxxxxxxxxxxxxx -> 62xxxxxxxxxxxxxxxx -> 63xxxxxxxxxxxxxxxx
Traceback (most recent call last):
File "rebalance.py", line 223, in
main()
File "rebalance.py", line 62, in main
response = Logic(lnd, first_hop_channel_id, last_hop_channel, amount, channel_ratio, max_fee_factor).rebalance()
File "/usr/local/src/rebalance-lnd/logic.py", line 48, in rebalance
response = self.lnd.send_payment(payment_request, [route])
File "/usr/local/src/rebalance-lnd/lnd.py", line 98, in send_payment
return self.stub.SendToRouteSync(request)
File "/usr/lib64/python2.7/site-packages/grpc/_channel.py", line 565, in call
return _end_unary_response_blocking(state, call, False, None)
File "/usr/lib64/python2.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.UNKNOWN
details = "unable to send, no routes provided"
debug_error_string = "{"created":"@1559085040.643817147","description":"Error received from peer ipv4:127.0.0.1:10009","file":"src/core/lib/surface/call.cc","file_line":1046,"grpc_message":"unable to send, no routes provided","grpc_status":2}"

is this issue with LND or rebalance-lnd? any ideas?

Crash: Unable to find a path to destination

I tried rebalancing a channel I opened to my Eclair wallet on my phone.

Traceback (most recent call last):
  File "./rebalance.py", line 223, in <module>
    main()
  File "./rebalance.py", line 62, in main
    response = Logic(lnd, first_hop_channel_id, last_hop_channel, amount, channel_ratio, max_fee_factor).rebalance()
  File "/home/cotto/rebalance/logic.py", line 34, in rebalance
    if not routes.has_next():
  File "/home/cotto/rebalance/routes.py", line 24, in has_next
    self.update_routes()
  File "/home/cotto/rebalance/routes.py", line 43, in update_routes
    self.request_routes(num_routes_to_request)
  File "/home/cotto/rebalance/routes.py", line 47, in request_routes
    routes = self.lnd.get_routes(self.last_hop_channel.remote_pubkey, self.get_amount(), num_routes_to_request)
  File "/home/cotto/rebalance/lnd.py", line 78, in get_routes
    response = self.stub.QueryRoutes(request)
  File "/home/cotto/.local/lib/python2.7/site-packages/grpc/_channel.py", line 547, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/cotto/.local/lib/python2.7/site-packages/grpc/_channel.py", line 466, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "unable to find a path to destination"
	debug_error_string = "{"created":"@1555085155.113312254","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1036,"grpc_message":"unable to find a path to destination","grpc_status":2}"

Make `low_local_ratio_after_sending` regard fees

Currently, the first channel might be emptied to less than 50% local capacity due to fees. This is not intentional, and should be fixed. The logic in low_local_ratio_after_sending might be a good start to fix this.

Improve routing efficiency by utilizing source_pub_key argument in QueryRoutes

If the FROM channel is specified on the command line, the script should probably call GetRoutes while specifying the source_pub_key argument as the first node in the route (the node on the other end of the FROM channel). Currently, GetRoutes tries to find N routes indiscriminately and the check to match the FROM channel happens after routes are returned. This is inefficient and thus it takes longer to find the suitable route and could result in higher chances of routing failure.

I observed that often the 15/30/.. routes returned by the RPC call do not match the FROM channel and the rebalancing fails for that reason

Cleanup arguments

The arguments (and -h) are hard to understand. Maybe it helps using argument groups: https://docs.python.org/2/library/argparse.html#argument-groups

Impossible amount is used when specifying first and last hop

The amount used for the rebalance, if not given, is computed based on last hop (-t). This won't work if the first hop (-f) does not have enough satoshis on the local side.

Odd behaviour when balancing one of the two channels with same node

I've been using it for a month now. It is useful and works very well! thanks for this tool!

Nevertheless, I've noticed that whenever I try to balance one of the two channels with a the same node (ie, I have two channels open with the same node), it behaves oddly.

First time I experienced that was a around a month ago. I reduced the amount each time it wasn't able to route up until a payment of 1000 sats went through but to the wrong channel. I closed one of the channels and everything worked fine for the only channel available.

Right now I have the same situation with a different node. Two channels with 1Msats and 2Msats. Here it is a list of what I thing was some weird behaviour:

• Forcing with option '-f' of two channels, no success in any of them.
  -Without using '-f' and specifying an amount with '-a' at 40ksats, it went through, but with what I expected lower fees. I paid 18sats in fees is higher than the default fee isn't it?. If not mistaken, max fee is given by the max fee factor at 10, which is 10 times the fee at 10*(1sat + 1e-6*Amount), and I would have expected a fee at less than 10.4sat.

Any hints on what can I do to give more information or debug what's happening?

admin@thundroid:~/rebalance-lnd$ python3 rebalance.py -t 636###### -f 624######
Sending 40,272 satoshis to rebalance to channel with ID 636######
Forced first channel has ID 624#####
requesting 15 routes from lnd, please wait.
lnd returned 15 routes that will now be tested
requesting 30 routes from lnd, please wait.
lnd returned 30 routes that will now be tested
requesting 45 routes from lnd, please wait.
lnd returned 45 routes that will now be tested
requesting 60 routes from lnd, please wait.
lnd returned 60 routes that will now be tested
admin@thundroid:~/rebalance-lnd$ python3 rebalance.py -t 636###### -f 627######
Sending 20,270 satoshis to rebalance to channel with ID 636######
Forced first channel has ID 627######
requesting 15 routes from lnd, please wait.
lnd returned 15 routes that will now be tested
requesting 30 routes from lnd, please wait.
lnd returned 30 routes that will now be tested
requesting 45 routes from lnd, please wait.
lnd returned 45 routes that will now be tested
requesting 60 routes from lnd, please wait.
lnd returned 60 routes that will now be tested
admin@thundroid:~/rebalance-lnd$ python3 rebalance.py -t 636###### -a 40000
Sending 40,000 satoshis to rebalance to channel with ID 636######
requesting 15 routes from lnd, please wait.
lnd returned 15 routes that will now be tested
Trying route #1
635###### -> 636###### -> 621###### -> 636######
Success! Paid 18 Satoshi in fees
Tried routes:
635###### -> 636###### -> 621###### -> 636######
Successful route:
635###### -> 636###### -> 621###### -> 636######
payment_preimage: "######"
payment_route {
  total_time_lock: 579893
  total_fees: 18
  total_amt: 40018
  hops {
    chan_id: 635######
    chan_capacity: 400000
    amt_to_forward: 40018
    expiry: 579849
    amt_to_forward_msat: 40018040
    fee_msat: 40
    pub_key: "03######"
  }
  hops {
    chan_id: 636######
    chan_capacity: 1000000
    amt_to_forward: 40018
    expiry: 579845
    amt_to_forward_msat: 40018000
    fee_msat: 40
    pub_key: "02######"
  }
  hops {
    chan_id: 621######
    chan_capacity: 5000000
    amt_to_forward: 40000
    fee: 18
    expiry: 579815
    amt_to_forward_msat: 40000000
    fee_msat: 18000
    pub_key: "03######"
  }
  hops {
    chan_id: 636######
    chan_capacity: 4000000
    amt_to_forward: 40000
    expiry: 579815
    amt_to_forward_msat: 40000000
    pub_key: "03######"
  }
  total_fees_msat: 18080
  total_amt_msat: 40018080
}
payment_hash: "######"

Deal with message limits

  File "/home/cotto/rebalance/lnd/lnd.py", line 41, in get_graph
    graph = self.stub.DescribeGraph(ln.ChannelGraphRequest())
  File "/home/cotto/.local/lib/python2.7/site-packages/grpc/_channel.py", line 547, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/cotto/.local/lib/python2.7/site-packages/grpc/_channel.py", line 466, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.RESOURCE_EXHAUSTED
	details = "Received message larger than max (4230348 vs. 4194304)"
	debug_error_string = "{"created":"@1545772110.601449580","description":"Received message larger than max (4230348 vs. 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":174,"grpc_status":8}"

Feature Request: Exclude Channels

It would be helpful to be able to exclude one or more channels when rebalancing.

In my case, I have multiple channels to a merchant. After sending them funds, I like to top-up the local balance on that channel so I have max available to spend in the future. But because I have multiple channels to them, the script will often select the other channel to them to balance from unless I specify a single -f channel. This takes a bit of time since I will often have to try a few channels individually before finding a viable route. If instead I could just indicate which channels I did not want the script to use, it could try all the rest without having to re-run the script manually for each -f channel.

Optimization: Add Chan ID to return

FYI: LND can have more than one channel between nodes, so returning only "Node Key" isn't sufficient. The chan ID should be returned, and used as well.

python3: type mismatch in routes.py

trying to run on windows with git bash, when i run ./rebalance.py 26 100000 the result was:

• File "./rebalance.py", line 113, in
  main()
  File "./rebalance.py", line 52, in main
  response = Logic(lnd, remote_pubkey, amount).rebalance()
  File "D:\tests\rebalance-lnd-master\logic.py", line 22, in rebalance
  if not routes.has_next():
  File "D:\tests\rebalance-lnd-master\routes.py", line 24, in has_next
  self.update_routes()
  File "D:\tests\rebalance-lnd-master\routes.py", line 43, in update_routes
  self.request_routes(num_routes_to_request)
  File "D:\tests\rebalance-lnd-master\routes.py", line 50, in request_routes
  modified_route = self.add_rebalance_channel(route)
  File "D:\tests\rebalance-lnd-master\routes.py", line 74, in add_rebalance_channel
  self.update_amounts(hops)
  File "D:\tests\rebalance-lnd-master\routes.py", line 122, in update_amounts
  hop.amt_to_forward = amount_to_forward_msat / 1000
  File "C:\Users\fr4nz\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\python_message.py", line 662, in field_setter
  new_value = type_checker.CheckValue(new_value)
  File "C:\Users\fr4nz\AppData\Local\Programs\Python\Python37-32\lib\site-packages\google\protobuf\internal\type_checkers.py", line 133, in CheckValue
  raise TypeError(message)
  TypeError: 100000.0 has type <class 'float'>, but expected one of: (<class 'int'>,)

I don't know if is a os related bug but i solved with
Fr4nZ82@ed9e90e

Ignore edge/node if route is skipped due to high fees

Otherwise lnd might return the same route again and again.

Specify a different lnd directory

The script should have an option on the command line to specify a different location to the .lnd directory. I don't have it in my home folder, so currently the only way I can use the script is to edit lnd.py and change the hardcoded directory.

Refactor class structure

rebalance and logic are meant to be the same. Each class should have a clear resposibility.

Only consider channels where at least 1 satoshi is required for rebalance

In corner cases a channel is shown with a required rebalance amount of 0. In this case the channel should be considered rebalanced, although the local and remote sides are not identical (they differ by 1 satoshi).

lnd 0.7 beta: Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.

I can't even do ./rebalance.py -l, seems to be related to lnd 0.7-beta.

Try and force the first hop

The current logic looks at channels which could use some incoming funds, but does not pair that with channels which could be emptied a bit to reach a 50/50 state. The goal is to be able to diagnose candidate routes with this in mind, and to support the rebalance mode with that constraint too.

Deal with illegal arguments (wrong pubkey etc.)

Make sure something useful happens if the user enters bogus data.

Feature Request: Show interesting channels (statistics)

I'd like to show usage information about channels, possibly based on lncli fwdinghistory. One output could be "Channel X had 123 incoming and 456 outgoing transactions in the last 24 hours, with a net change of -789 Satoshi.". Together with the information of rebalancy.py -l (-i/-o) this could help identify the channels worthy of a rebalance.

Gather intel on failing paths in order to drive routing heuristics

There are still many failures on payment attempts. While we don't have much visibility on why these are failing, we can probably infer two pieces of information from these failures, and use that to prioritise certain routes in the future for faster and more successful rebalances.
The main reasons for failure today are:

• liquidity on channels
• offline/out of service nodes

A small concept PR on my repo displays paths tried during the rebalancing process (https://github.com/wamde/rebalance-lnd/pull/4/files).

We could start by storing the paths tried for a given amount and whether it worked or failed. We can add the node pubkeys to store directionality information too.
Crossing information between such tries could show that there is some probability that a certain channel is either full in a direction or just offline. If a channel is full in one direction we can use this as a signal that it may have liquidity in the other direction.

There are many caveats with such an approach but it could probably help with rebalancing these days when the network is not so dynamic yet.

Important: Work around QueryRoutes numroutes deprecation

See lightningnetwork/lnd#2497

Currently we ask for n routes and attempt to construct a rebalance paymant based on that. In the future we will only get a single route. Right now I don't understand if that will be a problem.

Multiple instances of bitcoind and lnd?

My raspiblitz was running low on memory, so I accessed it via ssh and ran sudo htop. I was surprised to see so many bitcoind and lnd processes. Is this normal?

bitcoind says it has 20 connections and lnd is connnected to 19 peers, so could it be the case that every connection/peer spawns its own child process? or is this a problem?

Simple auto mode

Add a simple auto mode, where the tool automatically finds out which is the best rebalance and carries it out (with some constraints, like max fees)

Crash: channel not found

Sometimes this happens, I have no idea why:

Traceback (most recent call last):
File “./rebalance.py”, line 223, in 
main()
File “./rebalance.py”, line 62, in main
response = Logic(lnd, first_hop_channel_id, last_hop_channel, amount, channel_ratio, max_fee_factor).rebalance()
File “/home/cotto/rebalance/logic.py”, line 41, in rebalance
if self.route_is_invalid(route):
File “/home/cotto/rebalance/logic.py”, line 71, in route_is_invalid
if self.low_local_ratio_after_sending(first_hop, route.total_amt):
File “/home/cotto/rebalance/logic.py”, line 88, in low_local_ratio_after_sending
remote = channel.remote_balance + total_amount
AttributeError: ‘NoneType’ object has no attribute ‘remote_balance’

Python3: import problem on the grpc_generated folder

Got the following error on files inside grpc_generated folder:

    import rpc_pb2 as rpc__pb2
ModuleNotFoundError: No module named 'rpc_pb2'

Resolution is adding from grpc_generated to the import line import rpc_pb2 as rpc__pb2
Other files inside grpc_generated folder have similar problem.

P.S. I am using python 3.7.4 on my raspberry pi

INCORRECT_OR_UNKNOWN_PAYMENT_DETAILS

failure {
code: INCORRECT_OR_UNKNOWN_PAYMENT_DETAILS
failure_source_index: 6
height: 604253
}

Related to #65.
Failure previously had name UNKNOWN_PAYMENT_HASH, see #69.

Provide verbose logging, cleanup current default log messages

The script should provide only necessary information in the default setting, with more helpful (debug) information using -v (and possibly -vv etc.). The use of print to STDOUT/STDERR should be cleaned up as part of this, possibly involving a proper logging framework.

can't execute

Background
I try to excecute and have this error :

$ sudo python3 rebalance.py -l
> Traceback (most recent call last):
>   File "rebalance.py", line 8, in <module>
>     from lnd import Lnd
>   File "/home/admin/rebalance-lnd/lnd.py", line 4, in <module>
>     import grpc
> ImportError: No module named 'grpc'

My environment
Raspberry Pi3b linux -> nodo as Stadicus Raspibolt
lncli version 0.5.2-beta
bitcoind

Steps to reproduce
I git clone the repository to /home/admin/rebalance-lnd
I cd ~/rebalance-lnd and then python3 rebalance.py -h

Expected behaviour
Script to work

Try with lower amounts on failure

If rebalancing did not work for amount 2N, try again (twice?) with amount N (recursively?).

Support configurable channel ratios

Instead of the command with no arguments returning "channels with 50%+ on one side"
Return channels where X amount is on either side. Default could be 75%.

No one is going to try to rebalance a channel with 50.0001% on local side. Running a large hub, I wouldn't do any rebalancing until the amount left is smaller than a typical transaction... which about 5-20%, depending on the channel.

And, anyone doing manual rebalancing is going to want to know all channels that need rebalancing, not just the ones with high local balance.
You might even make them separate commands... one to show Local balance > X% and a 2nd argument to return only channels with remote balance > X%

Move from lncli to gRPC

TypeError: not all arguments converted during string formatting

I am following the instructions but I can't get it to work and the error message is helping in solving the issue. What is the problem?

$ python rebalance.py -t 614105831440187392 -p 20
Sending 29,302 satoshis to rebalance to channel with ID 614105831440187392
Traceback (most recent call last):
  File "rebalance.py", line 241, in <module>
    success = main()
  File "rebalance.py", line 70, in main
    max_fee_factor).rebalance()
  File "/home/john/Desktop/rebalance-lnd/logic.py", line 46, in rebalance
    success = self.try_route(payment_request, route, routes, tried_routes)
  File "/home/john/Desktop/rebalance-lnd/logic.py", line 53, in try_route
    if self.route_is_invalid(route, routes):
  File "/home/john/Desktop/rebalance-lnd/logic.py", line 114, in route_is_invalid
    debugnobreak("Target channel is first hop, " % first_hop.chan_id)
TypeError: not all arguments converted during string formatting
(base) 

Does not work with python 3: AttributeError: 'str' object has no attribute 'decode'

Using Python 3.6.8 on Ubuntu 18.04, latest code from master, lnd version 0.7.0-beta commit=v0.7.0-beta-rc3 installed with routerrpc, latest requirements.txt installed with pip. I'm trying to rebalance a channel with the following command and getting an error.

$ python rebalance.py -f 627916797039411200 -t 630216975379267584 -a 500000 --max-fee-factor 3
Sending 500,000 satoshis to rebalance to channel with ID 630216975379267584
Forced first channel has ID 627916797039411200

Trying route #1
627916797039411200 -> 623648492953534464 -> 631089987556999169 -> 630216975379267584
Traceback (most recent call last):
  File "rebalance.py", line 227, in <module>
    main()
  File "rebalance.py", line 63, in main
    Logic(lnd, first_hop_channel_id, last_hop_channel, amount, channel_ratio, excluded, max_fee_factor).rebalance()
  File "/home/bitcoin/rebalance-lnd/logic.py", line 54, in rebalance
    response = self.lnd.send_payment(payment_request, route)
  File "/home/bitcoin/rebalance-lnd/lnd.py", line 110, in send_payment
    request.payment_hash = self.hex_string_to_bytes(payment_request.payment_hash)
  File "/home/bitcoin/rebalance-lnd/lnd.py", line 115, in hex_string_to_bytes
    return hex_string.decode("hex")
AttributeError: 'str' object has no attribute 'decode'

Make it easier to run this tool remotely

The "server address" and "macaroon/cert locations" are currently hardcoded into the script at lnd.py in a way that assumes that the script will be run in the same environment as the lnd node. Specifically for running the scripts remotely, it would be useful if there was some way to allow the user to define:

1. a common folder location where credentials (admin.macaroon & tls.cert) are stored, and
2. the ip:port that the lnd node is listening on for rpc connections

Zap Desktop folks have a decent UX defined for how this process works. Their project could be a good place to start for guidance for anyone looking to take a crack at this issue. One idea could be to implement --serverip and --credentialspath arguments to accomplish this.

Debug mode for route candidates

Routes tend to fail for various reasons, often unclear.
We should add a proper debug mode to show routes constructed to be able to diagnose what's up or what they have in common which might fail which may lead to heuristics to try and add diversity to them.

UNKNOWN_PAYMENT_HASH

Using Python 2.7.15+ on Ubuntu 18.04, latest code from master, lnd version 0.7.0-beta commit=v0.7.0-beta-rc3 installed with routerrpc, latest requirements.txt installed with pip. I'm trying to rebalance a channel with the following command and getting an error. I've tried this with different "to" channels and different amounts and I get the same error.

$ python rebalance.py -f 627916797039411200 -t 630216975379267584 -a 500000 --max-fee-factor 3
Sending 500,000 satoshis to rebalance to channel with ID 630216975379267584
Forced first channel has ID 627916797039411200

Trying route #1
627916797039411200 -> 623648492953534464 -> 631089987556999169 -> 630216975379267584
failure {
  code: UNKNOWN_PAYMENT_HASH
  failure_source_pubkey: "\003\201\030\340I\017\215=\360{k\334^p4\336\351\rv\220\022vM1\277\353\026ri\206h$\360"
}

Unknown error code 1
Could not find any suitable route

Possible failed routing because last leg is private

Running ./rebalance.py -t 6xxxxxxx -a 100000 fails with the message:

Checking validity route:
6xxxxx -> 6xxxxx -> 6xxxxx
Total fees (msat): 100
Local/Remote/ratio: 782306 100345 0.886 (0.500)
Trying route #1
6xxxxx -> 6xxxxx -> 6xxxxx
Error: unable to route payment to destination: FeeInsufficient(htlc_amt==100000000 mSAT, update=(lnwire.ChannelUpdate) {
 Signature: (lnwire.Sig) (len=64 cap=64) {
....
 },
 ChainHash: (chainhash.Hash) (len=32 cap=32) 0000000000xxxx,
 ShortChannelID: (lnwire.ShortChannelID) 5xxxxx:xxxx:1,
 Timestamp: (uint32) 1554661864,
 MessageFlags: (lnwire.ChanUpdateMsgFlags) 00000001,
 ChannelFlags: (lnwire.ChanUpdateChanFlags) 00000000,
 TimeLockDelta: (uint16) 30,
 HtlcMinimumMsat: (lnwire.MilliSatoshi) 600000 mSAT,
 BaseFee: (uint32) 0,
 FeeRate: (uint32) 800,
 HtlcMaximumMsat: (lnwire.MilliSatoshi) 1999800000 mSAT,
 ExtraOpaqueData: ([]uint8) <nil>
}

It looks like the total routing cost returned by QueryRoutesRequest underestimates the actual final routing fees. It is possible that the reason for that is that the last leg of that particular route is private. The Total fees (msat) in this case show 100 msat, which is exactly the fees to route over the second leg in my case. The fee to route to the third (and final) leg should be (an additional) 80000 msat (based on 800 feerate as shown in the error message and 100K rebalancing amount).

The way to reproduce this is to request a private channel with lnbig.com with some balance on their side and try to rebalance to that channel.

Possible solution could be (1) checking if the last leg is private (2) if yes, query the other party's fees (3) add that extra fee to the routing attempt.

Not sure how you can query their fees using RPC, but on the command line it's lncli getchaninfo.

Error when trying to rebalance

Hi,

I'm trying to use the latest rebalance-lnd on master, with lnd lnd version 0.7.0-beta commit=v0.7.0-beta-rc1-9-gd6d87e12feeb197f599a329c079ecef17a4f2b4a.

I keep getting the following error about

~/scripts/rebalance-lnd (master ✘)✭ ᐅ ./rebalance.py -t 2 -a 10000
Sending 10,000 satoshis to rebalance to channel with ID 628199371496423424

Trying route #1
<id1> -> <id2> -> <id3> -> <id4>
Traceback (most recent call last):
  File "./rebalance.py", line 227, in <module>
    main()
  File "./rebalance.py", line 63, in main
    Logic(lnd, first_hop_channel_id, last_hop_channel, amount, channel_ratio, excluded, max_fee_factor).rebalance()
  File "/home/admin/scripts/rebalance-lnd/logic.py", line 55, in rebalance
    response = self.lnd.send_payment(payment_request, route)
  File "/home/admin/scripts/rebalance-lnd/lnd.py", line 111, in send_payment
    return self.router_stub.SendToRoute(request)
  File "/home/admin/.local/lib/python2.7/site-packages/grpc/_channel.py", line 565, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/admin/.local/lib/python2.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.UNIMPLEMENTED
	details = "unknown service routerrpc.Router"
	debug_error_string = "{"created":"@1560903041.474343437","description":"Error received from peer ipv4:127.0.0.1:10009","file":"src/core/lib/surface/call.cc","file_line":1046,"grpc_message":"unknown service routerrpc.Router","grpc_status":12}"

Could you let me know what I'm doing wrong? Thanks!