payment failure with wallet that open channels on the fly about galoy

[ryanthegentry]

@esneider are you the right person to help here?

[champo]

Hi! I'd like to test this out, is https://apps.apple.com/sv/app/bitcoin-beach-wallet/id1531383905 the official app for iOS?

[nicolasburtey]

also still doesn't work with lntxbot

[krtk6160]

Doesn't work with freshly installed phoenix wallet also

[nicolasburtey]

also may not work with new Muun Wallet.

may also be related to #22

[nicolasburtey]

Some feedback from Muun through some user:

This is going to have to do with them, since we do not have enough information to understand what is happening. The fact that Muun spontaneously opens channels when you need them shouldn't cause these problems. On the other hand, if I can think of two things that could be causing this, I will mention them to you in case you talk to them:

• RouteHints: when the recipient node is private - something very common in self-custodial wallets like Muun - the invoices carry a component called RouteHints that some services do not recognize and support.
• Commission policy from them: some lightning services have commission policies that, when not followed, generate unexpected failures and often with an inconsistent error message. If this is the case, some amounts should work and some shouldn't.

@ryanthegentry who's the person at Muun to tag if we have more questions?

[esneider]

CCing @champo

[nicolasburtey]

yes this is @champo

[champo]

So I took a look at the code and found the problem. Seems like you're using payment probes with random payment hashes[1]. Muun's public node intercepts HTLCs and rejects any unknown hashes directly, apps only receive payments that are known to succeed. Since the error comes from an intermediate node it's considered a failure by the probe. AFAIK Breez takes a similar approach, and Pheonix used to but changed it so that all messages reach the apps.

Strike also does probing and was having the same issue. The solution had three parts:

1. Detecting Breez/Muun invoices (they have impossible short channel ids in their route hints)
2. Using a deterministic payment hash for the probe
3. Making the probe accept errors from the hub node.

At some point we're going to refactor the flow so that apps can reply to message directly, but it's not an immediate priority for us right now.

[1] The backend calls probeForRoute, which then calls payViaRoutes without specifying a payment hash which casues it to default to a random payment hash.

[nicolasburtey]

thanks for the detail feedback @champo

1. Detecting Breez/Muun invoices (they have impossible short channel ids in their route hints)

would you happen to have a code snippet for that? some other open source wallet have implemented this logic?

I still am not sure why this would fail because we have a fall back option. if the probe failed, we fall back on payViaPaymentDetails which is the normal way of doing a transaction. any clue on this one?

[champo]

would you happen to have a code snippet for that? some other open source wallet have implemented this logic?

I don't. The logic would be something like:

if (routeHint[0].channel_id <= 500000) {
 // this is breez
} else if (routeHint[0].channel_id >= 8388608) {
 // this is Muun
}

I still am not sure why this would fail because we have a fall back option. if the probe failed, we fall back on payViaPaymentDetails which is the normal way of doing a transaction. any clue on this one?

I think probe is done to estimate fees, so this fails way before the payment is attempted.

[nicolasburtey]

I think probe is done to estimate fees, so this fails way before the payment is attempted.

probe is optional. payment can still be send if probe fails. in that case fees are just an upper level estimation and when reel fees are know the difference is being returned to the user

[champo]

Ah did not notice that! Will investigate further then.

[champo]

Ok so I was able to get a few successful payments in and a few failures too. I think I found the trick to it: If I tap pay before the probing fails, the payment is sucessful.

I believe the problem lies in how LND's mission control is marking the channel as non-existant since the interceptor returns an error that indicates permament failure back. Then when the payment asks for routes, the last hop is considered bad and rejected. Further validation of this would be running lncli resetmc between probe failure and pay, which should allow the payment to be made. I'm testing using the live app so I can't say for sure, @nicolasburtey can you help me verify this? Another way could be using lncli querymc and checking what it says about the destination node from the invoice.

If this is correct, then a mitigation might be skipping probing for Muun/Breez. Or maybe doing probing up to the public node? Not sure what other avenues there might be; I'll sleep on it.

[nicolasburtey]

oh interesting. thank you @champo I have not thought about the mission controller being an issue. I've never dive into it. I can see why, if the probe to this destination failed, that the mc would say that there is no need to send a payment. maybe my probing logic is flawed without considering the mission controller.

any insight on this @Roasbeef?

a quick fix would indeed be to not probe to those destinations for now.

[Roasbeef]

Seems related to lightningnetwork/lnd#5230? It's fixed in 0.13 which should be released this week.

[Roasbeef]

A temp channel failure is sent as well (already): https://github.com/lightningnetwork/lnd/blob/master/htlcswitch/interceptable_switch.go#L150

I think this might be related to the fact that we won't apply a channel update when it comes from a private channel. We have a fix for this (lightningnetwork/lnd#5218) but it didn't quite make it into 0.13. We're looking to do an 0.13.1 release that includes this and other important fixes however.

[vindard]

a quick fix would indeed be to not probe to those destinations for now.

I was just doing some testing again on this from my own node by doing fee probes first and observing payment patterns after. Can confirm what @champo said where there are 2 scenarios where we can successfully do a payment after failed fee probe:

1. resetmc and immediately retry
2. wait a certain amount of time after failed probe and then retry without probe (without resetting mc)

@nicolasburtey while we wait for this to be fixed, should we do something like automatically skipping the probe if we see Muun's magnetron node in the route hints for now? This may be a better alternative than not being able to pay Muun at all after fee probes.