connectd hanging while being unable to connect to peers about lightning

[kilrau]

OK sth different then, we'll go and test #7365

[kilrau]

So looks like #7365 didn't fix anything for you...

[grubles]

This is more of a medium-sized node and wasn't running into the CPU usage that PR addresses, so I'm not sure. Also the other machine I tested on has a single signet channel and was experiencing the issue described above.

[michael1011]

I can reproduce that problem on a fresh, new node:

1. Create a new node with latest master
2. Connect to some peers
3. Watch connectd spike to 100% CPU

For convenience to reproduce this I created a little script.
Run lightning-cli listnodes > nodes.json and then this python script to connect to some nodes and you'll see connectd go wild:

#!/usr/bin/env python3
import json
import subprocess

with open('nodes.json') as f:
    nodes = json.load(f)["nodes"]

print(f"Got {len(nodes)} nodes")

with_address = []

for node in nodes:
    if "addresses" not in node or len(node["addresses"]) == 0:
        continue

    with_address.append(node)

print(f"{len(with_address)} with address")

ipv4 = []

for node in with_address:
    for address in node["addresses"]:
        if address["type"] != "ipv4":
            continue

        ipv4.append(f"{node['nodeid']}@{address['address']}:{address['port']}")

print(f"{len(ipv4)} with IPV4 address")

for (i, address) in enumerate(ipv4):
    print(f"Connecting to {i+1}/{len(ipv4)}: {address}")
    res = subprocess.Popen(
        f"timeout 10 lightning-cli connect {address}",
        shell=True, 
        stdout=subprocess.PIPE,
    ).stdout.read()
    try:
        print(json.dumps(
            json.loads(res),
            indent=4,
        ))
    except:
        print("Connect timed out")

Edit:

This is definitely a regression since v24.05. I created a new node with v24.05 and ran the script; it was just fine. Updated to master, ran it again and connectd jumped to 100% CPU before it even connected to the first peer.