ayeowch / bitnodes Goto Github PK

The culprit appears to be a recent change to start.sh: 5e72029#diff-d0b2ef35f4d7cb0284dd5cf566b62f51

For example, crawl.f9beb4d9.conf does not exist.

Perhaps the relevant conf files just need to be git added?

Now that the site has been collecting data for a long time, do you think you could add more views, like 180d and 1y, maybe even All?

I think your log should be written to a file (someone can always tail the file if they want to watch the log). Then every 100 nodes should print the node count found. (or every 1000, whatever).

I find having to use an SQL editor to look at the nodes found kinda annoying.

Hi,

is there a way to activate a node via the API?
Or what are the alternatives to use the Bitnodes page?

The API states

Set Bitcoin address for an activated node. Node must be activated separately, i.e. from https://getaddr.bitnodes.io/nodes/<ADDRESS>-<PORT>/, before it can be updated using this endpoint.

start.sh shell needs custom config like crawl.MAGIC_NUMBER.conf. Can you tell me how to do that or explain every part?

Does it make sense for https://getaddr.bitnodes.io/nodes/ to filer out its own USER_AGENT (/getaddr.bitnodes.io:0.1/)? 'Reachable nodes' have gone undefined, but /getaddr.bitnodes.io:0.1/ is certainly a 'peer' in the reference client sense.

Or can you make it available?

Thanks.

Hi,

I am new to bitnodes. I want to map bitcoin topology. I mean from the API, I can just get the total nodes. I want to know how they are connected to each other. I mean I want to reproduce the connection graph of bitcoin.

Could anyone can guide me on how can I do that ? Also, How to install bitnodes to work on it ?

Easily reproducible:

1. enter search term "Satoshi", save ranking of first 50 entries
2. clear search term, compare with first 50 entries saved before
   Even for the same no. 1 entry (212.9.185.194:8333 atm), the PIX values are different (not the ranking in this case).

Otherwise, great tool, thanks!

We just started shipping our decentralized home mining/node solution for consumers (futurebit.io) and will soon have thousands of nodes coming online from these devices. Would be awesome if there is an option to filter through custom flagged user agent strings on top of the version agent. I know blockchair filters through the full user agent string but their crawler does not seem to be as comprehensive as yours.

Hello. I'm trying to check if my node is running, but when I submit my ip address, system says, it not valid. Its really strange, cause I have static and reachable ip 194.44.53.70.

How hard would it be to run a Litecoin version of this software?

https://bitnodes.21.co/nodes/leaderboard/?q=satushi

My node for the last few months has been getting an N value of around 0.43 (>0.51) yesterday, resulting in my node getting to position 1 in the leadboard. Today, the N value is zero. Has something changed at leaderboards or is my node suddenly (without reason I am fathom) stopped sending addresses? (Seems very unlikely that it can still be up yet not sending addresses).

This can be easily seen in the leaderboard:

https://bitnodes.earn.com/nodes/leaderboard/?q=0.16.0

All of them have Si=0.5

From version 0.16.0 onwards, bitcoin started advertising the NODE_NETWORK_LIMITED bit (1 << 10) that for a node wilth all services, becomes 1037 (example https://bitnodes.earn.com/nodes/46.32.97.82-8333/).

You should consider 1037 as Si = 1

i use ’https://bitnodes.earn.com/api/v1/snapshots/latest/‘ to download the node's informations for analysis
. but i can'nt find the node with USER AGENTS: Bitcoin ABC 0.17.1

Some of us run nodes on Dynamic IPs, making it difficult to spot our nodes or even participate in the incentive program. Some suggestions to solve this:

1. We (nodes) register our Dynamic DNS names (e.g. mynode.now-ip.org) with our service, you periodically resolve all those names to IPs and replace those IPs with names in your reporting.
2. Or, you create a new API call we (nodes) call once a day. In the call we use the name we want to appear in the reporting. When you receive the API call you make a note of the IP it came from plus the name used in call. Then replace those IPs with names in your reporting.

Is there a latency between a node coming online and it being reachable by your scripts. I'm currently running an onion node and can verify that I'm receiving inbound connections. I can also verify the node is reachable using bsd-netcat:

nc -v -n -z -X 5 -x 127.0.0.1:9050 somebtconionnode.onion 8333

But when I try to activate somebtconionnode.onion:8333 (made up name) through https://bitnodes.io/nodes/somebtconionnode.onion-8333/ it never transitions from DOWN to UP. What's more, I'm connected (outgoing) to a half dozen onion nodes. Since I'm connected to them, they must be reachable for inbound connections on the network. If I try activating those nodes, I get the same results as my node.

Do I need to have my onion up for 24 hours or something before it can be activated on bitnodes.io? Is there anything else I should do?

I've also included my actual node info in an attached PGP file for @ayeowch

msg-ayeowch.asc.gz

Right now the leaderboard API call brings back a list. Add functionality to search for a specific IP to get the leaderboard position that IP has. Another approach would be adding the number to the existing API call to get node information.

Just updated to Bitcoin Core 0.21.0 and I run it behind Tor. Now with the Tor V3 support, my node is not visible on bitnodes. Am I correct into thinking it's not yet supported?

When viewing a node's services, only NODE_NETWORK, NODE_BLOOM and NODE_WITNESS seem to be displayed. NODE_XTHIN currently is not.

Also, it would be great if it was possible to search for nodes based on service bit.... :)

I notice there are bitnode probes running on testnet - is there a website to see the results from these probes?

or just nodes list ?

i liked

https://bitnodes.21.co/dashboard/transactions/

Hello, I'm a developer on the Bitcoin Unlimited project. We've been looking at various types of connection slot attack mitigation and I noticed that the behavior of the bitnodes crawlers seems to be to NOT issue a disconnect when it's done making it's handshake and gathering whatever into it needs. This causes a slot to be used up unnecessarily but more importantly our slot mitigation software flags an that an eviction when another connection comes in to bump off the bitnodes connection. While this only happens when the slots are all full it would be great if the crawlers would disconnect on their own since we end up banning nodes (still in testing) for a short while if they exceed their eviction limits. If bitnodes crawlers would do their disconnect then we could set our limits lower and prevent more bad actors from trying to game the connection slots.

I do not receive email notifications when I subscribe to a node.

According to the leaderboard ranking rules, Bi is

BI = Block index
    BI = 1.0 if valid block is returned in response to getdata request, 0.0 if otherwise

But for my node this value is always zero. Could you clarify how is it calculated and how can I troubleshoot why is it always zero?

Thanks!

like in main page https://bitnodes.earn.com/ below Join the network?

It would be nice to include at least a short guide on how to use this tool with some examples. Assume I want to get the latest snapshot of Bitcoin peers: which file do I run? Do I have to manually configure anything? Where do I expect to get the results? What do I do if I want to get the same stats for another network, say, the Bitcoin testnet?

Sure, one can figure out the answers to these questions by reading the source code, but even a short manual would greatly speed up this process.

The current service is great to see day by day what the current network looks like, but it's not possible to visualise how things change over time. It would be great if you could plot these on a graph by day then we could really see how things are changing.

got it ----> pip install -r requirements.txt :D

cat crawl.master.out

Traceback (most recent call last):
File "crawl.py", line 31, in
from gevent import monkey
ImportError: No module named gevent

is there any install manual or so ? required libs?

I was wondering if you have any solution to the edge case where a full node maxed out his connections (nMaxConnections usually at 125). Are you able to validate it in any other way, or do you simply ignore that edge case?

From my experiments with Satoshi:0.15.0.1, calling getaddr() immediately after handshake() always returns only the single node you are connected to. This is because the handshake causes the remote node to send an "addr" message containing just information about itself, which is then retrieved by getaddr() instead of the actual response to the "getaddr" message that it sends. To demonstrate this, call handshake() and then get_messages().

Hence the first "addr" message received after the handshake should be discarded to ensure you're using the actual response to "getaddr" for the list of nodes.

Clicking the "Check Node" button on the bottom of the page reports that the node associated with this IP address is up; however, at the top of the page, "Node Status" says it isn't.

If you look at the user agents tab its showing "/BitcoinUnlimited:1.0.1.3/" as 359 nodes. Now click on that string so that it is run as a search in the search box. I see 515 nodes.

https://www.supload.com/SkMwxw9ag

I have been working for a day figure out what to change if I want bitnodes to work with altcoins (coin based on Bitcoin source code). Can you specify all the changes needed?

Thanks for your very good work

1. Is it possible to access the block propagation delay (not daily but for each block) during one or more months? In fact, the charts provide information for a day or month with just one value.

2. In addition, is it possible to access 90% or more for block propagation?

by regards

Up until today the crawler worked for the Bitcoin Cash network so that people could see Bitcoin Cash as well as Bitcoin nodes.

Bitcoin Cash's latest node software has transitioned over to using a different message magic number now, so as to be totally separate from Satoshi nodes.

As of today all new Bitcoin ABC 0.16.2 nodes will no longer be seen by your crawler.

Are there plans to fix this and add support back for NODE_CASH nodes?

I can also submit a PR to fix this if you like.

The leaderboard is including nodes that have not been online for almost a month.

Hello, so I'm trying to install on CentOS 7 directly from:

curl https://bitnodes.earn.com/install-full-node.sh | sh

I'm getting the following error (last line):

Downloading Bitcoin Core binaries..
Checksum passed: bitcoin-0.17.0.1.tar.gz (6ccc675ee91522eee5785457e922d8a155e4eb7d5524bd130eb0ef0f0c4a6008)
Signature passed: Signed checksum.asc verified successfully!

Installing Bitcoin Core v0.17.0.1
Cannot find files to install.

When I ls bitcoin-core I get:

total 0
drwxrwxr-x 2 gold gold  6 Nov 28 22:24 bin
drwxr-xr-x 6 gold gold 56 Oct 26 11:53 bitcoin-0.17.0

You see the problem? The version number somehow is missing a 'decimal' if we could call it that.

Going to try to hack the script manually for my own install case, but would be cool to fix that for other CentOS 7 users.

Cheers and keep up the good work!

Both the map (land area) and the nodes have a grayish-bluish color. This works and looks well when you look at the whole map (more nodes near each other, darker colors), but as you zoom in, nodes lessen and one individual node barely stands out against the background.

I think it would be better to be able to easily spot a node, thus suggest giving nodes a different hue and/or making them darker.

Just thought I would report this since I'm not sure this is intended.

My node is a NODE_NETWORK node as well as NODE_GETUTXOS node (services field in version message equals 3)

This is my node: https://getaddr.bitnodes.io/nodes/leaderboard/?q=212.60.121.120

As you can see the SI value is "-". Should it not be 1.0, as the node is a NODE_NETWORK node, although it's also a NODE_GETUTXOS node?

Hi, can you tell me how the check node test at website test the node server connection? It uses the protocol.py code?

It looks like there will be changes to how geo data is accessed. This could possibly affect bitnodes implementation. Please see this article for more information.

https://bitnodes.21.co/nodes/leaderboard/?q=satushi

Shows one of these nodes as having WU and WL as zero, even though it's been up for over 24 hours. This seems incorrect...

btcd (https://github.com/conformal/btcd) is a full node bitcoin implementation written in Go. btcd does two separate writes, one for the header and the other for the payload:

        // Write header.
        n, err := w.Write(hw.Bytes())
        if err != nil {
                totalBytes += n
                return totalBytes, err
        }
        totalBytes += n

        // Write payload.
        n, err = w.Write(payload)
        if err != nil {
                totalBytes += n
                return totalBytes, err
        }
        totalBytes += n

protocol.py fails with a short payload:
if (data_len - HEADER_LEN) < msg['length']:
self.required_len = HEADER_LEN + msg['length']
raise PayloadTooShortError("got {} of {} bytes".format(
data_len, HEADER_LEN + msg['length']))

If I add more debugging:

2014-02-28 15:02:25,683,683.5 DEBUG (connect) Payload: got 24 of 493 bytes:
msg {'checksum': '\xd4\x1d\xf7\xa2', 'length': 469, 'magic_number': '\xf9\xbe\xb4\xd9', 'command': 'inv'}

I believe you need to keep reading until there is no more data instead of erroring there.

Hello,

I try to check this node but I receive invalid IP address 20.188.42.254.
Is it correct?

Thanks

I cannot access to the API via tor request:
torify curl -H "Accept: application/json; indent=4" https://bitnodes.io/api/v1/nodes/x.y.z.x-8333
cloudflare blocks the request.

Hello,

I'm getting this error while running the pcap python process:

Traceback (most recent call last):
  File "pcap.py", line 319, in <module>
    sys.exit(main(sys.argv))
  File "pcap.py", line 313, in main
    cron()
  File "pcap.py", line 257, in cron
    cache.cache_messages()
  File "pcap.py", line 99, in cache_messages
    self.extract_streams()
  File "pcap.py", line 136, in extract_streams
    pcap_reader = dpkt.pcap.Reader(pcap_file)
  File "/path-to-home/.virtualenvs/bitnodes/local/lib/python2.7/site-packages/dpkt/pcap.py", line 257, in __init__
    raise ValueError('invalid tcpdump header')
ValueError: invalid tcpdump header

Did you ever encounter that error? And do you have any clues on how to solve it?

Thanks :)

Could you add some way, even if it's just by mouse over, to know the percentage of user-agents?

Would it also be possible to collect client version number stats? I think that could be very useful metric to know how up to date the network is.

This link did not work in the email:
https://bitnodes.io/nodes/{torv3}.onion-8333/detach-user/?email={email}