Display list of transactions for a given address about btc-rpc-explorer

[janoside]

I like the ElectrumX server idea.

Thinking out loud...If I go in that direction, I'll want to minimize trust placed in those servers. I think a pretty good middle-ground could be achieved with:

1. Only query txids for a given address from Electrum, then use the txids to get transaction data from the app's connected node.
2. Support configuring multiple electrum servers and corroborating the txids-for-address results among them.

I still like the idea that the app could support a few methods of displaying address details, but I like the feel of this one as a first option that I'll try to tackle soon. Thanks for the suggestion.

[TheHolyRoger]

I think personally I'm going to attempt to scan addresses into a db with nedb, will let you know how i get on!

[janoside]

@louiseth1 Sorry about this. The whole thing feels like a bit of a mess at the moment and I'm trying to get back to it soon. This really should've been done in a feature branch too...hindsight. Anyway, I believe the code, as is, requires a newer version of node (10+) to deal with the Promise.all().catch().finally. I'd like to remove that requirement and intend to do so soon. Secondly, the format for electrumXServers would be in your example:

electrumXServers:[ {host:"electrum.chainhost.io", port:50002}, {host:"electrum-server.ninja", port:50002} ]

[askz]

Hi, any work in progress on this?

[janoside]

Still researching the best way to go about implementing address pages. This issue was the first thought I had, but it may not be the best. Out of curiosity: do you run a node with addrindex?

[askz]

Not atm. But I can do it fast, I run some small blockchains for test purposes

[TheHolyRoger]

Have you got any other ideas for bitcoind without addrindex?

[janoside]

The idea floating in my head, but that I've been too busy to research is to interact with the node in a fashion similar to a lite wallet. I believe that an implementation in that vein would involve sending/receiving bloom-filtered P2P messages to the target node and building a list of transactions on the fly.

Again, since a design goal of this project is to be database-free, there will almost certainly be a tradeoff here: spending more time building lists of relevant transactions instead of storing such lists in a database.

Thoughts on this idea?

[askz]

Interesting idea you got here.
It would in fact take much longer for each new address, also a cache system would be appreciable.

I'll try in the next days to implement addrindex

[TheHolyRoger]

Considering addrindex is not officially implemented in Bitcoin Core or any of the Core node software for most altcoins, I think it would be best avoided.

I won't be running a node with addrindex on my network given the implications of keeping addrindex up to date with Core releases.

This explorer is great and I love that it's database free, but I think an optional database would be best to cache addresses.

Happy to lend a hand to help out with this!

If you go down the addrindex route I'll have to switch to Bitcoin ABE though!

[askz]

Maybe we can instead rely on a much lighter database like sqlite ?

[askz]

I am for the addrindex because, the coin and its asset chain I use (komodo, KMD) implemented addrindex, spentindex etc.

[janoside]

Thanks for the thoughts/conversation. I agree with you @TheHolyRoger - the lack of addrindex support means I definitely don't want addrindex to be the solution to this problem. However, @askz, I'm leaning toward a flexible/configurable solution where addrindex could be used if desired.

[volbil]

Any luck @TheHolyRoger :) ?

[TheHolyRoger]

Not yet, I've used bitcoin-abe for now.

I started it, but not sure when I'll get round to doing more on it

[karelbilek]

Look at indexd if you have time

https://github.com/bitcoinjs/indexd

[karelbilek]

This is not possible without address index, sorry. The question is how to make it simple.

[karelbilek]

For interest - I am from Trezor, we are having these issues too.

We used to run Insight/bitcore. insight/bitcore does an unholy thing and adds an address index INTO THE SOURCE CODE - so all insight instances actually run a modified version of bitcoin/altcoin. That is a mess to maintain.

We are now migrating to blockbook - https://github.com/trezor/blockbook - but I dont think it is the right solution for this project at all.

[geek-at]

@karel-3d the "easiest" way to keep track of all balances is to run a full node and using a .dat file parser to extract transactions and store them in a database. MongoDB or Redis would work for example.

To collect all new blocks and transactions going forward you can setup a webhook for new blocks

[karelbilek]

@chrisiaut nope, you still need to deal with orphaned blocks so you don't have the tx twice for example

so in the end you will end up doing similar stuff as indexd, or insight/bitcore, or blockbook.

[geek-at]

@karel-3d sure you can use tools like fast-dat-parser to get the longest chain and then parse it.

[karelbilek]

You don't want to parse whole chain and re-save it again to database on every block reorg

…

On Wed, Jul 25, 2018, 10:41 Christian Haschek ***@***.***> wrote: @karel-3d <https://github.com/karel-3d> sure you can use tools like fast-dat-parser <https://github.com/bitcoinjs/fast-dat-parser> to get the longest chain and then parse it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGZ8cPCaoM5SEuodRunTo70okoFxzU2ks5uKC8kgaJpZM4PDLIu> .

[frndxyz]

@janoside but if you used bloom-filtered P2P messages old altcoin not support by explorer. because i see that old coin not support bip37.

[martinschwarz]

What about querying local or public Electrum servers (electrumx)? Many fullnode users already use electrum to connect Trezor to their fullnode.

[askz]

I like the idea too ! Don't know why we didn't thought about it before.

Maybe we can contribute to add HTTP(S)/WebSocket support to ElectrumX server, so the client will directly query the txids and send it to the explorer. So the explorer server doesn't get overloaded.
Or just implement a cache with invalidation etc.

What do you think?

EDIT:
We actually forked an electrum client for node, to add connection persistence : https://github.com/Monaize/node-electrum-client/tree/electrumx-persistence

[janoside]

All, please let me know what you think about the current implementation. As discussed, I went with trust-minimized ElectrumX querying as a first implementation. You can see the feature in action at the link below. It's currently using the "electrum-client" npm package for querying Electrum - I still need to switch to the forked, persistent-connection version @askz referred to (@askz Is that fork available via npm?)

https://btc.chaintools.io/address/3NPGpNyLLmVKCEcuipBs7G4KpQJoJXjDGe

Edit - After deploying to the demo site the electrum connections hung and I went ahead and switched to use the persistent-connection fork of electrum-client. We'll see how it fares on the demo site.

[bitlewis]

I saw you were working on this today and did see it live on your demo site.

Seems to work well and all the relevant information appeared. However, when I refresh from time to time I see sometimes the transaction informations are not displayed on the address page.

[janoside]

@louiseth1 The sometimes-blank tx data is due to the socket connection to ElectrumX timing out and not reconnecting properly. I've been periodically restarting the demo site as I push new code and the tx data is visible for a while after each restart.

Once I get this socket issue "addressed", I'll be closing this issue...finally.

Thanks for the feedback.

[bitlewis]

Thanks for the explanation. I'll then wait until you fixed it to update on my end.

I did try to pull it but had issues building it.

[askz]

@janoside we didnt deployed it on npm because we wanted to wait the PR to be accepted, but the maintainer seems.... busy/dead.

[bitlewis]

I tried deploying fresh today. I had to downgrade my npm version to be able to install the electrum-client. (Seems to have installed properly.)

Only had it working using npm 3.10.10. Newer versions I tried wouldn't work.

Otherwise everything installed fine. Could run the npm build aswell, however having an issue on the address pages (nothing in the logs appeared though):

Failed to load address 1HsRTMvG6fjnqJtrqKQJFa9qxWtnYcFQfk (TypeError: Promise.all(...).catch(...).finally is not a function)

I then tried adding the electrum addresses but it seems I can't get the format right because it shows as undefined in the running log.

Log output shows some issues:
[root@explorer btc-rpc-explorer]# node bin/www Running RPC Explorer for Bitcoin Connecting via RPC to node at 72.10.166.164:8332 Connecting to ElectrumX Server: undefined:undefined OnError:Error: connect ECONNREFUSED 127.0.0.1 Error 137rg023xx7gerfwdd: Error: connect ECONNREFUSED 127.0.0.1, when trying to connect to ElectrumX server at undefined:undefined Error 120387rygxx231gwe40: Error: connect ECONNREFUSED 127.0.0.1 Error 31207ugf4e0fed: Error: connect ECONNREFUSED 127.0.0.1, while initializing ElectrumX API OnClose:undefined
Could you indicate how you would format the following array in config.js?
electrumXServers:[ "electrum.chainhost.io:50002", "electrum-server.ninja:50002" ],

[bitlewis]

Got it, updated the servers config line and it worked just fine.

Got an issue because app couldnt find coinConfig.baseCurrencyUnit.multiplier. The value in the config file was CurrencyUnits, I added an array named baseCurrencyUnit with same values to fix the issue I've had.

Everything else worked fine aside from the Balance of the address which displays 0. Haven't seen any errors in console.log or the nodejs stdout. Tried reading the code but not that familiar with node so couldn't figure it out.