Coder Social home page Coder Social logo

xpub-scan's People

Contributors

glethuillier avatar meriadec avatar pavlogs avatar vcluzeau-ledger avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

xpub-scan's Issues

Custom derivation paths

Some wallets use different derivation paths.
xpub-scan should support paths such as:

  • m/0'/0' Bitcoin Core harden addresses
  • m/0'/0 Multibit/BRD/breadwallet

It would be nice if this tool could support those hardened paths also.
I imagine this might take considerable effort as you'd essentially be importing a xpriv key?

Improve requests

Currently, the requests are synchronous and the underlying implementation does not properly handle exceptions (no retry, a network error harshly throws an un-recoverable error, no JSON parsing exceptions handling, etc.).

xpub-scan/src/helpers.ts

Lines 13 to 35 in 3b64b7f

function getJSON(url: string, APIKey?: string) {
let headers = {};
if (APIKey !== undefined) {
headers = {
'X-API-Key': APIKey
}
}
const res = request('GET', url, {headers: headers} );
if (res.statusCode !== 200) {
console.log(chalk.red('GET request error'));
throw new Error(
"GET REQUEST ERROR: "
.concat(url)
.concat(", Status Code: ")
.concat(String(res.statusCode))
);
}
return JSON.parse(res.getBody('utf-8'));
}

This basic implementation could be greatly improved by:

  • Performing async requests using a proper module (e.g. axios)
  • Adding a retry mechanism
  • Making the headers more flexible (to handle different external providers)
  • Taking into account the limitations from external APIs (e.g. by ensuring some delay between each request)

From the user perspective, the new implementation should:

  • Not be noticeable when operating optimally (that is: when no network error and no exception): the derived addresses should be sorted by a) address type (legacy, Segwit), then, b) by account number and c) index. They also should be displayed one by one rather than at once, at the end of the whole analysis.
  • When recoverable exceptions occur, a transient error message should be displayed (e.g. "Network issue, retrying...") and be removed once the exception had been properly handled.
  • When an un-recoverable exception occurs, the reason should be clearly explained to the user (e.g. "Network error: authentication issue", "JSON parsing error: {raw response body}")

Comparisons Feature: Option to Show Diffs Only

When performing a comparison between imported operations and actuals ones, all categories of comparison are shown: matches, mismatches, etc.

It can be tedious to find the relevant information in these conditions, especially with thousands of operations.

An improvement of the current implementation could be the creation of an option (e.g., --show-diffs-only) that would only display the discrepancies: mismatches, missing operations, and extra operations.

xpub-scab for BCH throws an error

When I run xpub-scan --currency bch <xpub address> (actual address removed for privacy), I get:

(Data fetched from the default provider)

Active addresses

Scanning Bitcoin Cash addresses...
- scanning external addresses -
Bitcoin Cash    m/0/0       1FTQDiaeLfSNoUuj72ZYB1uDaRHF67pHG3  qz0gla2ugy2y3l3heu4jhukj86mul7t76gewxvu2wr    analyzing...node:internal/process/promises:279
            triggerUncaughtException(err, true /* fromPromise */);
            ^

<ref *1> Error: connect ECONNREFUSED 13.51.66.122:443
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1195:16) {
  errno: -61,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '13.51.66.122',
  port: 443,
  config: {
    transitional: {
      silentJSONParsing: true,
      forcedJSONParsing: true,
      clarifyTimeoutError: false
    },
    adapter: [Function: httpAdapter],
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 0,
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    maxBodyLength: -1,
    validateStatus: [Function: validateStatus],
    headers: {
      Accept: 'application/json, text/plain, */*',
      'User-Agent': 'axios/0.26.1'
    },
    method: 'get',
    url: 'https://rest.bitcoin.com/v2/address/details/bitcoincash:qz0gla2ugy2y3l3heu4jhukj86mul7t76gewxvu2wr',
    data: undefined
  },
  request: <ref *4> Writable {
    _writableState: WritableState {
      objectMode: false,
      highWaterMark: 16384,
      finalCalled: false,
      needDrain: false,
      ending: false,
      ended: false,
      finished: false,
      destroyed: false,
      decodeStrings: true,
      defaultEncoding: 'utf8',
      length: 0,
      writing: false,
      corked: 0,
      sync: true,
      bufferProcessing: false,
      onwrite: [Function: bound onwrite],
      writecb: null,
      writelen: 0,
      afterWriteTickInfo: null,
      buffered: [],
      bufferedIndex: 0,
      allBuffers: true,
      allNoop: true,
      pendingcb: 0,
      constructed: true,
      prefinished: false,
      errorEmitted: false,
      emitClose: true,
      autoDestroy: true,
      errored: null,
      closed: false,
      closeEmitted: false,
      [Symbol(kOnFinished)]: []
    },
    _events: [Object: null prototype] {
      response: [Function: handleResponse],
      error: [Function: handleRequestError],
      socket: [Function: handleRequestSocket]
    },
    _eventsCount: 3,
    _maxListeners: undefined,
    _options: {
      maxRedirects: 21,
      maxBodyLength: 10485760,
      protocol: 'https:',
      path: '/v2/address/details/bitcoincash:qz0gla2ugy2y3l3heu4jhukj86mul7t76gewxvu2wr',
      method: 'GET',
      headers: {
        Accept: 'application/json, text/plain, */*',
        'User-Agent': 'axios/0.26.1'
      },
      agent: undefined,
      agents: { http: undefined, https: undefined },
      auth: undefined,
      hostname: 'rest.bitcoin.com',
      port: null,
      nativeProtocols: {
        'http:': {
          _connectionListener: [Function: connectionListener],
          METHODS: [
            'ACL',         'BIND',       'CHECKOUT',
            'CONNECT',     'COPY',       'DELETE',
            'GET',         'HEAD',       'LINK',
            'LOCK',        'M-SEARCH',   'MERGE',
            'MKACTIVITY',  'MKCALENDAR', 'MKCOL',
            'MOVE',        'NOTIFY',     'OPTIONS',
            'PATCH',       'POST',       'PROPFIND',
            'PROPPATCH',   'PURGE',      'PUT',
            'REBIND',      'REPORT',     'SEARCH',
            'SOURCE',      'SUBSCRIBE',  'TRACE',
            'UNBIND',      'UNLINK',     'UNLOCK',
            'UNSUBSCRIBE'
          ],
          STATUS_CODES: {
            '100': 'Continue',
            '101': 'Switching Protocols',
            '102': 'Processing',
            '103': 'Early Hints',
            '200': 'OK',
            '201': 'Created',
            '202': 'Accepted',
            '203': 'Non-Authoritative Information',
            '204': 'No Content',
            '205': 'Reset Content',
            '206': 'Partial Content',
            '207': 'Multi-Status',
            '208': 'Already Reported',
            '226': 'IM Used',
            '300': 'Multiple Choices',
            '301': 'Moved Permanently',
            '302': 'Found',
            '303': 'See Other',
            '304': 'Not Modified',
            '305': 'Use Proxy',
            '307': 'Temporary Redirect',
            '308': 'Permanent Redirect',
            '400': 'Bad Request',
            '401': 'Unauthorized',
            '402': 'Payment Required',
            '403': 'Forbidden',
            '404': 'Not Found',
            '405': 'Method Not Allowed',
            '406': 'Not Acceptable',
            '407': 'Proxy Authentication Required',
            '408': 'Request Timeout',
            '409': 'Conflict',
            '410': 'Gone',
            '411': 'Length Required',
            '412': 'Precondition Failed',
            '413': 'Payload Too Large',
            '414': 'URI Too Long',
            '415': 'Unsupported Media Type',
            '416': 'Range Not Satisfiable',
            '417': 'Expectation Failed',
            '418': "I'm a Teapot",
            '421': 'Misdirected Request',
            '422': 'Unprocessable Entity',
            '423': 'Locked',
            '424': 'Failed Dependency',
            '425': 'Too Early',
            '426': 'Upgrade Required',
            '428': 'Precondition Required',
            '429': 'Too Many Requests',
            '431': 'Request Header Fields Too Large',
            '451': 'Unavailable For Legal Reasons',
            '500': 'Internal Server Error',
            '501': 'Not Implemented',
            '502': 'Bad Gateway',
            '503': 'Service Unavailable',
            '504': 'Gateway Timeout',
            '505': 'HTTP Version Not Supported',
            '506': 'Variant Also Negotiates',
            '507': 'Insufficient Storage',
            '508': 'Loop Detected',
            '509': 'Bandwidth Limit Exceeded',
            '510': 'Not Extended',
            '511': 'Network Authentication Required'
          },
          Agent: [Function: Agent] { defaultMaxSockets: Infinity },
          ClientRequest: [Function: ClientRequest],
          IncomingMessage: [Function: IncomingMessage],
          OutgoingMessage: [Function: OutgoingMessage],
          Server: [Function: Server],
          ServerResponse: [Function: ServerResponse],
          createServer: [Function: createServer],
          validateHeaderName: [Function: __node_internal_],
          validateHeaderValue: [Function: __node_internal_],
          get: [Function: get],
          request: [Function: request],
          maxHeaderSize: [Getter],
          globalAgent: [Getter/Setter]
        },
        'https:': {
          Agent: [Function: Agent],
          globalAgent: Agent {
            _events: [Object: null prototype],
            _eventsCount: 2,
            _maxListeners: undefined,
            defaultPort: 443,
            protocol: 'https:',
            options: [Object: null prototype],
            requests: [Object: null prototype] {},
            sockets: [Object: null prototype],
            freeSockets: [Object: null prototype] {},
            keepAliveMsecs: 1000,
            keepAlive: false,
            maxSockets: Infinity,
            maxFreeSockets: 256,
            scheduling: 'lifo',
            maxTotalSockets: Infinity,
            totalSocketCount: 1,
            maxCachedSessions: 100,
            _sessionCache: [Object],
            [Symbol(kCapture)]: false
          },
          Server: [Function: Server],
          createServer: [Function: createServer],
          get: [Function: get],
          request: [Function: request]
        }
      },
      pathname: '/v2/address/details/bitcoincash:qz0gla2ugy2y3l3heu4jhukj86mul7t76gewxvu2wr'
    },
    _ended: true,
    _ending: true,
    _redirectCount: 0,
    _redirects: [],
    _requestBodyLength: 0,
    _requestBodyBuffers: [],
    _onNativeResponse: [Function (anonymous)],
    _currentRequest: <ref *2> ClientRequest {
      _events: [Object: null prototype] {
        response: [Function: bound onceWrapper] {
          listener: [Function (anonymous)]
        },
        abort: [Function (anonymous)],
        aborted: [Function (anonymous)],
        connect: [Function (anonymous)],
        error: [Function (anonymous)],
        socket: [Function (anonymous)],
        timeout: [Function (anonymous)]
      },
      _eventsCount: 7,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: false,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: false,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: false,
      socket: <ref *3> TLSSocket {
        _tlsOptions: {
          allowHalfOpen: undefined,
          pipe: false,
          secureContext: SecureContext { context: SecureContext {} },
          isServer: false,
          requestCert: true,
          rejectUnauthorized: true,
          session: undefined,
          ALPNProtocols: undefined,
          requestOCSP: undefined,
          enableTrace: undefined,
          pskCallback: undefined,
          highWaterMark: undefined,
          onread: undefined,
          signal: undefined
        },
        _secureEstablished: false,
        _securePending: false,
        _newSessionPending: false,
        _controlReleased: true,
        secureConnecting: true,
        _SNICallback: null,
        servername: null,
        alpnProtocol: null,
        authorized: false,
        authorizationError: null,
        encrypted: true,
        _events: [Object: null prototype] {
          close: [
            [Function: onSocketCloseDestroySSL],
            [Function],
            [Function: onClose],
            [Function: socketCloseListener]
          ],
          end: [ [Function: onConnectEnd], [Function: onReadableStreamEnd] ],
          newListener: [Function: keylogNewListener],
          connect: [ [Function], [Function], [Function] ],
          secure: [Function: onConnectSecure],
          session: [Function (anonymous)],
          free: [Function: onFree],
          timeout: [Function: onTimeout],
          agentRemove: [Function: onRemove],
          error: [Function: socketErrorListener],
          drain: [Function: ondrain]
        },
        _eventsCount: 11,
        connecting: false,
        _hadError: true,
        _parent: null,
        _host: 'rest.bitcoin.com',
        _readableState: ReadableState {
          objectMode: false,
          highWaterMark: 16384,
          buffer: BufferList { head: null, tail: null, length: 0 },
          length: 0,
          pipes: [],
          flowing: true,
          ended: false,
          endEmitted: false,
          reading: true,
          constructed: true,
          sync: false,
          needReadable: true,
          emittedReadable: false,
          readableListening: false,
          resumeScheduled: false,
          errorEmitted: true,
          emitClose: false,
          autoDestroy: true,
          destroyed: true,
          errored: [Circular *1],
          closed: true,
          closeEmitted: true,
          defaultEncoding: 'utf8',
          awaitDrainWriters: null,
          multiAwaitDrain: false,
          readingMore: false,
          dataEmitted: false,
          decoder: null,
          encoding: null,
          [Symbol(kPaused)]: false
        },
        _maxListeners: undefined,
        _writableState: WritableState {
          objectMode: false,
          highWaterMark: 16384,
          finalCalled: false,
          needDrain: false,
          ending: false,
          ended: false,
          finished: false,
          destroyed: true,
          decodeStrings: false,
          defaultEncoding: 'utf8',
          length: 203,
          writing: true,
          corked: 0,
          sync: false,
          bufferProcessing: false,
          onwrite: [Function: bound onwrite],
          writecb: [Function: bound onFinish],
          writelen: 203,
          afterWriteTickInfo: null,
          buffered: [],
          bufferedIndex: 0,
          allBuffers: true,
          allNoop: true,
          pendingcb: 1,
          constructed: true,
          prefinished: false,
          errorEmitted: true,
          emitClose: false,
          autoDestroy: true,
          errored: [Circular *1],
          closed: true,
          closeEmitted: true,
          [Symbol(kOnFinished)]: []
        },
        allowHalfOpen: false,
        _sockname: null,
        _pendingData: 'GET /v2/address/details/bitcoincash:qz0gla2ugy2y3l3heu4jhukj86mul7t76gewxvu2wr HTTP/1.1\r\n' +
          'Accept: application/json, text/plain, */*\r\n' +
          'User-Agent: axios/0.26.1\r\n' +
          'Host: rest.bitcoin.com\r\n' +
          'Connection: close\r\n' +
          '\r\n',
        _pendingEncoding: 'latin1',
        server: undefined,
        _server: null,
        ssl: null,
        _requestCert: true,
        _rejectUnauthorized: true,
        parser: null,
        _httpMessage: [Circular *2],
        [Symbol(res)]: TLSWrap {
          _parent: TCP {
            reading: [Getter/Setter],
            onconnection: null,
            [Symbol(owner_symbol)]: [Circular *3],
            [Symbol(handle_onclose)]: [Function: done]
          },
          _parentWrap: undefined,
          _secureContext: SecureContext { context: SecureContext {} },
          reading: false,
          onkeylog: [Function: onkeylog],
          onhandshakestart: {},
          onhandshakedone: [Function (anonymous)],
          onocspresponse: [Function: onocspresponse],
          onnewsession: [Function: onnewsessionclient],
          onerror: [Function: onerror],
          [Symbol(owner_symbol)]: [Circular *3]
        },
        [Symbol(verified)]: false,
        [Symbol(pendingSession)]: null,
        [Symbol(async_id_symbol)]: 58,
        [Symbol(kHandle)]: null,
        [Symbol(lastWriteQueueSize)]: 0,
        [Symbol(timeout)]: null,
        [Symbol(kBuffer)]: null,
        [Symbol(kBufferCb)]: null,
        [Symbol(kBufferGen)]: null,
        [Symbol(kCapture)]: false,
        [Symbol(kSetNoDelay)]: false,
        [Symbol(kSetKeepAlive)]: true,
        [Symbol(kSetKeepAliveInitialDelay)]: 60,
        [Symbol(kBytesRead)]: 0,
        [Symbol(kBytesWritten)]: 0,
        [Symbol(connect-options)]: {
          rejectUnauthorized: true,
          ciphers: 'TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!SRP:!CAMELLIA',
          checkServerIdentity: [Function: checkServerIdentity],
          minDHSize: 1024,
          maxRedirects: 21,
          maxBodyLength: 10485760,
          protocol: 'https:',
          path: null,
          method: 'GET',
          headers: {
            Accept: 'application/json, text/plain, */*',
            'User-Agent': 'axios/0.26.1'
          },
          agent: undefined,
          agents: { http: undefined, https: undefined },
          auth: undefined,
          hostname: 'rest.bitcoin.com',
          port: 443,
          nativeProtocols: { 'http:': [Object], 'https:': [Object] },
          pathname: '/v2/address/details/bitcoincash:qz0gla2ugy2y3l3heu4jhukj86mul7t76gewxvu2wr',
          _defaultAgent: Agent {
            _events: [Object: null prototype],
            _eventsCount: 2,
            _maxListeners: undefined,
            defaultPort: 443,
            protocol: 'https:',
            options: [Object: null prototype],
            requests: [Object: null prototype] {},
            sockets: [Object: null prototype],
            freeSockets: [Object: null prototype] {},
            keepAliveMsecs: 1000,
            keepAlive: false,
            maxSockets: Infinity,
            maxFreeSockets: 256,
            scheduling: 'lifo',
            maxTotalSockets: Infinity,
            totalSocketCount: 1,
            maxCachedSessions: 100,
            _sessionCache: [Object],
            [Symbol(kCapture)]: false
          },
          host: 'rest.bitcoin.com',
          servername: 'rest.bitcoin.com',
          _agentKey: 'rest.bitcoin.com:443:::::::::::::::::::::',
          encoding: null,
          singleUse: true
        }
      },
      _header: 'GET /v2/address/details/bitcoincash:qz0gla2ugy2y3l3heu4jhukj86mul7t76gewxvu2wr HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'User-Agent: axios/0.26.1\r\n' +
        'Host: rest.bitcoin.com\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: Agent {
        _events: [Object: null prototype] {
          free: [Function (anonymous)],
          newListener: [Function: maybeEnableKeylog]
        },
        _eventsCount: 2,
        _maxListeners: undefined,
        defaultPort: 443,
        protocol: 'https:',
        options: [Object: null prototype] { path: null },
        requests: [Object: null prototype] {},
        sockets: [Object: null prototype] {
          'rest.bitcoin.com:443:::::::::::::::::::::': [ [TLSSocket] ]
        },
        freeSockets: [Object: null prototype] {},
        keepAliveMsecs: 1000,
        keepAlive: false,
        maxSockets: Infinity,
        maxFreeSockets: 256,
        scheduling: 'lifo',
        maxTotalSockets: Infinity,
        totalSocketCount: 1,
        maxCachedSessions: 100,
        _sessionCache: { map: {}, list: [] },
        [Symbol(kCapture)]: false
      },
      socketPath: undefined,
      method: 'GET',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      path: '/v2/address/details/bitcoincash:qz0gla2ugy2y3l3heu4jhukj86mul7t76gewxvu2wr',
      _ended: false,
      res: null,
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'rest.bitcoin.com',
      protocol: 'https:',
      _redirectable: [Circular *4],
      [Symbol(kCapture)]: false,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype] {
        accept: [ 'Accept', 'application/json, text/plain, */*' ],
        'user-agent': [ 'User-Agent', 'axios/0.26.1' ],
        host: [ 'Host', 'rest.bitcoin.com' ]
      }
    },
    _currentUrl: 'https://rest.bitcoin.com/v2/address/details/bitcoincash:qz0gla2ugy2y3l3heu4jhukj86mul7t76gewxvu2wr',
    [Symbol(kCapture)]: false
  },
  response: undefined,
  isAxiosError: true,
  toJSON: [Function: toJSON]
}

Node.js v17.9.0

nodejs interface

Is there any way to use the scan feature directly via nodejs?

HTML Reports: Add Pagination

When an xpub is associated with an important number of operations, it can be cumbersome to read the generated HTML report. Indeed, the Transactions tab can contain hundreds or thousands of rows...

A CSS pagination could improve the readability of such reports.

Scan also for unused addresses

Can we have an option to include unused addresses (those without balance) to appear on output? That would be a great help if we experience gap limit error.

[Linux/Windows] Special arrow unicode glyphs are not properly rendered

Sent to self operations are associated with the following glyph: (U+2B82).

It is not being properly rendered on Ubuntu (20.04):
image

This glyph is being used here:

xpub-scan/src/display.ts

Lines 98 to 100 in 3b64b7f

const header =
'\ndate\t\t\tblock\t\taddress\t\t\t\t\t\treceived (←) or sent (→) to self (⮂) or sibling (↺)';
console.log(chalk.grey(header));

xpub-scan/src/display.ts

Lines 132 to 150 in 3b64b7f

if (operationType == OperationType.Out_Self) {
// case 1. Sent to the same address
status =
status
.concat(' ⮂');
}
else if (operationType == OperationType.Out_Sibling) {
// case 2. Sent to a sibling address
// (different non-change address belonging to same xpub)
status =
status
.concat(' ↺');
}
else {
// case 3. Sent to external address
status =
status
.concat(' →');
}

The expected output is the following one:
image

From this screen capture, it appears that this glyph is not very well rendered on MacOS too. Therefore, it would be acceptable for the fix to suggest another glyph to evoke the idea of sending to the same address (and that would be different from the glyph related to an address sending to another one belonging to the same xpub, and rendered with ).

It should be noted that both and glyphs are rendered as ? on Windows.

Make Xpub Scan more easily configurable

The configuration options are set in src/settings.ts.

Therefore, each time an end-user configures the tool, it has to be rebuilt.

A more user-friendly approach could be to make it a JSON (or, preferably, a YAML) file automatically parsed (and validated) when the user runs the tool.

Make Xpub Scan more user-friendly

Xpub Scan is not user-friendly (yet): it is a command-line tool that has to be built by the end-user.

Here are some ideas of improvement in this context:

  • The tool could be interfaced with a custom webpage. This webpage would allow the user to enter the xpub, select the actions to perform (full scan, match address, validate CSV file...), and then run the action
  • The dependencies installation and build processes could perhaps be simplified.

On this topic, see also: #14

Optimize the Comparisons Based on Type A CSVs

Xpub Scan does not check the date fields from Type A CSVs.

It could be improved to check the maximum number of Type A CSVs fields. (Or at least ascertain a certain consistency between creation and broadcasting dates).

Handle Aggregated Operations

Type A and Type B CSV files generally handle operations on a TXID basis: in most cases, one row corresponds to one TXID.

As a consequence, Xpub Scan will falsely identify some mismatches because it operates at a more granular level.

An example that speaks for itself: in this screenshot, the imported CSV shows only one operation (on the left) while Xpub Scan identifies two distinct operations (on the right):

Screenshot 2021-03-30 at 19 58 17

The first comparison is labeled as a mismatch, because the amounts differ, and the second one is labeled as a missing operation, because of a missing expected imported operation.

It quickly appears that the imported operation is the aggregation of the two operations identified by Xpub Scan: the imported amount is the sum of the actual amounts (0.00000547 × 2 = 0.00001094). In other words, the result of this comparison should not be labeled as a mismatch.

Therefore, the current implementation could be improved by having the ability to perform aggregated comparisons when required. A specific color could be used in the HTML report to highlight such a match (or even mismatch), allowing to spot at first glance that an aggregated-type comparison has been performed for a given transaction.

Import Type B JSONs

Currently, Xpub Scan can automatically import and analyse type A and type B CSVs.

Type B JSONs can be conveniently generated via a command instead of manually. Making Xpub Scan compatible with this format would then facilitate the analysis of transactions at a CI/CD level.

Generalize support for external providers

The tool only handles two specific external providers: a default (free) one, and a custom (paid) one.

The implementation could be improved by generalizing the support for external providers. Notably, a more robust way of transforming the raw responses into raw transaction and transaction models could be suggested.

Here are some possible checks:

  • Ensure that the API key is valid (i.e. perform one request and check that there is no authorization issue before running the full scan)
  • Validate the contents of the response: is it the expected structure? Are the types valid?

Refactor the code to have some kind of dispatcher to select the relevant provider.

Note that the requests underlying implementation should be improved beforehand.

In this context, see also: #7 (review)

Add coins and networks

The current implementation is limited to Bitcoin mainnet and Litecoin mainnet.

  • Coins based on the same addresses derivation principle could be added
  • Bitcoin and Litecoin networks could include testnet

Strengthen Privacy

The master public key is never sent over the Internet, but its derived addresses are—sequentially.

A solution could strengthen privacy in this regard (not necessarily by avoiding completely any communication to a third-party, at least at this stage).

xpub-scan is crashing with high number of address gaps

It seems some API call restrictions or it can't handle the back to back api calls?

probing address gap...node:internal/process/promises:289

        triggerUncaughtException(err, true /* fromPromise */);
        ^

[UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "AxiosError: Request failed with status code 429".] {
code: 'ERR_UNHANDLED_REJECTION'
}

Node.js v19.1.0

The workaround would be to await the response, something like this

try {
const result = await axios.post(YOUR_URL, {});
} catch (error) {
console.error(error);
}

Add unit tests

The current implementation does not have any unit test:

"test": "echo \"Error: no test specified\" && exit 1"

New unit tests could notably ensure that:

  • The address type (legacy, SegWit, etc.) is associated with the correct derivation
  • Check that the balance associated with a "frozen" xpub is the expected one
  • Exceptions are well handled
  • The address matching mechanism (perfect and partial match) is working as expected
  • The --import feature (automatic validation of imported CSV files) correctly identifies optimal (i.e. no discrepancy) and erroneous (missing transactions, incorrect amount, etc.) CSV files

Sent Transactions: Show All Addresses Belonging to the Xpub

Currently, when several addresses belonging to the xpub have sent funds in the context of a given transaction, only one is processed and displayed.

An improvement would be to handle multiple addresses taking part in a Send transaction:

  • Operations history: display each relevant addresses instead of only one
  • Comparison: compare all addresses instead of just ensuring that at least one is present

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.