metamask / eth-block-tracker

A JS module for keeping track of the latest Ethereum block by polling an ethereum provider

@metamask/eth-block-tracker

This module walks the Ethereum blockchain, keeping track of the latest block. It uses a web3 provider as a data source and will continuously poll for the next block.

Installation

yarn add @metamask/eth-block-tracker

or

npm install @metamask/eth-block-tracker

Usage

const createInfuraProvider = require('@metamask/eth-json-rpc-infura');
const { PollingBlockTracker } = require('@metamask/eth-block-tracker');

const provider = createInfuraProvider({
  network: 'mainnet',
  projectId: process.env.INFURA_PROJECT_ID,
});
const blockTracker = new PollingBlockTracker({ provider });

blockTracker.on('sync', ({ newBlock, oldBlock }) => {
  if (oldBlock) {
    console.log(`sync #${Number(oldBlock)} -> #${Number(newBlock)}`);
  } else {
    console.log(`first sync #${Number(newBlock)}`);
  }
});

API

Methods

new PollingBlockTracker({ provider, pollingInterval, retryTimeout, keepEventLoopActive, usePastBlocks })

• Creates a new block tracker with provider as a data source and pollingInterval (ms) timeout between polling for the latest block.
• If an error is encountered when fetching blocks, it will wait retryTimeout (ms) before attempting again.
• If keepEventLoopActive is false, in Node.js it will unref the polling timeout, allowing the process to exit during the polling interval. Defaults to true, meaning the process will be kept alive.
• If usePastBlocks is true, block numbers less than the current block number can used and emitted. Defaults to false, meaning that only block numbers greater than the current block number will be used and emitted.

getCurrentBlock()

Synchronously returns the current block. May be null.

console.log(blockTracker.getCurrentBlock());

async getLatestBlock()

Asynchronously returns the latest block. if not immediately available, it will fetch one.

async checkForLatestBlock()

Tells the block tracker to ask for a new block immediately, in addition to its normal polling interval. Useful if you received a hint of a new block (e.g. via tx.blockNumber from getTransactionByHash). Will resolve to the new latest block when done polling.

Events

latest

The latest event is emitted for whenever a new latest block is detected. This may mean skipping blocks if there were two created since the last polling period.

blockTracker.on('latest', (newBlock) => console.log(newBlock));

sync

The sync event is emitted the same as "latest" but includes the previous block.

blockTracker.on('sync', ({ newBlock, oldBlock }) =>
  console.log(newBlock, oldBlock),
);

error

The error event means an error occurred while polling for the latest block.

blockTracker.on('error', (err) => console.error(err));

Contributing

Setup

• Install Node.js version 16 or greater
  • If you are using nvm (recommended) running nvm use will automatically choose the right node version for you.
• Install Yarn v1
• Run yarn setup to install dependencies and run any requried post-install scripts
  • Warning: Do not use the yarn / yarn install command directly. Use yarn setup instead. The normal install command will skip required post-install scripts, leaving your development environment in an invalid state.

Testing and Linting

Run yarn test to run the tests once. To run tests on file changes, run yarn test:watch.

Run yarn lint to run the linter, or run yarn lint:fix to run the linter and fix any automatically fixable issues.

Release & Publishing

The project follows the same release process as the other libraries in the MetaMask organization. The GitHub Actions action-create-release-pr and action-publish-release are used to automate the release process; see those repositories for more information about how they work.

1. Choose a release version.

   • The release version should be chosen according to SemVer. Analyze the changes to see whether they include any breaking changes, new features, or deprecations, then choose the appropriate SemVer version. See the SemVer specification for more information.

2. If this release is backporting changes onto a previous release, then ensure there is a major version branch for that version (e.g. 1.x for a v1 backport release).

   • The major version branch should be set to the most recent release with that major version. For example, when backporting a v1.0.2 release, you'd want to ensure there was a 1.x branch that was set to the v1.0.1 tag.

3. Trigger the workflow_dispatch event manually for the Create Release Pull Request action to create the release PR.

   • For a backport release, the base branch should be the major version branch that you ensured existed in step 2. For a normal release, the base branch should be the main branch for that repository (which should be the default value).
   • This should trigger the action-create-release-pr workflow to create the release PR.

4. Update the changelog to move each change entry into the appropriate change category (See here for the full list of change categories, and the correct ordering), and edit them to be more easily understood by users of the package.

   • Generally any changes that don't affect consumers of the package (e.g. lockfile changes or development environment changes) are omitted. Exceptions may be made for changes that might be of interest despite not having an effect upon the published package (e.g. major test improvements, security improvements, improved documentation, etc.).
   • Try to explain each change in terms that users of the package would understand (e.g. avoid referencing internal variables/concepts).
   • Consolidate related changes into one change entry if it makes it easier to explain.
   • Run yarn auto-changelog validate --rc to check that the changelog is correctly formatted.

5. Review and QA the release.

   • If changes are made to the base branch, the release branch will need to be updated with these changes and review/QA will need to restart again. As such, it's probably best to avoid merging other PRs into the base branch while review is underway.

6. Squash & Merge the release.

   • This should trigger the action-publish-release workflow to tag the final release commit and publish the release on GitHub.

7. Publish the release on npm.

   • Be very careful to use a clean local environment to publish the release, and follow exactly the same steps used during CI.
   • Use npm publish --dry-run to examine the release contents to ensure the correct files are included. Compare to previous releases if necessary (e.g. using https://unpkg.com/browse/[package name]@[package version]/).
   • Once you are confident the release contents are correct, publish the release using npm publish.

eth-block-tracker's Issues

When destroying a PollingBlockTracker, unset the last timer created during the polling loop

Failure to do so is causing "Jest did not exit one second after the test run has completed" in the tests for eth-json-rpc-middleware. See MetaMask/eth-json-rpc-middleware#156.

Use Jest's fake timers instead of a custom stub

When rewriting the tests in Jest, I used a custom stub for setTimeout / clearTimeout instead of using Jest's fake timers. This turned out to be easier to use at the time because of the way that we use setTimeout in PollingBlockTracker. While writing those tests, I ended up adding some custom events that proved to be useful, one of which is _waitingForNextIteration. Now that this exists, I believe that this is the key to using Jest's fake timers.

Migrate to Yarn

Switch the project to Yarn

Move babelify to dev dependency

Is there any reason why babelify is dependency (not dev-dependency)? Our package ledger-wallet-provider (indirectly) depends on eth-block-tracker and thus on it depends on babelify, babel-core etc. which is clearly wrong to me.

cc: @banciur

Use shared ESLint config

The project should use the shared ESLint config

Use the `v8` coverage strategy instead of `babel`

When using the v8 coverage library built into Node 12.x (which we still test against), we are getting less than 100% coverage, but when using the babel strategy, we get 100%. This problem is fixed in Node >= 14.x. This leaves us to conclude that there are some bugs (or at least non-trivial differences in behavior) between Node v12's v8 and babel. So at the moment, we use babel. But when we drop v12, we should be able to switch to v8.

Figure out a way to completely stop a polling block tracker in tests

Since the block tracker runs out of band, it has the potential to wreak havoc on tests, especially if we are mocking requests using Nock and/or time using fake timers. We have to be very careful in these situations that requests to the provider do not start in one test and finish in another test.

We might think that we can stop the block tracker at the end of each test to ensure that this doesn't occur, but this doesn't always work.

First, it's important to understand how the block tracker works:

• The block tracker is started automatically whenever the latest or sync event is listened to (these events are aliases for each other).
• In the case of the polling block tracker, which is used much more commonly than the subscribe block tracker, a loop is started.
• The method that starts this loop is called _synchronize, and although it is an async method, it is never awaited. That means the loop happens out of band.
• Inside this loop, the block tracker makes a request for the latest block, and then it waits for a predetermined amount of time, using setTimeout wrapped in a promise. When the promise resolves, the loop continues.
• The block tracker is stopped automatically when the last instance of latest/sync is removed. The most common way this happens is when using getLatestBlock, where the latest event is listened to and is then removed once the event occurs.
• The block tracker can also get stopped manually via the destroy method, which we employ in tests.

We mentioned above that stopping the block tracker doesn't work as expected, but why? There are really two problems here:

1. The block tracker doesn't get immediately stopped. Whether the block tracker is automatically or manually stopped, the _maybeEnd method in BaseBlockTracker is called. This method does a few things, but it will primarily set the _isRunning property to false. As the loop in PollingBlockTracker checks to see if _isRunning is true to determine whether to keep iterating, once it is set to false, the loop should stop on the next iteration. The problem is that if we are faking timers in a test, this never actually happens, because the setTimeout call that occurs at the end of an iteration will never occur. What ends up happening in a test suite, then, is that all of these block tracker objects get created and started but never ended, so they're just sitting around in memory along with all of the pending setTimeout calls. These need to be cleared in order to guarantee that no additional requests are made that we haven't accounted for. Also, if the block tracker is in the middle of making a request when it's stopped, that request will go through. So ideally, we need to not only cancel the setTimeout call via clearTimeout but we also need to use an AbortController to cancel the pending request.
2. When the block tracker is automatically stopped, we have no way of waiting until it's really stopped. This is because the stopping occurs via _onRemoveListener. Although it calls _maybeEnd and _maybeEnd is an async method, it cannot await this method, because _onRemoveListener is called when the removeListener event occurs, and events aren't handled asynchronously. This makes sense from an event perspective, but it's a pretty glaring design flaw. There's really no reason to use EventEmitter for the block tracker, as we only use two one event. It would be much easier if instead of having consumers subscribe to the latest/sync event they just call .onLatest or something like that. That way we wouldn't also have to do things like override the removeAllListeners method to ensure that consumers can't remove our internal events.

Move to MetaMask org

So the rest of the team can more easily contribute to it. I should probably do the same with things like eth-sig-util.

[email protected] fails on subscriptions

Version 3.0.1 of eth-block-tracker fails when handling new data from providers that support subscriptions. New notification IDs are checked against an ID that's cached when setting up the subscription, but the notification ID is always undefined because the wrong property is being targeted.

Question: is SubscribeBlockTracker safe to use?

I noticed the SubscribeBlockTracker while digging around packages trying to lower the number of eth_getBlockByNumber requests in web3-provider-engine. Is it safe to use? Seems to be working. However it's not exported, nor mentioned anywhere in the docs.

Tagging @jamespic as he seems to have implemented it 2 years ago - hope you don't mind the ping.

Update Node version

CI is currently using 7.7.3, we should update the project to use the latest LTS

PollingBlockTracker - encountered an error while attempting to update latest block:

I am creating my dapp through walletconnect and infura when i use walletconnect web3provider and pass infura id to initalize the web3provider this error starts to appear and it sends so many requests to infura endpoint more than 100k in few hours and exceed the limits of requests of infura and it make difficult to further test and develop the dapp. The error is occuring in poling .js file while fetching the latest block. I have no clue what to do. I have not post any query on github so if any thing left from my side i can explain further thanks.

const provider = new WalletConnectProvider({

      infuraId: "8031b681fe9f440ba9dedc43c6d3e780",
      connector: connector,
      // chainId: connector.chainId,
      qrcode: false,
    });

poling.js

async _updateLatestBlock () {
    // fetch + set latest block
    console.log("Updating latest Block1");
    const waitFor = delay => new Promise(resolve => setTimeout(resolve, delay));
    await waitFor(5000);
    const latestBlock = await this._fetchLatestBlock()
    this._newPotentialLatest(latestBlock)
  }

i tried to tackle by myself by adding two lines of setTimeout Thanks

Consider using WS provider instead of polling

Both Geth and Parity support eth_subscribe method
https://github.com/ethereum/go-ethereum/wiki/RPC-PUB-SUB
https://github.com/paritytech/parity/wiki/JSONRPC-Eth-Pub-Sub-Module

It can come in handy for this module and eliminate the need to do polling

Provider type exported by this package should be compatible with provider type exported by `eth-json-rpc-middleware`

It's very common to see this in our code:

import { PollingBlockTracker, Provider } from 'eth-block-tracker';
import { JsonRpcEngine } from 'json-rpc-engine';
import { providerFromEngine } from '../src';

const engine = new JsonRpcEngine();
const provider = providerFromEngine(engine);

const blockTracker = new PollingBlockTracker({
  provider: provider as Provider,
});

We shouldn't have to typecast the provider like this.

Not able to install

Hey,

It seems there is some issue in installation. It has been installed locally as well as globally (to ensure it works in any case). Still saying it is not found.

My-Shell$ npm install eth-block-tracker
/Users/XXXXX/Desktop/webapp
└── [email protected] 

My-Shell$ node main.js 
module.js:471
    throw err;
    ^

Error: Cannot find module 'eth-block-tracker'
    at Function.Module._resolveFilename (module.js:469:15)
    at Function.Module._load (module.js:417:25)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/Users/Toshendra/Desktop/webapp/main.js:3:22)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)

The content of the main.js is:

const HttpProvider = require('ethjs-provider-http')
const BlockTracker = require('eth-block-tracker')

const provider = new HttpProvider('https://mainnet.infura.io/<my unique code>')
const blockTracker = new BlockTracker({ provider })
blockTracker.on('block', console.log)
blockTracker.start()

What might have gone wrong??

Even Runkit is showing not found.

https://npm.runkit.com/eth-block-tracker?t=1497419620361

Convert tests from Tape to Jest

Error while installing package

When I try to install the package via: npm i async-eventemitter

I recieve the following error:

npm ERR! code ENOLOCAL
npm ERR! Could not install from "node_modules\eth-tx-summary\node_modules\eth-block-tracker\async-eventemitter@github:ahultgren\async-eventemitter#fa06e39e56786ba541c180061dbf2c0a5bbf951c" as it does not contain a package.json file.

Anyone an idea how to fix this issue?

dogmi

1606938044258990275541962092341162602522202993782792835301376

Consider: Exclude `bundle.js` from git repo?

We have a pre-publish hook for building, so maybe we don't need the build in the repo. Thoughts?

w

a4ea25033f16ad4b076185df7e95151c1e69fa263326148c9cefbbedd28552cc

Endless failed request polling loop

The polling block tracker has been observed being stuck in an endless loop (e.g. MetaMask/metamask-extension#17040).

I was able to reproduce this scenario by using MetaMask extension to switch to a network that will fail to respond to requests. This results in an endless loop of failed requests, even after switching away (i.e. even after there are no more listeners). In the debugger I confirmed that the only active listeners were internal to the block tracker itself (specifically this listener).

I have a no loss/ no gain contract on my transaction?

No loss / no gain on a eth transaction resulting in a long overdue contract execution what does this mean is my eth gone?

Add CircleCI config

This project should run CircleCI on builds & PRs

Adapt to EIP-1193 provider changes

After SafeEventEmitterProvider is updated to support EIP-1193 and a new version of @metamask/eth-json-rpc-provider is released, we should adapt to the changes:

• We should bump @metamask/eth-json-rpc-provider to rely on the new changes.
• At that point, calling sendAsync will be deprecated; we should use request instead.
  • There are only a couple places in the implementation where we are using sendAsync.
  • However, there are ~20 references in the tests. In particular, the tests make use of a provider whose sendAsync method is mocked. We need to update the helper code to mock request instead.

Feature: detect chain reorgs

MetaMask State logs - v3.5.0 (787)

Check for and tolerate testrpc "block out of range" errors

When querying a block by number that is in the future, most clients return null, but this is a recent change.

TestRPC still returns an error, and so this block tracker is currently incompatible with testrpc.

While this has been reported to testrpc, and I've written a fix for testrpc, until users update their testrpc, they will find metamask incompatible with their old versions, unless we get eth-block-tracker tolerant of that error before we push this into production.

Add a `destroy` method so that the new NetworkController can use it in tests

When writing tests for the shared NetworkController that we want to create, we need a way to "shut down" the polling block tracker so that asynchronous operations are properly ended before the tests end.

sluggish block notifications

Here's a quick script that demonstrates that the eth-block-tracker is not keeping up with the current block. Is it possible this causes d'apps depending on metamask to receive filter events significantly delayed from transaction/block events?

const HttpProvider = require('ethjs-provider-http');
const Eth = require('ethjs-query');
const BlockTracker = require('eth-block-tracker');

const provider = new HttpProvider('https://kovan.infura.io/bOyWfPGcs8jj2g9UXNYr');
const eth = new Eth(provider);

const blockTracker = new BlockTracker({ provider })
blockTracker.on('block', block => {
  console.log("tracker: " + parseInt(block.number) + "\n")
});
blockTracker.start()

var current;

function check() {
  eth.blockNumber().then(number => {
      var string = number.toString();
      if (string !== current) {
        console.log("direct:" + string + "\n");
        current = string;
      }
    })
    .then(() => setTimeout(check, 1000));
}
check();

polling does not continue if created offline

when a block tracker is created offline and fails to fetch it throws an error in the form of a uncaught promise. This stops polling and does not resume polling this can be an issue the only way it seems to resume polling is if you the stop and start it again when back online.

Scope this package under our NPM namespace

This package should be called @metamask/eth-block-tracker to fit our other libraries.

Question: How to install v4 from npm?

What is the recommended way to install v4? It seems it's still not published on npm, so should I use this github repo as dependency?
Is v4 suitable for production? If no, is v3 suitable for production? v2 seems to be suitable.

Eth-Block-Tracker fails when used with a provider created by Web3.js beta 55

I've made a reproduction repo here with an Infura provider as well as a Ganache provider.

I suspect it has something to do with the way Eth-Block-Tracker expects the provider object to behave.

Upgrade to newer versions of ESLint config packages

The latest release of our ESLint config packages is 11.1.0, but this repo is currently using v9. We should upgrade these packages to take advantage of our type-aware TypeScript rules, among other changes.

Add .nvmrc file

The project should have an nvmrc file

ES5 dist files contain ES6 code

e.g. eth-block-tracker/dist/es5/index.js

line 10: class RpcBlockTracker extends EventEmitter {

This can cause problems for es5 code pulling in eth-block-tracker as a dependency and also for create-react-app projects in the build phase.

Generating lots of redundant calls

My app uses the ZeroClientProvider to connect to infura and query the transaction logs; initialization logic: https://github.com/channel/channel.github.io/blob/master/app/src/scripts/initialize.jsx#L27

The trouble I'm facing is that for each web3 LogFilter.get() request I make, the block tracker enqueues a getLogs request that never stops polling. As time goes on the amount of data being fetched grows and grows. You can see the pattern in the network tab at https://channel.github.io

My hack solution is to add self._blockTracker.stop() right after this line: https://github.com/MetaMask/provider-engine/blob/master/index.js#L37

I figure there's something off with the way requests are being queued up - not sure if this is the right place to report the issue but figured it was worth a shot.

I've attached a screenshot showing the tracker doing it's polling, and then (while the page is idle) making a bunch of getLogs requests, presumably once a new block comes in:

Compile error using craco in react app for eth-block-tracker

The is the following error I got. I've tried every trick under the sun and I can't get it to compile.
There seems to to be a compatibility or resolution issue between these packages.
Especially the latest ones.

Does anyone have working versions?
in package.json for metamask/utils eth-block-tracker and @coinbase/wallet-sdk

Failed to compile.

./node_modules/@coinbase/wallet-sdk/node_modules/eth-block-tracker/dist/logging-utils.js
Module not found: Can't resolve '@metamask/utils' in '/Users/mac/Documents/LiquidCrypto/landing-cms/landing/node_modules/@coinbase/wallet-sdk/node_modules/eth-block-tracker/dist'

This is my package.json

{
  "name": "crosswise_landing_v2",
  "version": "0.1.0",
  "private": true,
  "dependencies": {
    "@apollo/client": "^3.4.16",
    "@apollo/react-hooks": "^4.0.0",
    "@binance-chain/bsc-connector": "^1.0.0",
    "@coinbase/wallet-sdk": "^3.6.6",
    "@craco/craco": "^6.4.4",
    "@emotion/react": "^11.10.4",
    "@emotion/styled": "^11.10.4",
    "@ethersproject/providers": "5.5.2",
    "@metamask/eth-sig-util": "^5.1.0",
    "@metamask/utils": "^4.0.0",
    "@mui/icons-material": "^5.10.3",
    "@mui/material": "^5.10.5",
    "@reduxjs/toolkit": "^1.9.5",
    "@testing-library/jest-dom": "^5.11.4",
    "@testing-library/react": "^11.1.0",
    "@testing-library/user-event": "^12.1.10",
    "@transak/transak-sdk": "^1.3.0",
    "@web3-react/core": "6.1.9",
    "@web3-react/fortmatic-connector": "^6.1.6",
    "@web3-react/injected-connector": "^6.0.7",
    "@web3-react/portis-connector": "^6.2.11",
    "@web3-react/walletconnect-connector": "6.2.4",
    "@web3-react/walletlink-connector": "^6.2.14",
    "apollo-boost": "^0.4.9",
    "axios": "^0.27.2",
    "eth-block-tracker": "^7.0.1",
    "graphql": "^15.8.0",
    "immer": "^10.0.1",
    "moment": "^2.29.4",
    "react": "^17.0.2",
    "react-copy-to-clipboard": "^5.1.0",
    "react-csv": "^2.2.2",
    "react-dom": "^17.0.2",
    "react-ga": "^3.3.1",
    "react-redux": "^8.0.5",
    "react-router-dom": "^6.3.0",
    "react-scripts": "4.0.3",
    "typescript": "^4.8.3",
    "web-vitals": "^1.0.1",
    "web3": "^1.8.0"
  },
  "scripts": {
    "start": "craco start",
    "build": "craco build",
    "test": "craco test",
    "eject": "react-scripts eject"
  },
  "eslintConfig": {
    "extends": [
      "react-app",
      "react-app/jest"
    ]
  },
  "browserslist": {
    "production": [
      ">0.2%",
      "not dead",
      "not op_mini all"
    ],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  },
  "devDependencies": {
    "autoprefixer": "^9.8.8",
    "postcss": "^7.0.39",
    "tailwindcss": "npm:@tailwindcss/postcss7-compat@^2.2.17"
  }
}

Emit errors

Currently block-tracker will handle error events itself using console.error.

https://github.com/MetaMask/eth-block-tracker/blob/master/lib/index.js#L48

Previously, with provider-engine, an error event was emitted, which allowed me to hook into this to update the UI to show a connection failure.

Can we add this functionality back? I'll submit a PR.

Build steps should produce `bundle.js` without `async`

The current bundle.js does not have async, but I don't see how that was generated. It is not the npm run prepublish task, like it would seem.

@tmashuang?

const createInfuraProvider = require('eth-json-rpc-infura') const { PollingBlockTracker } = require('eth-block-tracker') const provider = createInfuraProvider({ network: 'mainnet', projectId: process.env.INFURA_PROJECT_ID }) const blockTracker = new PollingBlockTracker({ provider }) blockTracker.on('sync', ({ newBlock, oldBlock }) => { if (oldBlock) { console.log(`sync #${Number(oldBlock)} -> #${Number(newBlock)}`) } else { console.log(`first sync #${Number(newBlock)}`) } })