Coder Social home page Coder Social logo

solidity-coverage's Introduction

solidity-coverage

Gitter chat npm (tag) CircleCI codecov Hardhat

Code coverage for Solidity testing

coverage example

Requirements

  • Hardhat >= 2.11.0

Install

$ yarn add solidity-coverage --dev

Require the plugin in hardhat.config.js (Hardhat docs)

require('solidity-coverage')

Or, if you are using TypeScript, add this to your hardhat.config.ts:

import 'solidity-coverage'

Resources:

Run

npx hardhat coverage [command-options]

Trouble shooting

Missing or unexpected coverage? Make sure you're using the latest plugin version and run:

$ npx hardhat clean
$ npx hardhat compile
$ npx hardhat coverage

Typescript compilation errors?

$ npx hardhat compile
$ TS_NODE_TRANSPILE_ONLY=true npx hardhat coverage

Weird test failures or plugin conflicts?

# Setting the `SOLIDITY_COVERAGE` env variable tells the coverage plugin to configure the provider
# early in the hardhat task cycle, minimizing conflicts with other plugins or `extendEnvironment` hooks

$ SOLIDITY_COVERAGE=true npx hardhat coverage

Additional Help

Command Options

Option Example Description
testfiles --testfiles "test/registry/*.ts" Test file(s) to run. (Globs must be enclosed by quotes and use globby matching patterns)
sources --sources myFolder or --sources myFile.sol Path to single folder or file to target for coverage. Path is relative to Hardhat's paths.sources (usually contracts/)
solcoverjs --solcoverjs ./../.solcover.js Relative path from working directory to config. Useful for monorepo packages that share settings. (Path must be "./" prefixed)
matrix --matrix Generate a JSON object that maps which mocha tests hit which lines of code. (Useful as an input for some fuzzing, mutation testing and fault-localization algorithms.) More...

* Advanced use

Config Options

Additional options can be specified in a .solcover.js config file located in the root directory of your project.

Example:

module.exports = {
  skipFiles: ['Routers/EtherRouter.sol']
};
Option Type Default Description
skipFiles Array [] Array of contracts or folders (with paths expressed relative to the contracts directory) that should be skipped when doing instrumentation.(ex: [ "Routers", "Networks/Polygon.sol"]) ⚠️ RUN THE HARDHAT CLEAN COMMAND AFTER UPDATING THIS
modifierWhitelist String[] [] List of modifier names (ex: onlyOwner) to exclude from branch measurement. (Useful for modifiers which prepare something instead of acting as a gate.))
mocha Object { } Mocha options to merge into existing mocha config. grep and invert are useful for skipping certain tests under coverage using tags in the test descriptions. More...
measureStatementCoverage boolean true Computes statement (in addition to line) coverage. More...
measureFunctionCoverage boolean true Computes function coverage. More...
measureModifierCoverage boolean true Computes each modifier invocation as a code branch. More...
🖨️ OUTPUT
matrixOutputPath String ./testMatrix.json Relative path to write test matrix JSON object to. More...
mochaJsonOutputPath String ./mochaOutput.json Relative path to write mocha JSON reporter object to. More...
abiOutputPath String ./humanReadableAbis.json Relative path to write diff-able ABI data to
istanbulFolder String ./coverage Folder location for Istanbul coverage reports.
istanbulReporter Array ['html', 'lcov', 'text', 'json'] Istanbul coverage reporters
silent Boolean false Suppress logging output
♻️ WORKFLOW HOOKS
onServerReady* Function Hook run after server is launched, before the tests execute. Useful if you need to use the Oraclize bridge or have setup scripts which rely on the server's availability. More...
onPreCompile* Function Hook run after instrumentation is performed, before the compiler is run. Can be used with the other hooks to be able to generate coverage reports on non-standard / customized directory structures, as well as contracts with absolute import paths. More...
onCompileComplete* Function Hook run after compilation completes, before tests are run. Useful if you have secondary compilation steps or need to modify built artifacts. More...
onTestsComplete* Function Hook run after the tests complete, before Istanbul reports are generated. More...
onIstanbulComplete* Function Hook run after the Istanbul reports are generated, before the coverage task completes. Useful if you need to clean resources up. More...
⚠️ DEPRECATED
configureYulOptimizer Boolean false (Deprecated since 0.8.7) Setting to true should resolve "stack too deep" compiler errors in large projects using ABIEncoderV2
solcOptimizerDetails Object undefined (Deprecated since 0.8.7)) Must be used in combination with configureYulOptimizer. Allows you configure solc's optimizer details. Useful if the default remedy for stack-too-deep errors doesn't work in your case (See FAQ: Running out of stack ).

* Advanced use

Viewing the reports:

  • You can find the Istanbul reports written to the ./coverage/ folder generated in your root directory.

API

Solidity-coverage's core methods and many utilities are available as an API.

const CoverageAPI = require('solidity-coverage/api');

Documentation available here.

Detecting solidity-coverage from another task

If you're writing another plugin or task, it can be helpful to detect whether coverage is running or not. The coverage plugin sets a boolean variable on the globally injected hardhat environment object for this purpose.

hre.__SOLIDITY_COVERAGE_RUNNING === true

Example reports

Funding

You can help fund solidity-coverage development through DRIPS. It's a public goods protocol which helps distribute money to packages in your dependency tree. (It's great, check it out.)

Contribution Guidelines

Contributions are welcome! If you're opening a PR that adds features or options please consider writing full unit tests for them. (We've built simple fixtures for almost everything and are happy to add some for your case if necessary).

Set up the development environment with:

$ git clone https://github.com/sc-forks/solidity-coverage.git
$ yarn

solidity-coverage's People

Contributors

area avatar cag avatar celeduc avatar cgewecke avatar dependabot[bot] avatar e11io avatar frangio avatar fulldecent avatar fvictorio avatar gitter-badger avatar h3ph4est7s avatar hiddentao avatar ilovehackathons avatar itsnickbarry avatar jjgonecrypto avatar jtakalai avatar markus0x1 avatar maxsam4 avatar nslee333 avatar paulrberg avatar pcowgill avatar phiferd avatar pinkiebell avatar remedcu avatar rynobey avatar scnale avatar ukstv avatar vdrg avatar vreturn avatar yxliang01 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

solidity-coverage's Issues

Known issues with valid solidity

Generalising this issue to contain all cases of valid solidity that we know we can't instrument...

These are either going to need edge cases coded, or use of debug_traceTransaction to avoid instrumentation all together.

Hopefully, nothing here would be something realistically used in 'real' solidity contracts, and are just odds and ends for us to look at.

lcov html report points to instrumented files

This bug was introduced in 0.1.9 during by the exec.js refactor. Changed a hardcoded path to use a variable in instrumentation sequence and it's off by a character. This was caught trying to debug a failure to get Coveralls reports at Aragon. Opening a PR and publishing a fix for this shortly.

Does not install correctly using yarn

Unlike npm, yarn does not install binaries of dependencies to ./node_modules/.bin (see here). As a result when we try and run testrpc-sc, it fails. This could be fixed by executing ./node_modules/ethereumjs-testrpc-sc/bin/testrpc rather than ./node_modules/.bin/testrpc-sc. Alternatively, the onus could be on the use a tool like yarn-bin-fix.

Refactor node handling in parse.js

Instead of having a huge list that requires updates, we should just check for the existence of the function in the table before calling it
e.g:

parse[modifier.type] && 
parse[modifier.type](contract, modifier);

Incompatibility with package-lock.json

We changed our path to execute testrpc-sc from the customary /.bin path to ./node_modules/ethereumjs-testrpc-sc/bin/testrpc in order to accommodate yarn in #48 and now package-lock.json has broken pattern with regular npm install and nests the dependency inside the solidity-coverage folder.

Just head over heels in love with this multi-package manager locking mania that has swept code recently.

Likelyhood of avoiding package-lock is zero since its constantly hanging out in the untracked files, driving everyone insane. Likelyhood of avoiding yarn also remote. . . .

Question: decoupling the solidity-coverage from solc version?

What do you think about allowing the user to specify the solc version being used?

The use-case would be a compiler regression that affects contracts, such as this one.

Of course, one can always downgrade/pin the solidity-coverage version, but features developed since then will be unavailable without getting the offending solc version as well.

One way to achieve this without dealing with different NPM packages may be using legacy solc versions, and specifying it in the .solcover.js file (though I have not tried this).

Anyway, let me know if this makes any sense to you :)

Tuple Declaration parsing bug

var (, , , , , sharePrice) = performCalculations();

is not being parsed correctly - statement instrumentation is getting injected in front of the function call.

Prepare for [email protected]

A list of steps from #63:

  • publish an npm version of solidity-coverage with all the testrpc-sc deps pinned to commits instead of master so there's a safe state.
  • work out webpack / ganache-core in a dev branch
  • validate against a few projects
  • make testrpc 4.x new master (4.x is published from #coverage)
  • write some dev instructions for us so we know how to update the webpack build
  • npm publish testrpc-sc --> myrmi / cgewecke
  • publish an npm version of soldity-coverage pinned to specific testrpc-sc version on npm

Other notes:

  • check ethereumjs-vm dependency and upgrade
  • don't forget to integrate the gasLimit / default gas per transaction changes
  • look into debug_traceTransaction (per area's note in #52)

Comment immediately following function's opening brace breaks instrumentation

Hi,

using solidity-coverage 0.2.11 I get the following error on my contract:

/home/jerome/src/eth/contracts/coverageEnv/contracts/HeroCoin.sol:449:24: ParserError: Expected primary expression.
    returns (uint256) {/__FunctionCoverageHeroCoin('/home/jerome/src/eth/contracts/contracts/HeroCoin.sol',23);
                       ^
Compiliation failed. See above.
Cleaning up...

        There was an error generating coverage. Possible reasons include:
        1. Another application is using port 8555
        2. Truffle crashed because your tests errored

      Error: ENOENT: no such file or directory, open './allFiredEvents'
Exiting without generating coverage...

My truffle unit tests compile and pass.

Environment:
solcover 0.2.1
solc 0.4.11
ethereumjs-testrpc 4.0.1
truffle 3.4.5

Update Known Issues

  • Describe HDWalletProvider conflict in truffle.js / show zep fix.
  • Remove coveralls/codecov warning.
  • General cleanup - kind of an unreadable mess ATM
  • Rewrite testCommand instructions, advise that port 8545 is specified in .solcover.js and that user verifies no background instance of testrpc is running.

Catch sigint and clean up.

Problem:

$ ./node_modules/solidity-coverage/bin/exec.js
Generating coverage environment
mkdir: path already exists: ./coverageEnv
cp: copyFileSync: could not write to dest file (code=ENAMETOOLONG):coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/build/contracts/MultiSigWalletWithTimeLockExceptRemoveAuthorizedAddress.json
cp: cannot create directory './coverageEnv': No such file or direct

Solution: death

Disambiguate package name

NPM package name should be changed back solidity-coverage after fixing paths to testrpc in exec.js. Squash commits in Master, and re-run zeppelin-solidity tests to verify this all works.

Conditional Assignment to MemberExpression crashes

Found while testing against the Aragon contracts. They have a line like this:

vote.voted[voter] = isYay ? 1 : 2;

The left hand side parses as a member expression and there's a summary exit in the instrumented.js code for AssignmentExpressions to notify us if we encounter a node-type we didn't expect. Only handling DeclarativeExpressions and Identifiers at the moment.

Opening a PR for this shortly.

Handle events tests gracefully

  • Modify ethereumjs-vm-sc so that it doesn't add our events to the logs.
  • Unit test this with a simple events test that should find only its event, etc.

Strategy: in coverage.js, write topics to a file called scTopicList. In opFns.js load contents of that file into an array and check the topics of each Log OP against it. If it's ours, don't add it to the logs array?

Error parsing when using default configuration

When I run ./node_modules/.bin/solidity-coverage after installing in my project, I get the following error.

Launching testrpc on port 8555
Launching test command (this can take a few seconds)...
Using network 'development'.

SyntaxError: Error parsing /home/travis/prg/protocol/coverageEnv/contracts/governance/GovernanceProtocol.sol: Expected "contract", "import", "library", "pragma", "using", comment, end of input, end of line, or whitespace but "e" found. Line: 6, Column: 31
    at peg$buildStructuredError (/usr/local/lib/node_modules/truffle/node_modules/solidity-parser/build/imports_parser.js:543:12)
    at Object.peg$parse [as parse] (/usr/local/lib/node_modules/truffle/node_modules/solidity-parser/build/imports_parser.js:4142:11)
    at Object.parse (/usr/local/lib/node_modules/truffle/node_modules/solidity-parser/index.js:34:23)
    at /usr/local/lib/node_modules/truffle/node_modules/truffle-compile/profiler.js:200:36
    at /usr/local/lib/node_modules/truffle/node_modules/truffle-resolver/index.js:79:5
    at /usr/local/lib/node_modules/truffle/node_modules/truffle-resolver/node_modules/async/internal/onlyOnce.js:12:16
    at next (/usr/local/lib/node_modules/truffle/node_modules/truffle-resolver/node_modules/async/whilst.js:68:18)
    at /usr/local/lib/node_modules/truffle/node_modules/truffle-resolver/index.js:64:7
    at /usr/local/lib/node_modules/truffle/node_modules/truffle-resolver/fs.js:65:5
    at /usr/local/lib/node_modules/truffle/node_modules/truffle-resolver/node_modules/async/internal/once.js:12:16
Cleaning up...

    There was an error generating coverage. Possible reasons include:
    1. Another application is using port 8555
    2. Truffle crashed because your tests errored

  Error: ENOENT: no such file or directory, open './allFiredEvents'
Exiting without generating coverage...

May be worth nothing that there is no configuration stub generated in my truffle.json after this error, which should be there if I understand the readme correctly.

This is almost definitely something wrong on my end though, since it looks like you guys got it to work on the melonport protocol just fine, which is the code that I'm currently trying the coverage tool on.

Refactor options / exec.js cleanup

Options are pretty much incoherent ATM - esp. if someone uses the testrpc options one. Gaslimit should be hardcoded into testrpc-sc if it isn't already.

There's also a problem with the port . . . for example someone who doesn't specify a port but DOES use testrpcOptions will automatically run on 8545.

Also replace all instances of ${coverageDir} which are not part of a larger string. Should be coverageDir.

Allow file exclusion

  • add exclude option in .solcover.js which takes an array of paths relative to the root.
  • Istanbul accomplishes this using the /* istanbul ignore . . . */ strategy
  • ${coverageDir}/contracts/Migrations.sol is always ignored.

Invalid input source specified

When I ran solidity-coverage for the first time, it gave me this error:

Invalid input source specified.
Compiliation failed. See above.

After some debugging, it turns out that the problem was the paths on my imports. I'd been importing "Contract.sol" instead of "./Contract.sol" in my contracts. Since solidity-coverage is more particular about the current working directory than truffle is, someone else might run into this issue. It's not a solidity-coverage bug, but I'm creating an issue in the hopes that the next person to have that problem comes across it.

Start testrpc with '-i' switch

If we do this, we can set it to 'solidity-coverage' or something similar, and people can use web3.version.network in their tests to decide if they're running against coverage if they need to execute slightly differently (most likely, varying gas costs / responses to gas used).

EDIT: I know this can already be achieved with a custom testrpc options, but having it by default would be nice.

README updates

  • Clarify that the dir option also expects tests in the same directory as contracts
  • Use a table to document options

Testrpc upgrade

A new version of ganache-core is being published w/ various bugfixes and improvements. It's also moving from [email protected] to [email protected].

Todo:

  • Upgrade ethereumjs-vm-sc
  • Merge upstream ganache-core into ganache-core-sc
  • Build new testrpc-sc
  • Test on SC by installing testrpc-sc directly from repo
  • Publish testrpc-sc upgrade and set new pin on SC

Related to #63 by virtue of changing the vm.

testrpc-sc crashing with db error

Two cases:
From zeppelin CI build 433 at the start of the Claimable test. i.e. testrpc-sc launches ok, runs for a while and crashes.

/home/travis/build/OpenZeppelin/zeppelin-solidity/node_modules/ethereumjs-testrpc-sc/node_modules/web3-provider-engine/node_modules/solc/soljson.js:1
(function (exports, require, module, __filename, __dirname) { var Module;if(!Module)Module=(typeof Module!=="undefined"?Module:null)||{};var moduleOverrides={};for(var key in Module){if(Module.hasOwnProperty(key)){moduleOverrides[key]=Module[key]}}var ENVIRONMENT_IS_WEB=typeof window==="object";var ENVIRONMENT_IS_WORKER=typeof importScripts==="function";var ENVIRONMENT_IS_NODE=typeof process==="object"&&typeof require==="function"&&!ENVIRONMENT_IS_WEB&&!ENVIRONMENT_IS_WORKER;var ENVIRONMENT_IS_SHELL=!ENVIRONMENT_IS_WEB&&!ENVIRONMENT_IS_NODE&&!ENVIRONMENT_IS_WORKER;if(ENVIRONMENT_IS_NODE){if(!Module["print"])Module["print"]=function print(x){process["stdout"].write(x+"\n")};if(!Module["printErr"])Module["printErr"]=function printErr(x){process["stderr"].write(x+"\n")};var nodeFS=require("fs");var nodePath=require("path");Module["read"]=function read(filename,bi
NotFoundError: Key not found in database
    at /home/travis/build/OpenZeppelin/zeppelin-solidity/node_modules/level-sublevel/shell.js:101:18
    at /home/travis/build/OpenZeppelin/zeppelin-solidity/node_modules/level-sublevel/nut.js:121:19

From Gnosis build 100.1 at the start of the run, before truffle compile.

/home/travis/build/gnosis/gnosis-contracts/node_modules/solc/soljson.js:1
(function (exports, require, module, __filename, __dirname) { var Module;if(!Module)Module=(typeof Module!=="undefined"?Module:null)||{};var moduleOverrides={};for(var key in Module){if(Module.hasOwnProperty(key)){moduleOverrides[key]=Module[key]}}var ENVIRONMENT_IS_WEB=typeof window==="object";var ENVIRONMENT_IS_WORKER=typeof importScripts==="function";var ENVIRONMENT_IS_NODE=typeof process==="object"&&typeof require==="function"&&!ENVIRONMENT_IS_WEB&&!ENVIRONMENT_IS_WORKER;var ENVIRONMENT_IS_SHELL=!ENVIRONMENT_IS_WEB&&!ENVIRONMENT_IS_NODE&&!ENVIRONMENT_IS_WORKER;if(ENVIRONMENT_IS_NODE){if(!Module["print"])Module["print"]=function print(x){process["stdout"].write(x+"\n")};if(!Module["printErr"])Module["printErr"]=function printErr(x){process["stderr"].write(x+"\n")};var nodeFS=require("fs");var nodePath=require("path");Module["read"]=function read(filename,binary){filename=nodePath["normalize"](filename);var ret=nodeFS["readFileSync"
Error: Cannot find module 'levelup/lib/errors'
    at Function.Module._resolveFilename (module.js:485:15)
    at Function.Module._load (module.js:437:25)
    at Module.require (module.js:513:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/home/travis/build/gnosis/gnosis-contracts/node_modules/level-sublevel/shell.js:4:14)
    at Module._compile (module.js:569:30)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:503:32)
    at tryModuleLoad (module.js:466:12)
    at Function.Module._load (module.js:458:3)

Support for npm scripts which call npm packages

Try to run this on /0xProject/contracts. The tests run as is without issue (using npm run test, but throw an error when called via solidty-coverage. It seems to have something to do with the npm scripts:

"transpile": "rm -rf ./transpiled; copyfiles ./build/**/* ./transpiled; tsc",
"test": "npm run transpile; truffle test",

Error message:

Launching test command (this can take a few seconds)...

> [email protected] test /Users/primary/Projects/0x_contracts/coverageEnv
> npm run transpile; truffle test


> [email protected] transpile /Users/primary/Projects/0x_contracts/coverageEnv
> rm -rf ./transpiled; copyfiles ./build/**/* ./transpiled; tsc

sh: copyfiles: command not found
sh: tsc: command not found

npm ERR! Darwin 16.6.0
npm ERR! argv "/Users/primary/.nvm/versions/node/v7.10.0/bin/node" "/Users/primary/.nvm/versions/node/v7.10.0/bin/npm" "run" "transpile"
npm ERR! node v7.10.0
npm ERR! npm  v4.2.0

I also tried changing the script to

rm -rf ./transpiled; ./node_modules/.bin/copyfiles ./build/**/* ./transpiled; ./node_modules/.bin/tsc

This gets me a new error, which seems closer to success:

error TS5023: Unknown compiler option 'baseUrl'.
error TS5023: Unknown compiler option 'allowJs'.

Gnosis / Pythereum

  • Blocked by events.
  • Is their folder structure / the way tests are implemented somehow 'standard' for python based projects?
  • Adding a custom version of the vm to the build.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.