sc-forks / solidity-coverage Goto Github PK

Generalising this issue to contain all cases of valid solidity that we know we can't instrument...

These are either going to need edge cases coded, or use of debug_traceTransaction to avoid instrumentation all together.

Hopefully, nothing here would be something realistically used in 'real' solidity contracts, and are just odds and ends for us to look at.

If we do this, we can set it to 'solidity-coverage' or something similar, and people can use web3.version.network in their tests to decide if they're running against coverage if they need to execute slightly differently (most likely, varying gas costs / responses to gas used).

EDIT: I know this can already be achieved with a custom testrpc options, but having it by default would be nice.

• Blocked by events.
• Is their folder structure / the way tests are implemented somehow 'standard' for python based projects?
• Adding a custom version of the vm to the build.

Zeppelin, for example. High memory footprint issues have been solved in the latest testrpc They've also moved to a webpack build so we'll have to replicate that. Going to be a little tricky.

A new version of ganache-core is being published w/ various bugfixes and improvements. It's also moving from [email protected] to [email protected].

Todo:

• Upgrade ethereumjs-vm-sc
• Merge upstream ganache-core into ganache-core-sc
• Build new testrpc-sc
• Test on SC by installing testrpc-sc directly from repo
• Publish testrpc-sc upgrade and set new pin on SC

Related to #63 by virtue of changing the vm.

Also add link to github repo in package.

I think there is a race condition causing the test command to fail when TestRPC is not ready to accept requests by the time the tests begin making them. I was able to possibly address this via a workaround.

For more context, see the following Travis builds:

https://travis-ci.org/gnosis/gnosis-contracts/builds/253195228

https://travis-ci.org/gnosis/gnosis-contracts/builds/253307021

And this commit:

gnosis/pm-contracts@4606bb4

• Modify ethereumjs-vm-sc so that it doesn't add our events to the logs.
• Unit test this with a simple events test that should find only its event, etc.

Strategy: in coverage.js, write topics to a file called scTopicList. In opFns.js load contents of that file into an array and check the topics of each Log OP against it. If it's ours, don't add it to the logs array?

This is in question at issue #56.

• Describe HDWalletProvider conflict in truffle.js / show zep fix.
• Remove coveralls/codecov warning.
• General cleanup - kind of an unreadable mess ATM
• Rewrite testCommand instructions, advise that port 8545 is specified in .solcover.js and that user verifies no background instance of testrpc is running.

What do you think about allowing the user to specify the solc version being used?

The use-case would be a compiler regression that affects contracts, such as this one.

Of course, one can always downgrade/pin the solidity-coverage version, but features developed since then will be unavailable without getting the offending solc version as well.

One way to achieve this without dealing with different NPM packages may be using legacy solc versions, and specifying it in the .solcover.js file (though I have not tried this).

Anyway, let me know if this makes any sense to you :)

melonproject/protocol for example has a utilities folder located at project root which the tests require from e.g. require('../utilities/<file>')

• Clarify that the dir option also expects tests in the same directory as contracts
• Use a table to document options

When I ran solidity-coverage for the first time, it gave me this error:

Invalid input source specified.
Compiliation failed. See above.

After some debugging, it turns out that the problem was the paths on my imports. I'd been importing "Contract.sol" instead of "./Contract.sol" in my contracts. Since solidity-coverage is more particular about the current working directory than truffle is, someone else might run into this issue. It's not a solidity-coverage bug, but I'm creating an issue in the hopes that the next person to have that problem comes across it.

• Verify node type names are the same for our subset
• Global find/replace starts and ends with loc.start, loc.end
• Validate

A PR discussion at Open-Zeppelin caught this. And here is Istanbul on the same issue.

I wonder if we could (reliably) pre-process / instrument these into conditionals somehow. . .

solidity-coverage assumes the Truffle config is at truffle.js. There is an alternative filename truffle-config.js as seen in the documentation.

• We should be able to run solidity-coverage instrument <files>
• We should be able to require a cli.js file for our unit tests

This development here might be a breaking change? At the moment if we don't detect a coverage network we stub in a development network with high deployment gas. Not sure if this will affect us or not but worth checking when they publish.

• ChronoBank - (events have to be resolved)
• Numerai - (uses .solcover.js)
• Melonport (?) (requires #39)
• Bancorp (requires Node7)
• Aragon (Cannot figure this out . . . see below).
• 0xProject (. . . . .Typescript).
• status-network-token

And possibly PR these in? If useful?

Two cases:
From zeppelin CI build 433 at the start of the Claimable test. i.e. testrpc-sc launches ok, runs for a while and crashes.

/home/travis/build/OpenZeppelin/zeppelin-solidity/node_modules/ethereumjs-testrpc-sc/node_modules/web3-provider-engine/node_modules/solc/soljson.js:1
(function (exports, require, module, __filename, __dirname) { var Module;if(!Module)Module=(typeof Module!=="undefined"?Module:null)||{};var moduleOverrides={};for(var key in Module){if(Module.hasOwnProperty(key)){moduleOverrides[key]=Module[key]}}var ENVIRONMENT_IS_WEB=typeof window==="object";var ENVIRONMENT_IS_WORKER=typeof importScripts==="function";var ENVIRONMENT_IS_NODE=typeof process==="object"&&typeof require==="function"&&!ENVIRONMENT_IS_WEB&&!ENVIRONMENT_IS_WORKER;var ENVIRONMENT_IS_SHELL=!ENVIRONMENT_IS_WEB&&!ENVIRONMENT_IS_NODE&&!ENVIRONMENT_IS_WORKER;if(ENVIRONMENT_IS_NODE){if(!Module["print"])Module["print"]=function print(x){process["stdout"].write(x+"\n")};if(!Module["printErr"])Module["printErr"]=function printErr(x){process["stderr"].write(x+"\n")};var nodeFS=require("fs");var nodePath=require("path");Module["read"]=function read(filename,bi
NotFoundError: Key not found in database
    at /home/travis/build/OpenZeppelin/zeppelin-solidity/node_modules/level-sublevel/shell.js:101:18
    at /home/travis/build/OpenZeppelin/zeppelin-solidity/node_modules/level-sublevel/nut.js:121:19

From Gnosis build 100.1 at the start of the run, before truffle compile.

/home/travis/build/gnosis/gnosis-contracts/node_modules/solc/soljson.js:1
(function (exports, require, module, __filename, __dirname) { var Module;if(!Module)Module=(typeof Module!=="undefined"?Module:null)||{};var moduleOverrides={};for(var key in Module){if(Module.hasOwnProperty(key)){moduleOverrides[key]=Module[key]}}var ENVIRONMENT_IS_WEB=typeof window==="object";var ENVIRONMENT_IS_WORKER=typeof importScripts==="function";var ENVIRONMENT_IS_NODE=typeof process==="object"&&typeof require==="function"&&!ENVIRONMENT_IS_WEB&&!ENVIRONMENT_IS_WORKER;var ENVIRONMENT_IS_SHELL=!ENVIRONMENT_IS_WEB&&!ENVIRONMENT_IS_NODE&&!ENVIRONMENT_IS_WORKER;if(ENVIRONMENT_IS_NODE){if(!Module["print"])Module["print"]=function print(x){process["stdout"].write(x+"\n")};if(!Module["printErr"])Module["printErr"]=function printErr(x){process["stderr"].write(x+"\n")};var nodeFS=require("fs");var nodePath=require("path");Module["read"]=function read(filename,binary){filename=nodePath["normalize"](filename);var ret=nodeFS["readFileSync"
Error: Cannot find module 'levelup/lib/errors'
    at Function.Module._resolveFilename (module.js:485:15)
    at Function.Module._load (module.js:437:25)
    at Module.require (module.js:513:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/home/travis/build/gnosis/gnosis-contracts/node_modules/level-sublevel/shell.js:4:14)
    at Module._compile (module.js:569:30)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:503:32)
    at tryModuleLoad (module.js:466:12)
    at Function.Module._load (module.js:458:3)

For robustness, recommend:

• Using afterscript
• Using sudo: required
• Using a shell script that launches testrpc / setting norpc to true (copy zeppelin)

Gnosis has a test that takes like 10 minutes to cover and the file i/o at the LOG opcode could be part of that. It would be nice to find where the vm exits so we could write the logs in bulk.

And this, while visually balanced, is also insane.

Speeding things up is critical if the coverage tool is going to get plugged into a fuzzer

• reqCwd silently
• don't crash if there is no truffle.js
• allow interpolations outside of a function call

NPM package name should be changed back solidity-coverage after fixing paths to testrpc in exec.js. Squash commits in Master, and re-run zeppelin-solidity tests to verify this all works.

Problem:

$ ./node_modules/solidity-coverage/bin/exec.js
Generating coverage environment
mkdir: path already exists: ./coverageEnv
cp: copyFileSync: could not write to dest file (code=ENAMETOOLONG):coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/coverageEnv/build/contracts/MultiSigWalletWithTimeLockExceptRemoveAuthorizedAddress.json
cp: cannot create directory './coverageEnv': No such file or direct

Solution: death

A list of steps from #63:

• publish an npm version of solidity-coverage with all the testrpc-sc deps pinned to commits instead of master so there's a safe state.
• work out webpack / ganache-core in a dev branch
• validate against a few projects
• make testrpc 4.x new master (4.x is published from #coverage)
• write some dev instructions for us so we know how to update the webpack build
• npm publish testrpc-sc --> myrmi / cgewecke
• publish an npm version of soldity-coverage pinned to specific testrpc-sc version on npm

Other notes:

• check ethereumjs-vm dependency and upgrade
• don't forget to integrate the gasLimit / default gas per transaction changes
• look into debug_traceTransaction (per area's note in #52)

• Verified that ganache-core installs with [email protected] and [email protected]
• blockGasLimit = options.gasLimit || "0xfffffffffff"
• defaultTransactionGasLimit = '0xfffffff'

• add exclude option in .solcover.js which takes an array of paths relative to the root.
• Istanbul accomplishes this using the /* istanbul ignore . . . */ strategy
• ${coverageDir}/contracts/Migrations.sol is always ignored.

• commander
• mkdirp

Found while testing against the Aragon contracts. They have a line like this:

vote.voted[voter] = isYay ? 1 : 2;

The left hand side parses as a member expression and there's a summary exit in the instrumented.js code for AssignmentExpressions to notify us if we encounter a node-type we didn't expect. Only handling DeclarativeExpressions and Identifiers at the moment.

Opening a PR for this shortly.

Try to run this on /0xProject/contracts. The tests run as is without issue (using npm run test, but throw an error when called via solidty-coverage. It seems to have something to do with the npm scripts:

"transpile": "rm -rf ./transpiled; copyfiles ./build/**/* ./transpiled; tsc",
"test": "npm run transpile; truffle test",

Error message:

Launching test command (this can take a few seconds)...

> [email protected] test /Users/primary/Projects/0x_contracts/coverageEnv
> npm run transpile; truffle test


> [email protected] transpile /Users/primary/Projects/0x_contracts/coverageEnv
> rm -rf ./transpiled; copyfiles ./build/**/* ./transpiled; tsc

sh: copyfiles: command not found
sh: tsc: command not found

npm ERR! Darwin 16.6.0
npm ERR! argv "/Users/primary/.nvm/versions/node/v7.10.0/bin/node" "/Users/primary/.nvm/versions/node/v7.10.0/bin/npm" "run" "transpile"
npm ERR! node v7.10.0
npm ERR! npm  v4.2.0

I also tried changing the script to

rm -rf ./transpiled; ./node_modules/.bin/copyfiles ./build/**/* ./transpiled; ./node_modules/.bin/tsc

This gets me a new error, which seems closer to success:

error TS5023: Unknown compiler option 'baseUrl'.
error TS5023: Unknown compiler option 'allowJs'.

This bug was introduced in 0.1.9 during by the exec.js refactor. Changed a hardcoded path to use a variable in instrumentation sequence and it's off by a character. This was caught trying to debug a failure to get Coveralls reports at Aragon. Opening a PR and publishing a fix for this shortly.

We changed our path to execute testrpc-sc from the customary /.bin path to ./node_modules/ethereumjs-testrpc-sc/bin/testrpc in order to accommodate yarn in #48 and now package-lock.json has broken pattern with regular npm install and nests the dependency inside the solidity-coverage folder.

Just head over heels in love with this multi-package manager locking mania that has swept code recently.

Likelyhood of avoiding package-lock is zero since its constantly hanging out in the untracked files, driving everyone insane. Likelyhood of avoiding yarn also remote. . . .

var (, , , , , sharePrice) = performCalculations();

is not being parsed correctly - statement instrumentation is getting injected in front of the function call.

Instead of having a huge list that requires updates, we should just check for the existence of the function in the table before calling it
e.g:

parse[modifier.type] && 
parse[modifier.type](contract, modifier);

Unlike npm, yarn does not install binaries of dependencies to ./node_modules/.bin (see here). As a result when we try and run testrpc-sc, it fails. This could be fixed by executing ./node_modules/ethereumjs-testrpc-sc/bin/testrpc rather than ./node_modules/.bin/testrpc-sc. Alternatively, the onus could be on the use a tool like yarn-bin-fix.

Options are pretty much incoherent ATM - esp. if someone uses the testrpc options one. Gaslimit should be hardcoded into testrpc-sc if it isn't already.

There's also a problem with the port . . . for example someone who doesn't specify a port but DOES use testrpcOptions will automatically run on 8545.

Also replace all instances of ${coverageDir} which are not part of a larger string. Should be coverageDir.

When I run ./node_modules/.bin/solidity-coverage after installing in my project, I get the following error.

Launching testrpc on port 8555
Launching test command (this can take a few seconds)...
Using network 'development'.

SyntaxError: Error parsing /home/travis/prg/protocol/coverageEnv/contracts/governance/GovernanceProtocol.sol: Expected "contract", "import", "library", "pragma", "using", comment, end of input, end of line, or whitespace but "e" found. Line: 6, Column: 31
    at peg$buildStructuredError (/usr/local/lib/node_modules/truffle/node_modules/solidity-parser/build/imports_parser.js:543:12)
    at Object.peg$parse [as parse] (/usr/local/lib/node_modules/truffle/node_modules/solidity-parser/build/imports_parser.js:4142:11)
    at Object.parse (/usr/local/lib/node_modules/truffle/node_modules/solidity-parser/index.js:34:23)
    at /usr/local/lib/node_modules/truffle/node_modules/truffle-compile/profiler.js:200:36
    at /usr/local/lib/node_modules/truffle/node_modules/truffle-resolver/index.js:79:5
    at /usr/local/lib/node_modules/truffle/node_modules/truffle-resolver/node_modules/async/internal/onlyOnce.js:12:16
    at next (/usr/local/lib/node_modules/truffle/node_modules/truffle-resolver/node_modules/async/whilst.js:68:18)
    at /usr/local/lib/node_modules/truffle/node_modules/truffle-resolver/index.js:64:7
    at /usr/local/lib/node_modules/truffle/node_modules/truffle-resolver/fs.js:65:5
    at /usr/local/lib/node_modules/truffle/node_modules/truffle-resolver/node_modules/async/internal/once.js:12:16
Cleaning up...

    There was an error generating coverage. Possible reasons include:
    1. Another application is using port 8555
    2. Truffle crashed because your tests errored

  Error: ENOENT: no such file or directory, open './allFiredEvents'
Exiting without generating coverage...

May be worth nothing that there is no configuration stub generated in my truffle.json after this error, which should be there if I understand the readme correctly.

This is almost definitely something wrong on my end though, since it looks like you guys got it to work on the melonport protocol just fine, which is the code that I'm currently trying the coverage tool on.

• exec.js should be in /bin
• everything else should be in /lib

Hi,

using solidity-coverage 0.2.11 I get the following error on my contract:

/home/jerome/src/eth/contracts/coverageEnv/contracts/HeroCoin.sol:449:24: ParserError: Expected primary expression.
    returns (uint256) {/__FunctionCoverageHeroCoin('/home/jerome/src/eth/contracts/contracts/HeroCoin.sol',23);
                       ^
Compiliation failed. See above.
Cleaning up...

        There was an error generating coverage. Possible reasons include:
        1. Another application is using port 8555
        2. Truffle crashed because your tests errored

      Error: ENOENT: no such file or directory, open './allFiredEvents'
Exiting without generating coverage...

My truffle unit tests compile and pass.

Environment:
solcover 0.2.1
solc 0.4.11
ethereumjs-testrpc 4.0.1
truffle 3.4.5