crytic / crytic-compile Goto Github PK
View Code? Open in Web Editor NEWAbstraction layer for smart contract build systems
License: GNU Affero General Public License v3.0
Abstraction layer for smart contract build systems
License: GNU Affero General Public License v3.0
There are projects that managed to integrate Buidler and Crytic Compile, like fyDai, but they are using older versions of the tools.
With a clean-slate repo, [email protected] and a basic Greeter.sol contract, this is the error that you're getting:
$ crytic-compile . --compile-force-framework buidler
INFO:CryticCompile:'npx buidler compile' running
INFO:CryticCompile:Compiling...
Compiled 2 contracts successfully
ERROR:CryticCompile:Unknown file: ontracts/Greeter.sol
crytic-compile . --compile-force-framework buidler
Following the trailofbits/manticore#909 discussion:
We need to improve
crytic-compile/crytic_compile/crytic_compile.py
Lines 942 to 956 in 35370b6
To handle newer versions of the metadata encoding, including:
0xa2
0x65 'b' 'z' 'z' 'r' '1' 0x58 0x20 <32 bytes swarm hash>
0x64 's' 'o' 'l' 'c' 0x43 <3 byte version encoding>
0x00 0x32
0xa2
0x64 'i' 'p' 'f' 's' 0x58 0x22 <34 bytes IPFS hash>
0x64 's' 'o' 'l' 'c' 0x43 <3 byte version encoding>
0x00 0x32
We could also handle the parsing of the metatadata, like doing the key-values extraction.
Additionally, we should add a public function allowing external tools to access these functionalities. This would be useful for, at least, Manticore and evm-cfg-builder.
The etherscan test needs our APIKEY
Which is not available for external users. As a result external PRs will always fail. I see two solutions
1 - Disable etherscan for external PRs
2 - Ask external PR to provide an API key
I would favor (1) as (2) is probably going to be too annoying for contributors
Example: #97
It would be useful to know what platform is in use for a project, and possibly solc version, in order to know how to automatically run unit tests on a project without needing a Crytic user to specify this.
I'm not sure if it would be better to make this part of the standard output of crytic-compile, or to add an option like --project-info that outputs this information as json.
https://github.com/compound-finance/saddle
This framework is used by https://github.com/compound-finance/compound-protocol
Related: crytic/slither#571
We should parse the documentation returned by devdoc
/ userdoc
, and provides it to external tools.
In the event that crytic-compile
is run on a contract whose filename is prefixed with an address, it seems crytic-compile
assumes the input is an address, and tries to resolve the contract on etherscan using the filename.
Input:
slither --print function-id 0x0000000000b3F879cb30FE243b4Dfee438691c04_GasToken2.sol
Output:
Contract has no public source code
ERROR:Slither:Invalid compilation
ERROR:Slither:Contract has no public source code: https://api.etherscan.io/api?module=contract&action=getsourcecode&address=0x0000000000b3F879cb30FE243b4Dfee438691c04_GasToken2.sol
The path provided clearly references a filename, and it should read the local file as intended. This case can likely be quickly fixed with tighter constraints on what constitutes an etherscan query (should be no larger than a 20 byte hexadecimal integer prefixed with 0x
).
Many arguments are expected to be serialized str
e.g. solc_args
. However, crytic-compile will often need to deserialize them and also serialize again.
While it works perfectly for user inputs (e.g. arguments taken from CLI), it would be a unnecessary overhead when CryticCompile is used as a library that developers need to serialize for no use (because soon will be deserized in CryticCompile).
crytic-compile/crytic_compile/platform/truffle.py
Lines 211 to 228 in 758ce45
In 99% of the case, the issue is that truffle
is not found
To avoid another #46
If there is an import
import "my_package/contracts/test.sol"
Within my_package
itself, crytic-compile fail to find the file.
crytic-compile
should be able to return the underlying config filename if it detects one
We should add support to the upcoming json input format for Vyper: vyperlang/vyper#1594
It is similar to the solc json input for which we recently added the support: #29
We need some internal discussions on how to handle best these extensions of existing platforms (solc
versus solc_standard_json
, and vyper
versus the platform to handle this new input)
The archive mode will export a file containing all the sourcecode.
Most likely we can extend the existing standard export with
"files": {
"/absolute/path/to/filename" : "....",
"/absolute/path/to/filename2" : "...."
}
We need to add the proper API, so external tools can directly access to the files
The current name crytic.config.json
is extremely close to crytic's standard config file of crytic-config.json
. Maybe crytic-compile.config.json
to be more specific? Change . and - as desired...
With this name, some crytic customer is going to get very confused (I did).
We can add some generic config filenames to autodetect waffle projects:
.waffle.json
waffle.json
.waffle[.]config.json
waffle[.]config.json
Related: crytic/slither#508
In some cases, truffle can compile codebase with libraries without the need from a local npm install
. I am not sure when this happens, but my best guess is that the libraries are installed globally.
It happened most of the time with the OZ libraries.
If the libraries are not present in node_modules
, we should look in the globally installed libraries
For example
crytic-compile file.sol --solc-args "--optimize"
will fail, while
crytic-compile file.sol --solc-args "optimize"
works.
But if multiple args are provided the --
is required for every argument:
crytic-compile file.sol --solc-args "--optimize --optimize-runs 100"
This assert breaks if not solidity is installed. The output could be misleading in such case.
slither inside crytic failed for a buidler project
it tries to compile project with npx buidler compile
command.
if I run it locally, it produces the following output:
$ npx buidler compile
Warning: Buidler has been moved to @nomiclabs/buidler
Run `npm install --save-dev @nomiclabs/buidler` to get the latest version of Buidler.
I guess the command in buidler.py should be npx @nomiclabs/buidler compile
, which works for me locally
Similar to is_supported
, we should have a way for third party to know if any of the ignore compile flags set.
Alternatively, we might want to remove all the ignore compile flags, except for --ignore-compile
; I am not convinced it makes sense anymore to keep individual flag per platform
We should use npx
for embark/etherlime , to prevent a failed call if the tools are not installed
npx-based platforms (truffle, embark, ...) are sometimes slow to run only because of npx
.
If the user knows that he has the correct version of the platform installed, he should be able to skip npx
with a flag like --npx-disable
.
crytic-compile/crytic_compile/platform/waffle.py
Lines 60 to 74 in 7f6a1c6
Crytic-compile assumes that the compiler is always native
, leading to incorrect solc version used, and potentially crashing if waffle is using another compiler.
Example of project that uses solcjs
: https://github.com/Uniswap/uniswap-lib/blob/022648f4200c0b16813b817983885d358a439920/waffle.json#L2
If the embark-solc plugin is used, embark-contract-info will not work, because the hook on compilation will never be triggered. As a result, no json file is produced
If the flag is set to True, it will active --PLATFORM-ignore-compile
for all the platform implementing the feature
If the target is a directory, the build
folder will not be looked in the correct directory.
We might want if this behavior is also incorrect in other platforms
Related crytic/slither#508
For some reasons, crytic-compile does work locally with the latest etherlime release, but travis seems to be broken.
A frequent mistake for our tool is to run them on project_dir/contracts
instead of project_dir
.
If no platform is found, and the most right part of the path is contracts
, we should try to detect the platform on the parent directory
The dapp unit tests take almost one hour to run, which is significantly higher than the other test cases that take a few minutes
crytic-compile/scripts/travis_test_dapp.sh
Lines 10 to 11 in 05c92a8
We should consider using pre-compiled versions of the dapptools, watch their repo and update the precompiled binaries when needed.
Etherlime 1.1.3 added the flag deleteCompiledFiles
, which will remove the previous json artifacts, and prevents issues with missing contracts.
We could run etherlime --version
and add the flag by default according to the result
The 0x compiler seems to support the standard json configuration, if it is true, we should be able to output the compact AST.
Using crytic-compile with of contract that requires to link libraries, like this one:
library Test {
struct Storage{
bool flag;
}
function set(Storage storage st) public{
st.flag = true;
}
}
contract Contract {
using Test for Test.Storage;
Test.Storage st;
function set() public{
st.set();
}
}
could produce a .json file with an invalid bin
field if the libraries are not provided. For instance:
608060405234801561001057600080fd5b5060cb8061001f6000396000f3fe6080604052348015600f57600080fd5b506004361060285760003560e01c8063b8e010de14602d575b600080fd5b60336035565b005b600073__$71b117f040f276ba3c25e131c37e06e6a0$__
63cb7913b890916040518263ffffffff1660e01b81526004018082815260200191505060006040518083038186803b158015608657600080fd5b505af41580156099573d6000803e3d6000fd5b5050505056fea165627a7a72305820972574bd81452308fbcf0d1580c1f7dde707b2a5c3024bc32843e80ad812a9be0029
In this case, Echidna will read this field incorrectly because it uses hevm code, that assumes that all the libraries already linked. Please considering allow to crytic-compile to fail this compilation, perhaps by default of adding a flag --fail-if-unlinked
or something like that, if unlinked libraries are detected.
The use of external functions can make property testing more difficult to use than expected, since the functions from a contract cannot be called from it (without using external call). It will be nice to have a flag in crytic-compile to replace the use of all external
functions by public
ones, without modifying the original source (this should work for complex projects like truffle
to be really useful).
I ran the following steps:
cryticArgs: ["--compile-force-framework", "buidler"]
in my config.yml
fileechidna-test ./contracts/invariants/FintrollerInvariants.sol --config ./contracts/invariants/config.yml
And received the error below.
ethsec@dd7de7baf69b:/home/workspace$ echidna-test ./contracts/invariants/FintrollerInvariants.sol --config ./contracts/invariants/config.yml
echidna-test: Couldn't compile given file
stdout:
stderr:
INFO:CryticCompile:'npx buidler compile' running
Traceback (most recent call last):
File "/home/ethsec/.local/bin/crytic-compile", line 33, in <module>
sys.exit(load_entry_point('crytic-compile==0.1.9', 'console_scripts', 'crytic-compile')())
File "/home/ethsec/.local/lib/python3.6/site-packages/crytic_compile/__main__.py", line 157, in main
compilations = compile_all(**vars(args))
File "/home/ethsec/.local/lib/python3.6/site-packages/crytic_compile/crytic_compile.py", line 1019, in compile_all
compilations.append(CryticCompile(target, **kwargs))
File "/home/ethsec/.local/lib/python3.6/site-packages/crytic_compile/crytic_compile.py", line 123, in __init__
self._compile(**kwargs)
File "/home/ethsec/.local/lib/python3.6/site-packages/crytic_compile/crytic_compile.py", line 930, in _compile
self._platform.compile(self, **kwargs)
File "/home/ethsec/.local/lib/python3.6/site-packages/crytic_compile/platform/buidler.py", line 66, in compile
cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=self._target
File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
NotADirectoryError: [Errno 20] Not a directory: './contracts/invariants/FintrollerInvariants.sol'
Previous versions of slither allowed a user to compile a truffle directory whether or not the current working directory was the truffle directory to compile, ie:
slither .
(current directory)
slither .\truffle_project\
(not current directory)
crytic-compile does not properly calculate file paths, as it will try to join paths to the current working directory, where it should actually join to the directory specified via command line.
The problematic code can be found here:
crytic-compile/crytic_compile/utils/naming.py
Lines 46 to 54 in 1df06f2
Example:
/c/test/
, and we wish to compile a truffle directory located at /c/test/truffle_project/
.slither .\truffle_project\
/c/test/
, not /c/test/truffle_project/
.ERROR:CryticCompile:Unknown file: openzeppelin-solidity\contracts\token\ERC20\ERC20.sol
The solution to this issue should be careful in noting that the supplied path to compile can be relative or absolute, and should not be blindly joined in the middle of the current working directory and relative file paths.
Side note: Although this is titled 'truffle compilation', it likely applies to other compilation schemes as well.
We are not parsing the correct solc version used with Waffe
Related: crytic/slither#508
--compile-force-framework truffle
should work the same as --compile-force-framework Truffle
The sourceList
field does not always preserve its original order:
crytic-compile/crytic_compile/platform/solc.py
Lines 63 to 66 in 7f6a1c6
This is caused because crytic_compile.filenames
is a Set and converting it to a list won't always preserve its original ordering. Unfortunately, some external software like hevm relies on this to match code in different sources.
Truffle detects now codebase with duplicate names. We should log this info, so that external tools (including Slither) can highlight it.
I think at least brownie has a similar detection
crytic/slither#253 requires crytic_compile to provide the constructor arguments.
We should add the information to CompilerVersion
namedtuple: https://github.com/crytic/crytic-compile/blob/master/crytic_compile/compiler/compiler.py#L3
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.