Make use of features from solc 0.8.23 release about execution-spec-tests

[richardgreg]

Hi @danceratopz, I've gone through the docs and have a fair understanding of the program, though my lack of blockchain background holds me back a bit.

If I'm not wrong, the files in ethereum_test_tools help with test execution by customising pytest, right? Could you give me an example of such a command, using the solc command line?

[danceratopz]

Hey @richardgreg 👍

The pytest-specific config/customization is in src/pytest_plugins/. The libraries in src/ethereum_test_tools/ help generate test vectors/fixtures (that are subsequently used by ethereum execution clients) but try to be independent of pytest (i.e., you could use a different test framework/runner but still use these libraries to generate test fixtures).

ethereum_test_tools does many things such as defining all the different types required to define test fixtures (accounts, blocks, transactions, etc), define the "test fixture" specifications and help write these resulting fixtures to JSON files.

As you might have seen, the Yul class from ethereum_test_tools helps compile any Yul source constained in tests, here are the relevant lines.

execution-spec-tests/src/ethereum_test_tools/code/yul.py

Lines 83 to 97 in ccf25a3

	result = run(
	solc_args,
	input=str.encode(self.source),
	stdout=PIPE,
	stderr=PIPE,
	)
	if result.returncode != 0:
	stderr_lines = result.stderr.decode().split("\n")
	stderr_message = "\n".join(line.strip() for line in stderr_lines)
	raise Exception(f"failed to compile yul source:\n{stderr_message[7:]}")
	lines = result.stdout.decode().split("\n")
	hex_str = lines[lines.index("Binary representation:") + 1]

If you'd like to see how the framework is calling solc, you can set some breakpoints or add some print statements here, for exampe:

print('\necho "' + self.source + '" | ' + " ".join(str(arg) for arg in solc_args))

which results in:

echo "
              {
                  function f(a, b) -> c {
                      c := add(a, b)
                  }

                  sstore(0, f(1, 2))
                  return(0, 32)
              }
              " | solc --evm-version shanghai --assemble -

(running fill tests/homestead/yul/test_yul_example.py --fork=Shanghai -s -v).

[danceratopz]

This option doesn't help us, more info in #467 (comment).