open-dust / cairo-foundry Goto Github PK

Similar to: https://docs.swmansion.com/protostar/docs/tutorials/testing/cheatcodes/expect-revert or https://doc.rust-lang.org/book/ch11-01-writing-tests.html#checking-for-panics-with-should_panic

It specifies that a program crash is expected and returns success when it happens

We aim to create a DX-friendly testing experience. To achieve that, we'll need to access important moments of the execution process.
Namely, we'd like to have these known test utils: beforeAll, afterAll, beforeEach ( ✅ we have this pre_step_instruction), afterEach

The after_run hook would be the equivalent of afterAll and similar hooks. It would run at the very end of your process (per test function, i.e. per run). It does not need to execute hints (such as mocking etc.) just yet.

The aim of this issue is to invest brain power in proposing a sound strategy to properly hook into this particular step of the execution process.

only CLI arguments are supported today.
It would improve UX if we add support for configuration file, which can be passed as CLI argument with default value.

In an effort of having fully documented code, we need the way we add custom hints (mocking calls, skipping instructions, expecting revert) to be documented.

In src/cli/commands/test, please proceed to comment above the fn setup_hint_processor declaration in an equivalent way as pub fn compile is documented.

The compiler module will be shared by multiple commands (list tests, execute test, etc).
It would be nice to avoid multiple compilations if the compiled program is up-to-date (generated after) the program itself.
It will increase UX in the following scenarii:

• list test several times in a row
• list test then execute one of them in a row

How to?
Check file system timestamps

First step is to be able to run a simple Cairo file, with its main function being run.
Once this is done, we will be able to actually build the test runner around this concept.
A PR is waiting to be validated in Cleopatra in order to be able to chose the program entrypoint.

If a .cairo file contains a function named __setup__, execute it before executing the test.

Ideas on how to implement it:
Add a before_first_step hook in the VM, in run_until_pc, that will push a call to setup by storing current fp and pc in memory and moving pc to the address of __setup__ (simulating a call).

Note: this won't be useful until we have starknet integration in cairo-rs

The objective of this contribution is to find a way to support starknet files, not to implement them fully.
The expected outcome is a detailed list of issues provisioning tasks on how to support starknet files.

The best case scenario would be to compile starknet files with starknet-compile instead of the current cairo-compile and then execute the compiled program directly in Cleopatra thanks to cairo-rs.
The first part is pretty straightforward, but we don't know if cairo-rs currently support starknet.
If yes, it should be really easy to list the tasks to be done.
If not, we expect the contributor to come up with an alternative way to implement it.

IO is expensive.
Each test file can contain multiple tests.
Right now we re-read the compiled file before running each test in cairo.rs:

let program = match Program::new(path, entrypoint)

What we can do instead is use cairo-rs deserialize_program_json once. And then use build our Program directly from this in-memory json + the entrypoint

What to do:

• change cairo-rs deserialize_program implem by factorizing the second part of the function in another public method
• call deserialize_program_json once then pass it to this new function each time we want to run new

In an effort of having fully documented code, we need the way we list files in a repository to be documented.

In src > cli > commands > list, please proceed to comment above the functions and modules declarations in an equivalent way as pub fn compile is documented.

Create new CLI command list to list test files within current repository

Create a sequence diagram to describe low level and precise interactions that occur during a Cairo-Foundry run.

Good example from Kakarot's repo:

sequenceDiagram
    title Simple bytecode execution flow example: [PUSH1 0x01 PUSH1 0x02 ADD]
    actor User
    participant Kakarot
    participant ExecutionContext
    participant EVMInstructions
    participant ArithmeticOperations
    participant PushOperations
    participant Stack
    User->>+Kakarot: execute(value, code, calldata)
    Kakarot->>+EVMInstructions: generate_instructions()
    EVMInstructions->>-Kakarot: instructions
    Kakarot->>+ExecutionContext: compute_intrinsic_gas_cost()
    ExecutionContext->>-Kakarot: ctx
    Kakarot->>Kakarot: run(instructions, ctx)
    loop opcode
        Kakarot->>+EVMInstructions: decode_and_execute(instructions, ctx)
        EVMInstructions->>EVMInstructions: retrieve the current program counter
        Note over EVMInstructions: revert if pc < 0, stop if pc > length of code
        EVMInstructions->>EVMInstructions: read opcode associated function from instruction set
        Note over PushOperations, Stack: x2 PUSH a=1, PUSH b=2
        EVMInstructions->>+PushOperations: exec_push1(ctx)
        PushOperations->>Stack: push(stack, element)
        PushOperations->>-EVMInstructions: ctx
        EVMInstructions->>+ArithmeticOperations: exec_add(ctx)
        Note over PushOperations, Stack: x2 POP a, POP b
        ArithmeticOperations->>Stack: pop(stack)
        Stack->>ArithmeticOperations: element
        ArithmeticOperations->>Stack: push(stack, result)
        ArithmeticOperations->>-EVMInstructions: ctx
        EVMInstructions->>-Kakarot: ctx
    end
    Kakarot->>-User: ctx

Use cairo_runner.write_output() to display the execution output on stdout.
Right now we use

cairo_run::write_output(&mut cairo_runner).map_err(|e| {
			format!(
				"failed to print the program output \"{}\": {}",
				self.program.display(),
				e,
			)
		})?;

that prepend the program output with a Program Output: line. A line we don't want for our own output

After updating cairo-rs, it seems that "skip" hint is no more functionnal.

Here is the output of cairo-foundry test:

$ ./target/debug/cairo-foundry test
Error: binary '/home/robin/cairo_venv/bin/cairo-compile' failed to compile './test_cairo_contracts/test_invalid_program.cairo:18:17: Unknown identifier '__main__.balance.read'.
    let (res) = balance.read();
                ^**********^
'
Error: An ASSERT_EQ instruction failed: 50 != 55.
Error: An ASSERT_EQ instruction failed: 3 != 2.
Running tests in file ./test_cairo_hints/test_mock_call.cairo
[OK] test_mock_call (543.961µs)


Error: Got an exception while executing a hint: Custom Hint Error: skip

Maybe it's related to the following info in the changelog:
VirtualMachineErrors produced by HintProcessor::execute_hint() will be wrapped in a VirtualMachineError::Hint error containing their hint_index
https://github.com/lambdaclass/cairo-rs/pull/673/files

In an effort of having fully documented code, we need the way skipping instructions to be documented.

Please proceed to comment above the pub fn skip declaration in an equivalent way as pub fn compile is documented.

In an effort of having fully documented code, we need the way we run tests for a single file to be documented.

In src/cli/commands/test, please proceed to comment above the fn run_tests_for_one_file declaration in an equivalent way as pub fn compile is documented.

In an effort of having fully documented code, we need hooks to be documented.

Please proceed to comment above the pub fn pre_step_instruction declaration in an equivalent way as pub fn compile is documented.

In an effort of having fully documented code, we need our cairo runner to be documented.

Please proceed to comment above the pub fn cairo_run declaration in an equivalent way as pub fn compile is documented.

Instead of needing a compiled Cairo file as an input, we want to integrate the Cairo compiler to compile .cairo files into .json and then running their tests using Cleopatra.
As a first step, this can be done considering the Cairo compiler should be in the current user PATH.

In an effort of having fully documented code, we need our mocking process to be documented.

Please proceed to comment above the pub fn mock_call declaration in an equivalent way as pub fn compile is documented.

https://github.com/onlydustxyz/cairo-foundry/blob/main/test_cairo_hints/test_mock_call.cairo#L4-L6

As for now, we can only return a single felt from a mocked call.
func_to_mock should stay as it is.
mock_ret_value should become ptr_mock_ret_value and contain a pointer to a memory address where the first value to return is stored
ptr_mock_ret_value_len should be introduced.

We aim to create a DX-friendly testing experience. To achieve that, we'll need to access important moments of the execution process.
Namely, we'd like to have these known test utils: beforeAll, afterAll, beforeEach ( ✅ we have this pre_step_instruction), afterEach

The before_run hook would be the equivalent of beforeAll and similar hooks. It would run at the very beginning of your process (per test function, i.e. per run). It does not need to execute hints (such as mocking etc.) just yet.

The aim of this issue is to invest brain power in proposing a sound strategy to properly hook into this particular step of the execution process.

This formatting option will be useful when interacting with other softwares, like vscode extension

On the January 4th, @Eikix rebased the open-dust cairo-rs fork, cairo-foundry branch.
It appears it broke the build process of the repository cairo-foundry.

@ptisserand offered to try and debug it.

We aim to create a DX-friendly testing experience. To achieve that, we'll need to access important moments of the execution process.
Namely, we'd like to have these known test utils: beforeAll, afterAll, beforeEach ( ✅ we have this pre_step_instruction), afterEach

The after_step_instruction hook would be the equivalent of afterEach and similar hooks. It would run after every step of your process (per test function, i.e. per run). It does not need to execute hints (such as mocking etc.) just yet.

The aim of this issue is to invest brain power in proposing a sound strategy to properly hook into this particular step of the execution process.

We always call cairo_run with those two parameters set as false
Better simplify the code and get rid of those parameters totally.
You will have to remove some logic inside cairo_run:

	if print_output {
		write_output(&mut cairo_runner, &mut vm)?;
	}

must go.

and replace let mut vm = VirtualMachine::new(program.prime, trace_enabled); by let mut vm = VirtualMachine::new(program.prime, false);

The repo currently imports the former version of cairo-rs called cleopatra, we need to get rid of it and only use cairo-rs from now on.

Right now if a test contains an infinite loop, it will endlessly loop.
We should cap the time we want to spend on each test and stop the test if this time is exceeded.

We can take a --max-step argument in the CLI, with a default value so that if not passed it will still be one.

Then we can check the current number of steps (already recorded in vm.current_step) against a max step value we initialized inside theexecution_scopes

In an effort of having fully documented code, we need the way we run the tests (exec) to be documented.

In src > cli > commands > execute, please proceed to comment above the module and functions declarations in an equivalent way as pub fn compile is documented.

Hi There,

I know fuzzing may be far out of the horizon, but wanted to draw attention to a design constraint that Protostar has with the hopes that cairo-foundry could potentially avoid it in the future.

Effectively - there is no way in Protostar to generate a fuzzing input in __setup__ and then use it across multiple test cases. More details here:

software-mansion/protostar#1149

It add no value to the user (other tools do it better) and complexity our refacto

Our aim is to run tests. So we don't want to actually run the main program.
We want to execute the different tests included within the project.

The ability to list those tests already exists, it's what is executed inside the list command.
To valid this issue, you should reuse this list ability and then execute each one of the test listed, aka launch multiple cairo_run:

• cairo_run(&self.program, "test_abc", false, &hint_processor)
• cairo_run(&self.program, "test_xyz", false, &hint_processor)
• cairo_run(&self.program, "test_mno", false, &hint_processor)

I believe those can be run in parallel (probably using rayon).

Output should list all tests and wether they succeded of failed.

Using tools such as excalidraw or lucidchart, create a comprehensive diagram explaining high-levels interactions occurring during a Cairo-Foundry run.

A good example would be Kakarot's architecture diagram:

As with Protostar, assertions can be done using hints.
There might be some investigation to conduct to check how one can use Cleopatra hints to achieve this.

In an effort of having fully documented code, we need the way we write test output to the console to be documented.

In src/hints, please proceed to comment above the init_buffer, clear_buffer, get_buffer and write_to_output_buffer declarations in an equivalent way as pub fn compile is documented.

In an effort of having fully documented code, we need the way we run tests on a single Cairo entry point to be documented.

In src/cli/commands/test, please proceed to comment above the pub(crate) fn test_single_entrypoint declaration in an equivalent way as pub fn compile is documented.

In an effort of having fully documented code, we need the way we expect reverted calls to be documented.

In src/hints/expect_revert, please proceed to comment above the pub fn expect_revert declaration in an equivalent way as pub fn compile is documented.

There are several cargo Clippy warnings on the Cache module. This issue is aimed at fixing them.

only one compiler is currently supported, could be useful to let the user choose its preferred compiler.

In an effort of having fully documented code, we need the way we list entry points in a Cairo repository to be documented.

In src/cli/commands/test, please proceed to comment above the fn list_test_entrypoints declaration in an equivalent way as pub fn compile is documented.

Tim: Avoir la liste de toutes les fonctions qui vont être mockeés => en parsant (grep like) les .cairo

A bit like Protostar's mock-call, we need to be able to mock some calls to other Cairo functions.
We can specify in the mock what the mocked function returns.

In an effort of having fully documented code, we need the functions that run during a test run to be documented.

In src/cli/commands/test, please proceed to comment above the fn setup_hooks, fn list_cairo_files, compile_and_list_entrypoints and purge_hint_buffer declarations in an equivalent way as pub fn compile is documented.

Context

The cairo-rs vm has a trait HintProcessor, that is currently only implemented by their BuiltinHintProcessor.

This processor let us add custom hints, matching the hint code with an hardcoded string.

Goal

We want to go further with these custom hints, and be able to have a hint processor that will match hints based on a regex.
This would let us provide arguments to the Rust hint function associated with a regex.

I did a PR for this in the vm repo, but it was suggested that this should live in our project directly.

You can find some documentation here to learn more about their hint processor concept: https://github.com/lambdaclass/cairo-rs/tree/main/docs/hint_processor

What we expect from you

• have the RegexHintProcessor used in our repo to be able to add regex-based custom hints
• have improved the code from the PR to compile the regex at "hint processor compile time"

It is a typical pattern in testing frameworks to be able to define a test function that takes arguments and returns a value, and then to use a case feature to call this test multiple times with different inputs and expected output.

In the current state of our tool, tests are defined as Cairo functions and called through the cairo_run function as entry points.

Problem: cairo-rs is built to accept one canonical entry point, the main function, which does not take any argument.

I think we can modify our fork of cairo-rs so it will init some memory for the arguments before entering the test function

Generalize usage of thiserror crate in the repository.

cairo-foundry caches stuff during its execution:

• the compiled files
• files that contain hash of the cairo files in order to avoid recompilation

Those are never deleted.

We should introduce a clean command. When executed it deletes the whole cache.