Coder Social home page Coder Social logo

algorand / graviton Goto Github PK

View Code? Open in Web Editor NEW
17.0 6.0 8.0 207 KB

๐Ÿง‘โ€๐Ÿ”ฌ verify your TEAL program by experiment and observation

License: MIT License

Makefile 0.30% Python 27.82% Jupyter Notebook 71.88%
algorand avm blockchain pyteal pytest python smart-contracts tdd teal test-driven-development

graviton's Introduction

Algorand

About

Graviton is a software toolkit for blackbox testing of smart contracts written in TEAL.

Local Installation

The following instructions assume that you have make available in your local environment. In Mac OS and Linux this is most likely already available and in Windows one way to install is with chocolatey and the command choco install make.

To install all dependencies:

make pip-notebooks

Running Blackbox Integration Tests against a Sandbox

Prereq - Install and Symbolically Link to the Sandbox

If you would like to use the Makefile without modification and with full functionality, you should create a symbolic link to the algorand sandbox repo as described here. There are many ways to accomplish this. Assuming you have cloned the sandbox into the path /path/to/algorand/sandbox/ and that you've cd'ed into the cloned graviton directory you should create a symbolic link as follows:

Linux / Mac OS

ln -s /path/to/algorand/sandbox/ sandbox

Windows 10+

mklink sandbox \path\to\algorand\sandbox

Test the Sandbox

With your sandbox running to test that the sandbox is running properly, use the following:

make local-sandbox-test

Run the Integration Tests

make integration-test

Running and Testing Jupyter Notebooks

To run the notebook notebooks/quadratic_factoring_game.ipynb for example:

make local-notebook NOTEBOOK=notebooks/quadratic_factoring_game.ipynb

To non-interactively run all the jupyter notebook tests:

make notebooks-test

Ensuring that all is Copacetic Before Pushing to Github

To test in your local environment that everything is looking good before pushing to Github, it is recommended that you run make all-tests

If you would like to simulate the github actions locally, you'll need to install nektos act. On Mac OS with Docker previously installed you can use brew install act; on the other hand, on Linux and Windows follow the installation instructions in the nextos repo link above.

Once act is available you can simulate all the github actions integration tests with:

make local-gh-simulate

graviton's People

Contributors

ahangsu avatar aorumbayev avatar michaeldiamant avatar tzaffi avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

graviton's Issues

Better Demarcation of `act` Only Dependencies

Quoting @michaeldiamant

... we ought to disambiguate steps needed for act.

In order to get act to pass, I sourced various dependencies that I believe are not needed for github.
Disambiguation here means creating 1 or more steps dedicated to act dependencies. Goal is to make it easier to maintain the build.
I think the following dependencies are act-specific though it requires a round of testing:
curl, nodejs, python3-pip
Docker install (e.g. docker-ce)
Setup docker-compose

My own thoughts:

  • as much as possible, have the git build.yml refer back to our Makefile
  • create make commands that have act as their suffix (e.g. make dependencies-act)
  • have two commands that look almost the same make install-for-github v. make install-for-act that only differ in one command that runs only locally

Refactor `DryRunExecutor`

Problems Summary

Too many executing methods

See below for a list of DryRunExecutor methods that allow executing dryruns. It is cumbersome and confusing to have so many methods that basically do the same thing. Ideally, it would be nice to have a single method with the following API:

class DryRunExecutor:
    @singledispatch
    @classmethod
    def execute(
            cls: self,
            algod: AlgodClient,
            program: str,
            input: Any,
            *, 
            mode: ExecutionMode = ExecutionMode.Application,
            abi_method_signature: Optional[str] = None,
            ... several more params such as sender, sp, is_app_create, accounts ...
        ):
            raise NotImplementedError("Implement execute method")

This makes use of functools singledispatch decorator

Inconsistent abi-type handling

Currently, graviton's dry run execution are inconsistent in how abi-information is handled. Some allow abi_argument_types and abi_return_types to be provided while some also allow abi_method_signature to be provided. The last parameter includes all the information that abi_argument_types and abi_return_types provides and therefore it is confusing to provide all the parameters.

After an investigation into the usage of dry run execution in PyTeal, it became apparent that in all relevant situations, an ABIReturnSubroutine object is available for inspection, and it comes with a method method_signature() which could be used to populate an abi_method_signature parameter for dry run execution.

Action Items

Streamline the dry run execution API by refactoring the dry run executing methods as follows:

  • remove parameter abi_argument_types
  • remove parameter abi_return_type
  • add parameter abi_method_signature
  • make all the current dry run executing methods private these have been deleted
  • unify into a single execute() function using @singledispatch and @process.register to dispatch via input's type (Further investigation is required here. It may not be possible given the nesting of possible inputs (list[tuple] ... etc .... We may also prefer a multipledispatch approach taking into account the mode parameter's type. It may also be infeasible to dispatch directly and therefore we might need to match directly and delegate.) This didn't totally work out. However, we've unified everything into a run() method, with 2 companion convenience methods run_one() and run_sequence()
  • prepare a PyTeal PR which adapts to this new API (algorand/pyteal#628)
  • refactor supress_abi (not done)

List of dry run executing methods for refactoring

  • DryRunExecutor.execute_one_dryrun()
  • DryRunExecutor.dryrun_logicsig()
  • DryRunExecutor.dryrun_app()
  • DryRunExecutor.dryrun_logicsig_on_sequence()
  • DryRunExecutor.dryrun_multiapps_on_sequence()
  • DryRunExecutor.dryrun_app_pair_on_sequence()
  • ABIContractExecutor.dryrun_app_on_sequence()

Use Docker Image for C.I.

Description

Graviton was introduced into PyTeal's C.I. process in its PR #249. In that PR, a docker image of our Sandbox algod was introduced. This approach is significantly faster and more lightweight compared to the approach taken in this repo: build the sandbox from scratch each time, but leverage Github Action docker layer caching). We should.

Action Item

Modify Makefile and .github/workflows/build.yml to mimic those of PyTeal with regards to running the integration tests.

mypy: mark Type Aliases with typing.TypeAlias

There are several type aliases in the repo. They actually can be marked with a type: TypeAlias. EG:

from typing import TypeAlias, Union

MyCompoundType: TypeAlias = Union[str, int, None]

Start Publishing graviton on PyPi

pip install graviton ?

Currently, to install graviton locally you have to clone with:

  • git clone https://github.com/algorand/graviton.git
  • or pip install git+https://github.com/algorand/graviton

Graviton should be available as most other python libraries are.

Some points to consider:

Add Shields.io Badges

Repo Badges

#27 introduces badges for "powered by algorand" and for visitiors counts. It would be great to add additional badges for:

  • Test coverage status
  • CI result status
  • Licence
  • Anything else?

Refactor to use pysdk's DryRunTransactionResult

PySDK's PR #283 introduces DryRunInspector which is very similar in scope to blackbox.DryRunTransactionResult.

The two classes should be unified.

One possible approach is to rename to a existing class and subclass PySDK's as follows:

class  DryRunInspector(algosdk.DryRunTransactionResult):
    ...

Add updated budget info to `report()`

Summary

Since #33 we have better visibility into program budget via budget_added and budget_consumed. However, these weren't added to the report() function. E.g., part of a recent report looked like had a TXN AS ROW missing this info:

    TXN AS ROW: {' Run': 0, ' cost': -2009, ' last_log': '`0000000000011950', ' final_message': 'PASS', ' Status': 'PASS', 'steps': 89, ' top_of_stack': 72016, 'max_stack_height': 3, 's@000': 72016, 's@001': 3}

Action Item

  • Bring budget_added and budget_consumed into csv_row()

Stop using dry-run's cost

Cost is Deprecated in go-algorand's Dry Run

As of go-algorand PR #3957 the dry run response transaction top-level-field cost is being deprecated. There is one usage of this field in non-deprecated graviton code.

The good news is that new better fields BudgetConsumed and BudgetAdded are being introduced in the same PR and that the cost can be calculated as

net cost = BudgetConsumed - BudgetAdded

Action items

  • go-algorand # 3957 is released
  • have a PR that replaces cost with the net cost formula above and stops referencing the original cost field
  • have a PR that allows accessing BudgetConsumed and BudgetAdded in the same way that cost is
  • PR's above have been merged and released

Better assertion message for invariant predicates of 2 variables

Problem

As you can see in the full printout below, when we call invariant.validates() on an invariant that was defined
by a 2-variable predicate, we get an inscrutable expected description.

This invariant was defined via:

predicate = (lambda args, actual: "PASSE" == actual if args[0] else True)
name = 'DryRunProperty.status' 
invariant = Invariant(predicate, name=name)

and spat out the following msg:

<<<<<<<<<<<Invariant for 'DryRunProperty.status' failed for for args (2,): actual is [PASS] BUT expected [<function test_exercises.<locals>.<lambda> at 0x1082557e0>]>>>>>>>>>>>

Full error:

E           AssertionError: ===============
E               <<<<<<<<<<<Invariant for 'DryRunProperty.status' failed for for args (2,): actual is [PASS] BUT expected [<function test_exercises.<locals>.<lambda> at 0x1082557e0>]>>>>>>>>>>>
E               ===============
E               App Trace:
E                  step |   PC# |   L# | Teal              | Scratch   | Stack
E           --------+-------+------+-------------------+-----------+----------------------
E                 1 |     1 |    1 | #pragma version 6 |           | []
E                 2 |     2 |    2 | arg_0             |           | [0x0000000000000002]
E                 3 |     3 |    3 | btoi              |           | [2]
E                 4 |     7 |    6 | label1:           |           | [2]
E                 5 |     9 |    7 | store 0           | 0->2      | []
E                 6 |    11 |    8 | load 0            |           | [2]
E                 7 |    13 |    9 | pushint 2         |           | [2, 2]
E                 8 |    14 |   10 | exp               |           | [4]
E                 9 |     6 |    4 | callsub label1    |           | [4]
E                10 |    15 |   11 | retsub            |           | [4]
E               ===============
E               MODE: ExecutionMode.Signature
E               TOTAL COST: None
E               ===============
E               FINAL MESSAGE: PASS
E               ===============
E               Messages: ['PASS']
E               Logs: []
E               ===============
E               -----BlackBoxResult(steps_executed=10)-----
E               TOTAL STEPS: 10
E               FINAL STACK: [4]
E               FINAL STACK TOP: 4
E               MAX STACK HEIGHT: 2
E               FINAL SCRATCH: {0: 2}
E               SLOTS USED: [0]
E               FINAL AS ROW: {'steps': 10, ' top_of_stack': 4, 'max_stack_height': 2, 's@000': 2}
E               ===============
E               Global Delta:
E               []
E               ===============
E               Local Delta:
E               []
E               ===============
E               TXN AS ROW: {' Run': 1, ' cost': None, ' last_log': '`None', ' final_message': 'PASS', ' Status': 'PASS', 'steps': 10, ' top_of_stack': 4, 'max_stack_height': 2, 's@000': 2, 'Arg_00': 2}
E               ===============
E               <<<<<<<<<<<Invariant for 'DryRunProperty.status' failed for for args (2,): actual is [PASS] BUT expected [<function test_exercises.<locals>.<lambda> at 0x1082557e0>]>>>>>>>>>>>
E               ===============

TEAL Coverage in Graviton

Coverage Reports

Currently, dry runs return program execution traces which are exposed in the Inspector class via its report() method. Thus, in principle, it should be possible to compute statistics on which lines of associated TEAL program were executed over a sequence of dry runs. With such statistics in hand, coverage reports are therefore possible.

TODO

Flesh this issue out with deliverables and example code coverage frameworks that can serve as inspiration.

Links

  • Coverage in the ETH space: brownie

Transactions Groups in Graviton

Lingering TODO from #21:

  • Determine if dry runs can be made to handle transaction groups
  • Determine if dry runs can be made to handle reference types and foreign references
  • If the determination of the previous is "yes", enable graviton to handle such transaction groups + foreign references and then use to test abi methods with transaction/reference arguments. It should verify that when the expected pay / axfr / ... transactions are too far away (compared to the number of such args in the method signature and or foreign array) the program is rejected with error.

Convenience wrapper for running subroutines

Hi, found your project on the Algorand Discord and it's made my testing significantly easier, great job and thanks! I hope it will be fully integrated into PyTeal very soon.

While incorporating it into my own unit tests, I wrote an additional wrapper around the BlackboxWrapper that you included in the PyTeal repository to skip some of the steps that I was doing repeatedly, like compiling the TEAL, arranging inputs correctly, casting to numeric types etc.

This wrapper allows you to write tests like this:

import pyteal as pt

@SubroutineRunner(input_types=[pt.TealType.uint64, pt.TealType.uint64])
@pt.Subroutine(pt.TealType.uint64)
def uint64_subr(x: pt.Expr, y: pt.Expr) -> pt.Expr:
    return pt.Div(x, y)

@SubroutineRunner(input_types=[pt.TealType.uint64, pt.TealType.uint64])
@pt.Subroutine(pt.TealType.bytes)
def bytes_subr(x: pt.Expr, y: pt.Expr) -> pt.Expr:
    return pt.BytesDiv(pt.BytesMul(pt.Itob(x), pt.Bytes("base16", "0x03e8")), pt.Itob(y))

x, y = [8, 10, 50], [2, 9, 10]
for input, output in uint64_subr.run_sequence(y=y, x=x).inout:
    assert output == input["x"] // input["y"]
for input, output in bytes_subr.run_sequence(x=x, y=y).inout:
    assert output == (1000 * input["x"]) // input["y"]

...which I think maybe could be of interest to others. It automatically takes care of mapping keyword arguments to the correct order when calling the subroutine, calls stack_top() for uint64 and last_log() for bytes and converts to Python ints and calls the dryrun for apps or logicsigs depending on an optional pt.Mode argument in the decorator, but is otherwise a pretty simple extra wrapper that just gives a shorter path from the subroutine to the dryruns.

The PyTeal repository doesn't seem like the place to share it, since right now graviton is only used within the tests, so I don't know where I could do a pull request, as it depends on code I found within the PyTeal repo. In case you think it's of any interest now or in the future, just let me know where you think is the best place to share the code, otherwise feel free to close it. :)

Enable Running Blackbox Testing using the Offline Version of Dryrun

The goal clerk dryrun command (in opposition to the goal clerk dryrun-remote) is in theory capable of running dry runs without an actual node running. If we use this functionality, we could potentially save on most of the startup times of the C.I. in this repo (and pyteal as well). In theory, this would work as follows:

  1. add offline option to executions: e.g. DryRunExecutor.execute_one_dryrun(offline=True, ...)
  2. this delegates to an os-level process goal clerk dryrun ...
  3. the response should come back as a JSON
  4. the response's transactions are then consumed as usual by blackbox.DryRunInspctor
  5. inspections and assertions proceed as in the current approach

Docs Overhaul

Graviton Needs a Major Cleanup/Additions/Removals of its Docs

There's lots more to write here. But start with TASK 1:

  • TASK 1: add to this list with specific items that need updating/removing/adding

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.