Remove descriptions for multithreaded proof generation

Since multithreaded proof generation is not working yet, we should remove this from documentation, so nobody would try using it.

User guides - fixed by #71
Installation page - fixed by #102
Requester-producer flow - fixed by #102

This is how it was described and how it was supposed to work:

To generate proof using the multithreaded version, execute the following:

cd build
./bin/proof-generator/proof-generator-mt \
    --proof_out <output file> \
    --circuit_input <statement from Proof Market>
    --public_input <public input from Proof Market> \
    --smp <number of threads>
    ----shard0-mem-scale <scale>

Proof computation requires temporary space, always allocated to core0/shard0.
The following flags require attention for multithreaded use:

smp - number of threads.
This is ideally the number of CPU cores on the system.
If you have a 16-core CPU & 16 GB RAM, you can set smp to 16.
Each core gets access to 1 GB of RAM, so core0/shard0 will have access to 1 GB of RAM.
shard0-mem-scale - weighted parameter (weight) to reserve memory for shard0.
This is a natural number that allows for shard0 to have more access to RAM in comparison to others.
If you have a 16-core CPU & 16 GB RAM, you can set smp to 8 and shard0-mem-scale to 9.

We first compute ram_per_shard variable as follows:

ram_per_shard = TOTAL_RAM / (smp + shard0-mem-scale -1)
              = 16 / (8 + 8)
              = 1

Which equates to the following:

shard0 RAM =shard0-mem-scale * ram_per_shard = 9 GB
shard1 .... shard7 = ram_per_shard = 1 GB/per core = 7 GB

You should tune these two variables for the architecture/circuit for which the proof is generated.

There's also a neat trick that we could recommend:

Linux

--smp $(nproc)

macOS

--smp $(sysctl -n hw.logicalcpu)

See for example:

https://github.com/NilFoundation/zkllvm/blob/master/.github/workflows/build_linux.yml#L105
https://github.com/NilFoundation/zkllvm/blob/master/.github/workflows/build_macos.yml#L77

Make statements private by default

Statements have the field isPrivate, which is false by default. With active development, such as during our workshops, a lot of statements will be pushed to the Proof Market. We should make them private by default and let developers make them public explicitly.

prepare_statement.py --private
prepare_statement.py --public

Update zkllvm proof generator

Ambiguous field names in response of `requrest_tools.py push`

On the market there's a statement with key 93843771:

    {
        "_key": "93843771",
        "createdOn": 1689098910158,
        "description": "description",
        "input_description": "input_description",
        "isPrivate": false,
        "name": "nickvolynkin_example_circuit",
        "sender": "nickvolynkin",
        "type": "placeholder-zkllvm",
        "updatedOn": 1689098910158,
        "url": "url"
    }

Pushing a proof proposal for this statement:

/proof-market-toolchain# python3 scripts/request_tools.py push --cost 2 --file /opt/zkllvm-template/src/main-input.json --key 93843771

returns the following data:

{
    "_key": "93846511",
    "statement_key": "93843771",
    "cost": 2,
    "sender": "nickvolynkin",
    "status": "created"
}

It's unclear what the _key is. Must be the proposal's ID.

multithread version

Dockerfiles: implement Hadolint check in CI, fix issues

Add tests to CLI

Make zkLLVM dependencies binary

Rewrite proof market toolchain logic and make it dependent on zkllvm binaries (zkllvm clang, assigner, transpiler) and proof-producer binary.

Update zkllvm-circifier

Proofread the existing Proof Market documentation

The current docs have a few word choice, punctuation, and style issues. Let's fix them before making significant changes like #63.

Make branches from the branch https://github.com/nilfoundation/proof-market-toolchain/tree/00-gitbook-docs.

For each chunk of work, open a PR from your branch → 00-gitbook-docs. Make chunks small enough to make 2-3 PRs a day.

Statement Developer one-command-run

Add links to the GitHub repository of PMT

Our docs describe working with PMT nicely, but fails to provide a link for that repo on GitHub

These pages definitely should include it:

CLI guides - fixed by #94
Getting started -> Installation - fixed by #102

Remove the `generation_time` parameter from PMT documentation

Right now, the generation_time parameter is just ignored when you're submitting a proof request or proposal via CLI. It's added to arguments but seems to be never used in the push function: https://github.com/NilFoundation/proof-market-toolchain/blob/master/scripts/request_tools.py

So, we have to remove it for now, and when it's implemented we can restore the corresponding parts of the documentation

Short keys don't match long ones in proof_tools.py CLI

Keys were renamed recently, but the short versions were not. It makes CLI harder to understand and remember. What if we rename them accordingly?

https://github.com/NilFoundation/proof-market-toolchain/blob/master/scripts/proof_tools.py

-    parser_push.add_argument("-a", "--proposal_key", type=str, default=None, help="proposal_key")
+    parser_push.add_argument("-p", "--proposal_key", type=str, default=None, help="proposal_key")
-    parser_push.add_argument("-b", "--request_key", type=str, default=None, help="request_key")
+    parser_push.add_argument("-r", "--request_key", type=str, default=None, help="request_key")
     parser_push.add_argument(
         "-f", "--file", type=str, required=True, help="file with proof"
     )

-    parser_get.add_argument("-p", "--proof_key", type=str, help="key of the proof")
+    parser_get.add_argument("-k", "--proof_key", type=str, help="key of the proof")
     parser_get.add_argument("-f", "--file", type=str, help="file to write proof")
     parser_get.add_argument("-b", "--request_key", type=str, help="request_key")

Update toolchain for new app structure

Docker image doesn't have `requests` module

/proof-market-toolchain$ python3 scripts/auth_tools.py --help
Traceback (most recent call last):
  File "/proof-market-toolchain/scripts/auth_tools.py", line 5, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'

Keep just `auth.json` but not the basic user credentials

Current workflow

User registers with ./scripts/signup.py, script saves credentials in ./scripts/.user and ./scripts/.secret.
- Alternatively, user registers in the Web UI and manually saves credentials in in ./scripts/.user and ./scripts/.secret.
User authenticates with ./scripts/auth_tools.py, script saves JSON web token in ./auth.json
Other scripts use the JSON web token from ./auth.json.

There's a number of problems here:

It's very unsafe, and users can lose their credentials by committing them in git.

Working with docker requires mounting a lot of files:

docker run -it --rm \
  ...
  --volume $(pwd)/.config/.user:/proof-market-toolchain/scripts/.user \
  --volume $(pwd)/.config/.secret:/proof-market-toolchain/scripts/.secret \
  --volume $(pwd)/.config/auth.json:/proof-market-toolchain/auth.json \
  ...

Proposed workflow

User registers with ./scripts/signup.py on in WebUI and saves their credentials elsewhere: in a password manager, in memory, on a sticky note, whatever.
User runs ./scripts/auth_tools.py, the script prompts for credentials, gets a JSON web token, and saves it in ~/.config/proof-market/auth.json.
Other scripts use the JSON web token from that location.

Further on and out of scope of this issue, we'll make the following changes:

The proof-producer binary will look for its configuration file somewhere under /etc/proof-market/config.ini
Everything with a configuration or auth file will have optional --config and --auth parameters.

Update get query for public inputs

This change refers to the workflow where proof producers get public inputs from the order itself. JSON structuring requires changing as this was not compatible with the proof producer binary.

Running proof-generator without params or with -h/-v results in an attempt to find config.ini

When I run the proof-generator with empty input or with -h and -v, I get unexpected results:

root@5ffc715e5f63:/proof-market-toolchain# /proof-market-toolchain/build/bin/proof-generator/proof-generator
can not read options configuration file '/root/.config/config.ini'
terminate called after throwing an instance of 'boost::wrapexcept<boost::property_tree::ini_parser::ini_parser_error>'
  what():  /root/.config/config.ini: cannot open file
Aborted (core dumped)

root@5ffc715e5f63:/proof-market-toolchain# /proof-market-toolchain/build/bin/proof-generator/proof-generator -v
can not read options configuration file '/root/.config/config.ini'
terminate called after throwing an instance of 'boost::wrapexcept<boost::property_tree::ini_parser::ini_parser_error>'
  what():  /root/.config/config.ini: cannot open file
Aborted (core dumped)

# slightly different result, but still an error:
root@5ffc715e5f63:/proof-market-toolchain# /proof-market-toolchain/build/bin/proof-generator/proof-generator -h
Command_Line_Options:
  -h [ --help ]                         Display available command-line
                                        configuration arguments
  -c [ --configuration-files ] arg (=/root/.config/config.ini)
                                        Configuration files

NIL Proof Generator:
  -v [ --version ]                      Display version
  --proof_out arg                       Output file
  --circuit_input arg                   Circuit input file
  --public_input arg                    Public params input file

Json circuit description parsing failed: incomplete JSON
Json public_input parsing failed: incomplete JSON
terminate called after throwing an instance of 'boost::wrapexcept<std::invalid_argument>'
  what():  not an object
Aborted (core dumped)
root@5ffc715e5f63:/proof-market-toolchain# /proof-market-toolchain/build/bin/proof-generator/proof-generator
can not read options configuration file '/root/.config/config.ini'
terminate called after throwing an instance of 'boost::wrapexcept<boost::property_tree::ini_parser::ini_parser_error>'
  what():  /root/.config/config.ini: cannot open file
Aborted (core dumped)

The image is built from commit e7ac895, which is based on master branch commit 9008207

proof_tools.py get returns IndexError when proof is not ready

Limit request:	 {
    "_key": "96804968",
    "cost": 10,
    "sender": "nickvolynkin",
    "statement_key": "96079532",
    "status": "created"
}
root@3ba82297fb2d:/proof-market-toolchain# python3 scripts/proof_tools.py get --request_key 96804968
Traceback (most recent call last):
  File "/proof-market-toolchain/scripts/proof_tools.py", line 89, in <module>
    args.func(args=args)
  File "/proof-market-toolchain/scripts/proof_tools.py", line 66, in get_parser
    get(args.auth, args.request_key, args.proof_key, args.file)
  File "/proof-market-toolchain/scripts/proof_tools.py", line 52, in get
    res_json = res.json()[0]
IndexError: list index out of range

Update submodules using https instead of ssh

Stabil-proof-producer

Add CLI registration

./build.sh in Docker errors

Running sh ./build.sh in docker cause the following errors:

CMake Warning (dev) in CMakeLists.txt:
  No project() command is present.  The top-level CMakeLists.txt file must
  contain a literal, direct call to the project() command.  Add a line of
  code such as

    project(ProjectName)

  near the top of the file, but after cmake_minimum_required().

  CMake is pretending there is a "project(Project)" command on the first
  line.
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Error at CMakeLists.txt:26 (include):
  include could not find requested file:

    CMConfig


CMake Error at CMakeLists.txt:27 (include):
  include could not find requested file:

    CMDeploy


CMake Error at CMakeLists.txt:28 (include):
  include could not find requested file:

    CMSetupVersion


CMake Error at CMakeLists.txt:30 (cm_workspace):
  Unknown CMake command "cm_workspace".

Publish a ready-to-work docker image

Publish ready-to-work docker image:

All dependencies installed (apt, pip)
Proof-market-toolchain is precompiled
Correct working directory

Currently, users have to compile the project themselves, which takes a lot of time and reduces the success rate.
This image will help new users start working with Proof Market in several minutes.

Also, we will use this image in CI for checking integration between zkLLVM and Proof Market.

Image names:

# build-base image: all dependencies, but no compiled code
ghcr.io/nilfoundation/proof-market-toolchain:base

# ready-to-use image: all dependencies + compiled toolchain applications
ghcr.io/nilfoundation/proof-market-toolchain:latest

Implement standalone proof verifier

Local verifier is required for many cases, so it makes sense to implement it standalone form similar to the proof-producer (https://github.com/NilFoundation/proof-producer).

Add ccache with preheated cache to the image

Add ccache to the build image. Build proof market in it once from the main branch, and save the cache in the image.

As a result, when folks build the proof market in a container, the build will be blazing fast.

Blocked by #35

add MIT licence

publish docker image to repository

Current docker build requires users to run/build it locally. This task is to track publishing it to the docker repository.

Change terminology: bid/ask to request/proposal

Implement standalone proof producer

It seems that proof-market toolchain would be more convenient to use if the user could easily substitute the proof-producer binaries.
Because many teams will inevitably do custom implementations and optimizations of our prover (including hardware-specific deployments), it may make sense to give them a separate repository, which they could fork and do their job there.

Separate prover can be moved to https://github.com/NilFoundation/proof-producer

Check this issue after proof-producer is separated: NilFoundation/zkLLVM#178

Unified CLI for proof market

Update docs

Wrapper-for-verifier

Proof Producer one-command-run

Use base Docker image for building dev-mode proof producer images

The image built from ./scripts/proof_producer/build_scripts/Dockerfile can use ghcr.io/nilfoundation/proof-market-toolchain:base, built from ./Dockerfile.base. It will improve the build speed, mainly because it skips Boost compilation.

Things to do and check:

Build from the base image, not ubuntu:22.04
Check dependency lists. If there's something missing in the base image, we can put it there.
There's a requests==2.28.2 dependency manually installed in the dev-mode Dockerfile, and one in ./requirements.txt. Let's bring it to consistency and use one source.

Follow-up to #36

update readme

Update PMT readme and PM docs to support the terminology change

Follow-up to #49

Pages that should be updated:

Update the PMT docs structure

Right now we have some structural choices, like folders guides/ and guides-1/, repeating parts and not-so-obvious namings like Terminology, when it's more like Market mechanics. I suggest we update it and make the documentation more structured
https://www.notion.so/nilfoundation/nil-developer-hub-website-f87a3511a04b457b894c16ee7f3d5223?pvs=4#e251cea1ca9a45aebbb54417f041c86a

Unified zkLLVM/Placeholder Vanilla binary

statement_tools get: show statements in a neat table

statement_tools returns statements as a raw JSON output. There should be a way to see a neat table, like with docker image ls or

python3 scripts/statement_tools.py get 

KEY 		DESCRIPTION           			AVG_COST AVG_TIME URL	
79169223	Account state proof for Mina	6.32	 8.8s	  https://github.com/NilFoundation/mina-state-proof
32292		mina state proof				6.42	 9835s	  https://github.com/NilFoundation/mina-state-proof

Restructure readme: prefer containerized workflow

There's no need to go all the way with dependencies in the container-based workflow.

Put containter-based workflow to the top.
Add table of contents.
Run through the workflow with a real developer from the community.

`proof-generator` not working with input strings

Steps to reproduce:

Test File: charcpp_test.cpp

#include <cstddef>
[[circuit]] size_t strlen(const char *str) {
    size_t counter = 0;
    while (str[counter++] != '\0');
    return counter - 1;
}

Input File: charcpp_input.json

{
    "input": "abcdefgh"
}

Commands to reproduce

Getting the circuit with zkLLVM:

/home/robertorosmaninho/rv/zk-experiments/zkllvm/build/libs/circifier/llvm/bin/clang-16 -target assigner -Xclang -no-opaque-pointers -Xclang -fpreserve-vec3-type -std=c++20 -D__ZKLLVM__ -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/crypto3/algebra/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/build/include -I /home/robertorosmaninho/rv/boost_1_76_0/build/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/crypto3/block/include -I /home/robertorosmaninho/rv/boost_1_76_0/build/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/blueprint/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/crypto3/codec/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/crypto3/containers/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/crypto3/hash/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/crypto3/kdf/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/crypto3/mac/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/crypto3/marshalling/core/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/crypto3/marshalling/algebra/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/crypto3/marshalling/multiprecision/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/crypto3/marshalling/zk/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/crypto3/math/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/crypto3/modes/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/crypto3/multiprecision/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/crypto3/passhash/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/crypto3/pbkdf/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/crypto3/pkmodes/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/crypto3/pkpad/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/crypto3/pubkey/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/crypto3/random/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/crypto3/stream/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/crypto3/vdf/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/crypto3/zk/include -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/stdlib/libcpp -I /home/robertorosmaninho/rv/zk-experiments/zkllvm/libs/stdlib/libc/include -emit-llvm -O1 -S charcpp_test.cpp

Getting the circuit statement:

python ./scripts/prepare_statement.py -c=charcpp_test.ll -o=test_statement.json -n=test -t=placeholder-zkllvm

Generate the proof:

./build/bin/proof-generator/proof-generator --proof_out=test_output.bin --circuit_input=test_statement.json --public_input=charcpp_input.json

Current Output from `proof-generator`:

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

Expected output:

generatring zkllvm proof...
Proof is verified

Tools Version

zkLLVM

clang version 16.0.0 (https://github.com/NilFoundation/zkllvm-circifier.git 9e73aecf0f3db71cb1fb433f65d56a63654afd1a)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/robertorosmaninho/rv/zk-experiments/zkllvm/build/libs/circifier/llvm/bin

Proof Market

$> git log
commit 992025ddbf89a4dbd0c7c1512b25704846577daf (HEAD -> master, tag: v0.0.30, origin/master, origin/HEAD)

$> l
total 16K
drwxrwxr-x 4 robertorosmaninho robertorosmaninho 4.0K May 30 13:27 .
drwxrwxr-x 7 robertorosmaninho robertorosmaninho 4.0K Jun 15 16:20 ..
drwxrwxr-x 3 robertorosmaninho robertorosmaninho 4.0K Jun 15 16:25 proof-generator

I couldn't find a way to output the current version, so I just used git log and l.

Building in docker

Docs: describe registering as a proof producer

We don't have documentation for registering as a proof producer.

Pages that should be updated:

https://docs.nil.foundation/proof-market/market/user-guides/proof-producer on how to register - fixed by #71
https://docs.nil.foundation/proof-market/market/cmd-reference/user, add a section on registering as a proof provider
Readme, mention it briefly, and give a link to the docs.

Add blueprint assigner compatibility

Add proof_key in response for bid_tools get.

$ python3 scripts/bid_tools.py get --key=84304538
Limit bid:       {
    "_key": "84304538",
    "cost": 7,
    "sender": "robertogomes",
    "statement_key": "79169223",
    "status": "completed"
}

Currently proof-key is not returned in the response and we should add this to prevent users from using the -v flag.

root@61db058b84db:/proof-market-toolchain# python3 scripts/proposal_tools.py push --key 93846511 --cost 2
Limit proposal:
{
  '_key': '93849751',
  '_id': 'proposal/93849751',
  '_rev': '_gSXAKnm---',
  'statement_key': '93846511',
  'cost': 2,
  'sender': 'nickvolynkin',
  'wait_period': None,
  'eval_time': None,
  'createdOn': 1689100890776,
  'updatedOn': 1689100890776,
  'status': 'created'
}

As you can see, it has status=created. This request must return an error because there is no statement with this id.

Trying to find my proposal right after pushing it:

root@61db058b84db:/proof-market-toolchain# python3 scripts/proposal_tools.py get --key 93846511
Error: 204

93846511 is in fact a request for statement 93843771:

root@61db058b84db:/proof-market-toolchain# python3 scripts/request_tools.py get --key 93846511
Limit request:	 {
    "_key": "93846511",
    "cost": 2,
    "sender": "nickvolynkin",
    "statement_key": "93843771",
    "status": "created"
}

nilfoundation / proof-market-toolchain Goto Github PK

proof-market-toolchain's Issues