nucypher / nucypher-porter Goto Github PK

Occasionally, I'm getting timeouts:

david@Davids-MacBook-Pro-2 nucypher-contracts % curl -X GET <redacted>/get_ursulas -H "Content-Type: application/json" -d '{"quantity": 3}'
{"result": {"failure_message": "Execution stopped before completion - not enough available values (1 failures recorded)", "failures": [{"value": "0xb15d5A4e2be34f4bE154A1b08a94Ab920FfD8A41", "error": "HTTPSConnectionPool(host='<redacted>', port=9151): Read timed out. (read timeout=2)"}]}, "version": "3.1.1"}%  

It's possible that current default learning delay of 2 seconds is too short. See:

Describe the Bug
A clear and concise description of what the bug is, and what you expected to happen.

To Reproduce
Steps to reproduce the behavior

Traceback or Screenshots (Optional)
If applicable, add the full error message / Traceback or upload screenshots to help explain your problem.

System (please complete the following information):

• OS Version: ubuntu 18.3
• Nucypher Version: 6.0.0
• Ethereum Node Version:

Related to nucypher/nucypher-contracts#240

Current candidate is /taco_decrypt.

Once done - update nucypher-ts accordingly.

Add documentation for /bucket_sampling endpoint

Related to #74

From @piotr-roslaniec :

Just a question: If the decryption request fails, it must be because one of the participants failed. In that case, what do we do on the client side? Do we retry once and then rest our case or is there any other action we could take based on errors contents? At this point, the ritual is complete and validated.

From @derekpierre :

Good question. This is a possible improvement to our error handling. I think we've chatted previously (perhaps there is an issue somewhere) about possibly having integer error codes or something like that as part of the error information returned, then the errors can be contextualized appropriately eg. blip on connectivity vs something more problematic (wrong encrypting key used etc.). A blip in connectivity would signal a retry would possibly be successful.

That is something we can perhaps build upon.

Bucket sampling can also check for:

• operator funds,
• client version,
• block numbers (this one requires client update).

When a new version of nucypher-porter is released, a GitHub action is triggered that creates a new Docker image and uploads it to Docke hub:

https://github.com/nucypher/nucypher-porter/blob/development/.github/workflows/docker.yml

This action can be completed with some code to also create the following docker tags:

• latest
• tapir
• lynx

Porter can check the reservoir of values created to determine whether there are sufficient active nodes to service the get_ursulas request.

https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python

Related to nucypher/nucypher#2703.

If the responses for Alice endpoints /get_ursulas and potentially /revoke can be too large, we can think about paginating the response. See nucypher/nucypher#2666 (comment) for some context.

Related to discussions here - https://github.com/nucypher/tdec/issues/8.

The TreasureMap will be the source of truth for Ursula's encrypting key. The recipient will encrypt an ephemeral key for Ursula to encrypt her /retrieve responses with to prevent Porter from being able to collect decryption fragments during tDEC, and gain access to plaintext data.

Should we bump Porter's version? It is already 1.0.0 on mainnet. If semver, then we aren't introducing breaking API changes for TACo 7.0.0 so technically it would be a 1.1.0, although using 2.0.0 would be refreshing for the TACo 7.0.0 release.

During sampling when the version being restricted represents only a small percentage of the network eg. 8/20 nodes run a version that we are sampling for, then sampling can take a long time. In this case 8/20 nodes running 7.4.1 pre-release, and sampling 8 took ~30s. However, this is a bit of a special case during pre-release testing, but the investigation could lead to overall efficiency improvements nonetheless.

From Discord:

Porter typically connects to each node, gets the json data from the status endpoint - this is the common uptime check - and then if version is restricted, compares the version. This is a bit of a special case because 7.4.1 isn't a widespread version that nodes are running, so getting some number n <= 8 when only 8 / ~120 nodes are running the required version could take some time.

For example, if we want to get all 8 nodes running v7.4.1, the worst case is getting the final 8th node on the 120th node being connected to (i.e. the last node to check). That means that Porter had to connect to all 120 nodes before returning. Now, of course Porter actually does parallel connections using a thread pool, so there is some parallelism that helps to reduce the time - not that it couldn't be further improved of course.

That being said, if even 50% of the network was running 7.4.1, it's very likely the query would take much less time. 

When making a call to the community testnet porter at https://porter-tapir.nucypher.community/retrieve_cfrags with more than ~2-3 retrieval_kits I get a lot of the following errors:

RuntimeError: Ursula ((Ursula)⇀LightBlue Charlie LavenderBlush Mike↽ (0xcbE2F626d84c556AbA674FABBbBDdbED6B39d87b)) seems to be down while trying to complete ReencryptionRequest: <nucypher_core.ReencryptionRequest object at 0x7fc6cc11ac10>

Is this a limitation of the testnet porter?

Also it seems to work much better if I make parallel requests with a single retrieval_kit but not sure if that is intended behavior

Once nucypher/nucypher#3044 is addressed, then Porter should be updated to use the corresponding fixtures.

Should we consider:

1. Having a simple status type page for Porter
2. Monitoring/Alerting type functionality for Porter; perhaps a prometheus type setup.

#32 modified the Dockerfile to install a nucypher dev installation due to issues with the promtheus dependency in nucypher requiring a dev install.

Once that issue is wrapped up in nucypher, nucypher-porter's Dockerfile should go back to using a nucypher regular (non-dev) installation.

(cc @KPrasch , @manumonti )

Continuation of nucypher/nucypher#2703.

Implement /revoke once off-chain revoke is implemented; it is currently TBD after TMapConKFrags implementation (#2687 ) - see https://github.com/nucypher/nucypher/blob/main/nucypher/network/server.py#L278.

Original potential implementation for /revoke can be found here - nucypher/nucypher#2666 (comment). However, this implementation may/may not need to be modified based on what is eventually done for TMapConKFrags off-chain revocation.

There are constants, utilities, and cli option definitions being imported which can be handled locally. Consider the following modules:

porter/cli/help.py

from nucypher.config.constants import USER_LOG_DIR, DEFAULT_CONFIG_ROOT

porter/cli/porter.py

from nucypher.cli.options import (
    option_network,
    option_eth_provider_uri,
    option_federated_only,
    option_teacher_uri,
    option_registry_filepath,
    option_min_stake
)
from nucypher.cli.types import NETWORK_PORT
from nucypher.cli.utils import setup_emitter, get_registry
from nucypher.config.constants import TEMPORARY_DOMAIN

porter/controllers.py

from nucypher.config.constants import MAX_UPLOAD_CONTENT_LENGTH
from nucypher.network.resources import get_static_resources
from nucypher.utilities.concurrency import WorkerPoolException
from nucypher.utilities.emitters import StdoutEmitter
from nucypher.utilities.logging import Logger, GlobalLoggerSettings

Related to:

• #49
• #53

Requires changes from nucypher/nucypher#3337.

The question also is - do we care to provide this for PRE anyway i.e. spend the time to add this functionality to PRE. Currently a default timeout is used which is probably sufficient for PRE given the shift to TACo. (cc @cygnusv , @arjunhassard ).