hyperledger-archives / aries-cloudagent-loadgenerator Goto Github PK

To further improve the performance of the Issuer/Verifier set up a Postgres cluster.

In order to facilitate performance comparisons and testing of new changes, multiple aca-py versions should be supported, handled by the implementation and selectable by the user (see also #12 ).

Configuration file should allow to define:

• versions of aca-py to be used in the test run
• horizontal scaling (no. of instances per each applicable component)
• test scenario
• load configuration (request rate, or other parameters according to the selected test scenario)
• duration of the test

To ensure that the load generator does not become a bottleneck, monitor the utilization of the Tomcat threads.

Log the size of the issuer/verifier wallet database to the console.

The generation of a default set of data visualizations via the automated processing of logs data and test results should be implemented and documented

The first Load Generator enforces a maximum rate of parallel iterations. Thereby, the load is automatically reduced if the agent slows down over time. Next, we also want a load generator that schedules iterations at a fixed rate per minute.

OOB Proof Requests do not require a connection to be established beforehand.

• Add note to README to clone with submodules
• switch to fixed version for submodules instead of latest commit

Currently, communication between containers from the agent/loadgenerator networks to containers of the dashboard/logging network takes place via localhost of the host. To avoid this, all docker-compose files should use the same network and reference each other using the container names.

Setup

1. Create Schema
2. Create Credential Definition

Iteration

1. Create Connection Invitation
2. Accept Connection Invitation
3. Issue Credential
4. Request Credential

To scale the holder side we switched from one holder to many holders where each holder has its own wallet.

The load is distributed across the holder by the load generator. The initialization procedure for the following test flows still needs to be updated to support multiple holders. Each holder needs to be provisioned individually before starting the load generation.

• Credential Issuance Flow (Issuer/Verifier needs to initiate a connection with each holder during initialization)
• Proof Request Flow (Issuer/Verifier needs to initiate a connection with each holder during initialization AND issue a credential to each holder)

Scheduled = iterations scheduled by the Periodic Schedular
Started = iterations actually initiated by placing a HTTP Call to an AcaPy
FInished = all the iterations that have been finished

Scheduled and Started differs in case the thread pool responsible for executing the scheduled tasks to start iterations is exhausted as all the threads are currently busy trying to start new iterations.

OOB Credential Issuances do not require a connection to be established beforehand.

Components

• VON Network
• 2 AcaPys
• 2 Postgres DBs

Acceptance Criteria

• #6
• #5
• #7
• #8
• #4
  • Architecture Overview / List of Components
  • How to start
  • How to stop/down
  • Test logging setup

• #3
• #7

• Decide on integration strategy (use git subrepository vs pull docker image from a public registry)
• Automate run of forked aca-py and integrate into local deployment

Indy, Askar etc.

Implement a test-runner that simulates the tasks of an issuer.

One Iteration:

1. create a new connection to the holder
2. issue a new credential to the new connection

Try to set up a health check for AcaPy containers to ensure that they are restarted in case they get in trouble.
Maybe, the AcaPy /status endpoints can be used for this.

Log how long the HTTP calls to AcaPy take.

• Setup simple http server based on spring boot (kotlin)
• Create aca-py callback api skeleton
• Create load test api skeleton
• Create dockerfile

Currently, it looks like the actual bottleneck is the Holder AcaPy instead of the Issuer/Verifier AcaPy. As the Holder is not the one that is being load-tested we need to define a way to scale the Holder side to be able to push the Issuer/Verifier to his limits.

For debugging purposes having a horizontally scaled Isssuer/Verifier setup makes it harder to debug the AcaPy. Therefore, we want a single instance setup that can easily be started via the .env configuration file

To avoid connection exhaustion:

1. explicitly define the number of connections the Postgres DB should allow
2. limit the number of connections each AcaPy is allowed to open

Currently, we receive many errors via AcaPy Webhook calls. Maybe, just retrying them would allow us to still process them and therefore make the system more reliable and resiliant.

Use either Redis or Kafka to persist the inbound and outbound message queues.
Further, monitor the queue size in Grafana.

Implement three test-runners that only focus on one task each

1. create one connection per iteration
2. issue one credential per iteration (always to same connection)
3. request and verify one proof per iteration (always request same credential from same connection)

• logs format should be defined, documented and applied to each component
• results should be aggregated and consolidated into a predefined output format

Using an nginx as a load balancer allows us to scale Holder and Issuer/Verifier AcaPys horizontally. Unfortunately, the current setup is very inflexible as the nginx.conf lists all the available Holder and Issuer/Verifier AcaPys.

In the future, it should be possible to specify the number of Holder and Issuer/Verifier dynamically via the .env.

Currently, we do not retry failed HTTP requests to AcaPy. Maybe, retrying them would make the system more resilient and reliable.

Monitor the Issuer/Verifier and Holder Postgres to see if it becomes a bottleneck during the test runs.

Leveraging IzakMarais/reporter the Grafana Dashboard can now be exported as a PDF: f6eba8b

Log the Status Codes of each HTTP request to AcaPy.