primitivefinance / excalibur

Excalibur

Excalibur

Excalibur is an Ethereum Application Client optimized for speed and security.

Use case

Use Excalibur to directly interface with applications deployed on EVM networks.

Building the Windows .msi installer

We use cargo-wix to build Windows installers so Excalibur can be installed on Windows.

1. Create the wix file that is used to build the installer: cargo wix init --package excalibur-app --force
2. Run wix: cargo wix --package excalibur-app --nocapture --name Excalibur. The name flag will make this the application name saved in Program Files.

Building the MacOS .dmg installer

We use cargo-bundle to build MacOS and Linux bundles, but have not tested Excalibur on Linux.

1. Run cargo bundle --target {target}. target should match rust compiler targets.

Ethereum Application Client

Problem

Interacting with smart contracts that are deployed on Ethereum and other EVM blockchains is delivered through a fragmented technology stack written in multiple different languages and often hosted in a browser website. This creates centralization, expands attack vectors, and negatively impacts performance.

Solution

Excalibur is an Ethereum Application Client which vertically integrates all components of the EVM tech stack into one package. This enables end users to directly interface with even the lowest parts of the stack, e.g. Ethereum nodes. It also empowers existing capabilities to be more performant, for example, transaction simulation, fetching blockchain data, and transaction execution.

Journal

You will need to cargo install mdbook and cargo install mdbook-katex to be able to render the journal properly. To see the journal you can run from the root directory:

mdbook serve journal

and visit localhost:3000 in your browser.

Dependencies

• Anvil is installed and available in $PATH.
• Forge is installed and available in $PATH.
• Arbiter is installed and available in $PATH.

Installation part 1: rust

cargo install --path .

Installation part 2: foundry dependencies, artifacts, and bindings

There is one (let's keep it that way!) dependency: primitivefinance/solstat. This has its own dependencies forge-std and solmate, which we also use. Make sure to use forge to install this prior to building the contracts.

make

Run

Application

cargo run ui

Simulation

cargo run simulate <config>

Project layout

• assets/ - Static assets used in application
• benches/ - Benchmarks for all crates
• bin - CLI entrypoint
• configs/ - Simulation configurations
• crates/ - All rust code
• journal/ - Team knowledge corpus
• lib/ - Git submodule dependencies
• src/ - Smart contracts

Architecture

Excalibur is a full-stack client for interacting with EVM blockchain applications in simulated and live environments.

• 100% written in Rust.
• User interface is built using the iced framework.
• Excalibur's simulation engine is powered by Arbiter, a performant & open-source EVM simulation framework.
• Excalibur implements its own simulation management framework and communicates with Arbiter via an arbiter client. When combined, Excalibur is capable of running parallelized agent based simulations for any integrated ABS module.
• Excalibur integrates various RPC client connections to enable live transaction execution in the application, including a "dev" client that runs Anvil instances.

UI Components

Excalibur only has a few underlying primitives that can be chosen from to construct color and text.

Future

Excalibur's fully vertical application design makes it easy to plug in more components of the Ethereum stack. In the future, Excalibur will be able to easily connect to local RETH nodes.

Documentation

if you have mdbook installed you can run

```bash
mdbook serve
```

from the root of the repository to see our living documentation of our though processes and development. This is by no means proffesional or complete more so a compilation of the teams notes and journal throughout the development process.

Telegram

Disclaimer

Excalibur is provided "as is", without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose and noninfringement. In no event shall the authors or copyright holders be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the software or the use or other dealings in the software.

excalibur's Issues

Binary to run sim / cli

Should be very easy to:

• Run the sim from the cli
• Run the sim + analysis from the cli
• Run the analysis from the cli

Generic analysis techniques like Sharpe ratio

There are a collection of classic portfolio analysis techniques that we should integrate into our analyze toolkit

• Sharp Ratio
• Raw Growth
• Max Draw Down
• Volatility
• Momentum

Document + Journal Our Journey

We should take extensive notes to document how the simulation was built.

Maybe we can make it in style of REVM docs that @0xJepsen wrote?

feat: redo the Python plotting in Rust using Plotters

The types of plots we have set up in Python are a good start. We should rebuild them in Rust using the Plotters crate!

Main simulation loop

The main simulation loop is responsible for:

• Initializing the arbiter environment
• Deploying / setting up the state in the contracts
• Starting the loop
• Initializing + stepping the price path
• Triggering the agents
• Logging the data we want
• Shutting down after finishing

Initialize structure and important info initially

Meta issue of documentation for strategy prototyping

generalize price changer

Idea

Allow the price changer to be generic over a type of price process.

More information on priority of analysis tools and stats

Ship it with markdown. Maybe ad it to the Mdbook?

Harden volatility targeting and momentum strategy heuristics

Right now the momentum and volatility targeting strategies only use a very basic heuristic for updating weights (namely change the weight by 0.2% over the course of one day depending on realized volatility and net returns, respectively.) These strategies should account for a. how far from the volatility target the current strategy is (in the case of vol targeting) and b. how negative or positive the net returns are for the period (in the case of momentum). If the we are far from the volatility target, weights should likely update by an amount > 0.2%, and vice versa. If net returns are strongly negative or strongly positive, the change should be > 0.2%, and vice versa.

Optimal arbitrage agent

• Create an agent which handles arbitrage between the liquid exchange and the g3m pools

MDBook meta tracking issue

Document what we are doing, why we are doing. A place for math notes, mathematical descriptions of strategy. Almost like our quant manual for PIAB.

For example: Say we are doing our vol targeting strategfy and we want to analyse it. What kind of analysis should we do? -performance, risk, etc. We can put our results and configs in this.

• #53
• #54

feat: price changer

Is your feature request related to a problem? Please describe.
Create a basic price changing agent hooked up to a liquid exchange. Can mimic the one used in portfolio_simulations

Arbitrageur Math + Agent

Whenever the price of a pool changes, the arbitrageur agent uses some math to determine if they should trade in the pool. The math will compute the trade amounts to try and swap.

Use `PlotSettings` inside of the various plot structs

Now we should use this to be able to adjust the properties of (2D) plots

Parameter changer traits for G3M and RMM

Right now we have a weight changer for vol targeting and momentum. I think it would be good to just make a weight changer trait that can be implemented for those different types of strategies (i.e., they just have a method like compute_new_weights that can be implemented differently).

We can do the same for RMM parameters.

Then we can have a sim file for each strategy contract and the behavior of the parameters is just abstracted into a trait. We can decide which implementation we want to run for that sim in the config or something.

Vol Targeting Rebalance Agent

• Create an agent dedicated to using the smooth rebalancing to update weights on the g3m pools
• The agent should be configured with a target volatility and should have access to functions that calculate trailing volatility (see aera for methodology)

Data logging, transforming, and analyzing

What data we want to end up with:

• Track performance of spot assets vs. LP portfolio value performance
• Total balances & weights of the pool over time
• Realized volatility of the pool vs. assets volatilities
• Net arbitrage profits (lp portfolio value changes, if arbirtageur profits its positive, if LP profits its negative, can be graphed on bar chart).

Notes:

• Pipeline will be rust -> csv -> python
• Make a new directory for "box-analysis", holds the python code for processing the data
• All data generated in sim is exported to csv in ./output

Revenue Model Exploration and Quantitative Analysis

Asses viability of on and off chain fees models for Portfolio Protocol and/or generated Portfolio Strategies. Provide clear, quantitative examples of outcomes under different viable revenue models.

Goals

• Align incentives between Primitive and strategy authors. More AUM in Portfolio Strategies = more revenue for Primitive.
• Exercise strong pricing power. On or off chain revenue models must FORCE large AUM Portfolio Strategy authors to pay us.

Constraints

• As a decentralized protocol, there are limited methods for extracting revenue that scales with the growth of successful Portfolio Strategies.
• Compliance is a key concern, as we must retain discretion on who we collect revenue from. Generalized on-chain fee models are not viable.
• Fees directly implemented into Primitive-generated Portfolio Strategies can be removed by savvy firms after-the-fact by redeploying the strategy.
• Off-chain fees must be reliant on a consistent service or value-add provided by Primitive. This consistent service must be CRITICAL to the long term operation of successful Portfolio Strategies in order to exert pricing power. It is unclear what additional services Portfolio Strategy authors would be willing to pay Primitive a % of AUM for. If a strategy has $1B AUM, taking % management fees for basic tooling such as monitoring or anomaly detection will force the author to build their own stack to avoid paying fees.
• Fixed fees or SaaS subscription pricing does not align incentives with Portfolio Strategy Authors.
• We must be able to provide pricing discounts. Otherwise we prevent experimentation within the platform.

Viable Off-Chain Revenue Models

Commercial Open-Source Licenses on Primitive-generated Portfolio Strategies

Opt-In Risk Management Services for Primitive-generated Portfolio Strategies

Viable On-Chain Revenue Models

TBD

Events for just logging and data analysis.

How does Shadow do this?
This could make our existing tools really powerful.
For development we can make whatever contracts we want.

implement `IStrategy` for RMM

In order to use RMM effectively in simulations, we need the contract to implement the IStrategy interface.

Initial UI for analysis

Idea

Get a basic UI implemented for analysis that let's you pick a file to analyze and which events you want to take in.

feat: create structured output directories for data

When we generate data from a large amount of simulations in parallel over sweeps, we need to organize their output in a usable way.

Add CI for spelling on this

refactor: replace configs with more modular approach

Idea

The configs we use currently don't allow for an easy extension to having new agents or instantiating multiple copies of agents that perform slightly differently.
Also, it would be nice to have configs follow a more standard design pattern.

Momentum based strategy

New strategy prototyping.

Rather then computing a target vol, we would compute a notion of "momentum" and use it to describe the weight updates.

For example, if looking at an average price of an asset over time does it trend up or down -> positive / negative momentum.

What we could do is take the last 30 prices and compute a best fit line and draw correlation between slow and target weights. Would also benefit from time varying process parameters.

Add Liquidity on LP agent broken

Describe the bug
add_liquidity on the liquidity provider only works properly if the trajectory.initial_price is 1. Need to refactor the add_liquidity method to work with variable initial price.

Open and Edit Config (Static_volatilities.toml)

Lets allow these to just be text fields in the UI?
We can type check these.
(Provide dump user bumper rails)

Update arbiter version

Point to arbiter 0.6.0 feature branch

feat: Parallelize simulations

We need to be able to run simulations in parallel, so we should have some sort of way of batching simulations together to be run simultaneously.

analyzer UI parse events by reading the JSON file

We can have the analyzer UI automatically read in the selected file once chosen. Then it can use the reader module in analysis to give a list of the events so they can be displayed and chosen from. Then the event selection should collect a set of vectors of those events.

Possibly to test it out, the next AnalyzerState should display a scrollable list of these vectors and their values.

Meta Tracking issues: Make simulation an actual interactive application

Maybe we start with a CLI and evolve into a more sophisticated UI

Testing Time Varying volatility and new price processes

Ship up a nice CI flow for Excalibur

Should cover

• udeps
• codespell
• clippy
• build
• test

Migrate fee logic to strategies

analysis: volatility targeting strategy

We should generate some basic analysis for the volatility targeting strategy and visualize it.

Improve config ingestion

In order to provide sweeping parameter sets, we need to allow our config/settings/parameters to be more easily tunable for this. We should be able to pass either a fixed value or a range of values for relevant parameter sets.

feat(RMM): Allow for fee parameter to be given in WAD value.

Right now we are doing fees in terms of BPs and we cannot go smaller than one BP. Let's allow ourselves very small fees just so we can see what the right lower bound should be!

feat: label environments, stop one environment upon error, dump logs of errors

When we are running big simulations, if one environment has an error, everything halts. This SUCKS.

Let's do the following:

• Label the environments as we run them, we can use metadata from the simulation itself to provide a meaningful label. E.g., "gbm_drift=0_vol=0.1_trajectory=0".
• Catch an error in the simulation itself inside of the task and end a single simulation early upon receiving this error. Do not bubble up into the multi-sim context.
• When an error is caught, dump as much metadata as possible into a log file that explains which environment failed and where exactly.

Port over python postprocessing code

• Pull in the python framework for postprocessing
• generalize where possible

feat: arbitrageur trait

Is your feature request related to a problem? Please describe.
I will make arbitrageur a trait so that we can have a consistent interface for any strategy we use. I will then bring in the RMM-01 strategies on top of the g3m.

Create `Agents` struct and create a `looper()` function

We should bring the agents into a struct Agent instead of creating a vec in each simulation (e.g., steppers in the dynamic_weights.rs). This struct can have the builder design so agents can be easily added.

We should then pull out the loop used in this simulation since it is already generic. The run() methods for the simulations should be changed to setup and return an object that can be looped over.

Weight changer agent

An agent must be responsible for updating the weights of the portfolio. This needs to emit an event in the contract and an updated price.

@Autoparallel Builds agent
@clemlak Adds event to smart contract + function for weight changes

feat: ingest simulation data back into event structs via bindings

Right now we can ingest data from the JSON output into a nested mapping. Another feature we should add should allow for converting these events into the actual enum/structs that the bindings have. This should be pretty easy given that the bindings have both Serialize and Deserialize for all the structs/enum.

feat: optional seeds

Having seeds is excellent for replayability and debugging. But, if we want to most truly random simulations, we should have the ability to remove seeds. Why?

• E.g., if you use the same seed, say seed==2 for a GBM price path, then that path will only be scaled by drift/volatility but will qualitatively look the same.

Implement time varying volatility with volatility targeting strategy

If we use a default price process like GBM we pick static parameters.

We would like to have time dependent vol, or other dynamic parameters.

Short term: Use rustquant fork
Long term: Get it merged in

Initial UI for managing agents & running sim

Reference codebase: https://github.com/squidowl/halloy/blob/main/src/main.rs

Here's a spec for an initial application delivered via ui/:

• I want to load a config for a sim into application (parse config into some application state)
• I want to choose agents to add to simulation (push/pop/insert hasmap or vec)
• I want to run the simulation (button that runs the sim with all the active agents)

This can be the start then we can work on features like modifying the config, modifying individual agents, running multiple sims, etc.

Geometric Mean Market Maker Contracts

@clemlak is working on these in box-contracts/

Rough api:

• add_liquidity(uint liquidity) ?
• remove_liquidity(uint liquidity, uint min_amount_a, uint min_amount_b) ?
• swap(bool swap_direction, uint amount) ?
• update_weights(uint new_primary_weight) ?
• get_spot_price()
• get_primary_weight()
• create_pool(address token_1, address token_2) can probably put in constructor
• event for each action
• require() with error strings?

Note:

• primary weight is the main weight, secondary weight is 1 - primary weight. i.e. primary weight = 60%, secondary = 40%. Don't need to store secondary weight since it always sums to 1?

Configuration

Config file needs to have all the levers we want to change to minimize the amount of recompiling we want to do.

Rough config:

• ref: https://github.com/primitivefinance/portfolio_simulations/blob/alex/addressing-bugs/config.toml
• price process parameters
• environment parameters
• agent parameters, if any
• pool parameters / weight change parameters
• asset token parameters, should be default
• global sim parameters

Simulation Config Setup

• Look over the g3m contracts to figure out what parameters we should include in the config
• consider dependency contracts and their configuration, e.g. price path parameters for liquid exchange price updates

Deploy new g3m contracts and setup event logging

• Set up the bindings for the g3m contracts
• Update deploy.rs to handle all of the contract and dependency contract deployment logic
• Add event listener with specific filters that we want to listen on