autonity / autonity-oracle

L1 Data Oracle Server for Autonity

Autonity Oracle Server

Assumptions

This project assumes the following:

• Go 1.19.3
• Linux operating system

Overview

The Autonity Oracle Server offers exchange rate data for currency pairs from various data providers,
consolidating these data points into a standardized report. This information is then pushed to the
oracle contract deployed on the Autonity L1 network via a transaction. The oracle contract aggregates those supplied data points to supply a reference exchange rate data through a voting mechanism using a commit-reveal scheme.
DApps deployed on the Autonity L1 network can access these data points via the oracle contract interface.

The Oracle Server

• The Oracle Server is operated and maintained by an Autonity validator node operator.
• The validator must participate in the Oracle protocol if selected to be part of the consensus committee.
• A proof of ownership for the oracle account is required when submitting a registration transaction to the Autonity Contract.
• Oracle transactions are refunded if successful. The balance of the oracle account is required to be funded to cover at least one voting transaction.

Data adaptors - plugin architecture

The Autonity oracle client provides a unified interface between itself and plugins that adapt data from
different data providers, for example Binance and Coingecko, etc. Any party can build a plugin implementing
this unified interface and so provide an adaptor for any data source on demand. The oracle client will scan and load
plugins from the plugin directory during runtime. Detection of new or changed plugins is dynamic;
no shutdown of the oracle client is required to detect and apply the change.

Coordination of data sampling

Overview

To coordinate data sampling in the oracle network, the L1 oracle contract issues a round event on every vote period (30 ~ 60 blocks). The round event carries a tuple (RoundID, SampleTS, Height, VotePeriod), which tells the oracle servers that on round with ID RoundID, a data sample with timestamp SampleTS is required for the data submission. The Height stands for the start height of the new round, while the VotePeriod stands for the round length of the new round. Thus the oracle server can estimate and manage data pre-samplings for the new round and then pick up the nearest sample referring to the required SampleTS.

Data pre-sampling

To mitigate data deviation caused by the distributed system environment, a data pre-sampling mechanism is employed parameterised by SampleTS and Height log data from the round event. When approaching the next round's start boundary Height, the oracle server initiates data pre-sampling approximately 15 seconds in advance. During this pre-sampling window, the server samples data per second and selects the sample closest to the required SampleTS for data aggregation. The oracle server will then submit that sample to the L1 oracle contract as its price vote for the next oracle voting round.

In a production network, node operators should obtain real-time data from high-quality data sources. However, most commercial data providers price their services based on quality of service (QoS) and rate limits. To address this, a configuration parameter "refresh" has been introduced for each data plugin. This parameter represents the interval in seconds between data fetches after the last successful data sampling. A buffered sample is used before the next data fetch. Node operators should configure an appropriate "refresh" interval by estimating the data fetching rate and the QoS subscribed from the data provider. The default value of "refresh" is 30 seconds, indicating that the plugin will query the data from the data source once every 30 seconds, even during the data pre-sampling window. If the data source does not limit the rate, it's recommended to set "refresh" to 1, allowing the pre-sampling to fetch data every 1 second to obtain real-time data. If the default "refresh" of 30 seconds is kept, then the oracle server will be sampling data up to 30 seconds old rather than in real-time.

Version

Print the version of the oracle server:

$./autoracle version
v0.1.5

Configuration of oracle server

There are three ways to config oracle server.

Oracle Server Configuration File

A template configuration file oracle-server.config is made in the build bin when you build the project, and it is also included in the release package, you can config and start the oracle server with it:

#This is a configuration file for autonity-oracle server.
#Set the gas priority fee cap to issue the oracle data report transactions.
tip 1
#Set oracle server key file
key.file ./UTC--2023-02-27T09-10-19.592765887Z--b749d3d83376276ab4ddef2d9300fb5ce70ebafe
#Set the password to decrypt oracle server key file
key.password 123%&%^$
#Set the logging level, available levels are:  0: NoLevel, 1: Trace, 2:Debug, 3: Info, 4: Warn, 5: Error
log.level 3
#Set the WS-RPC server listening interface and port of the connected Autonity Client node
ws ws://127.0.0.1:8546
#Set the directory of the data plugins.
plugin.dir ./plugins
#Set the plugins' configuration file
plugin.conf ./plugins-conf.yml

Start oracle server with a config file:

$./autoracle --config="./oracle-server.config"

System Environment Variables

A set of system environment variables can be used too to config oracle server:

Env Variable	Required?	Meaning	Default Value	Valid Options
PLUGIN_DIR	Yes	The directory that stores the plugins	"./plugins"	any directory that saves plugins
KEY_FILE	Yes	The encrypted key file path that contains the private key of the oracle client.	"./UTC--2023-02-27T09-10-19.592765887Z--b749d3d83376276ab4ddef2d9300fb5ce70ebafe"	any key file that saves the private key
KEY_PASSWORD	Yes	The password of the key file that contains the private key of the oracle client.	"123"	any password that encrypted the private key
AUTONITY_WS	Yes	The web socket RPC URL of your Autonity L1 Node that the oracle client communicates with.	"ws://127.0.0.1:8546"	the web socket rpc endpoint url of the Autonity client.
PLUGIN_CONF	Yes	The plugins' configuration file in YAML.	"./plugins-conf.yml"	the configuration file of the oracle plugins.
CONFIG	No	Use a configuration file to start oracle server.	""	the configuration file of the oracle server.
GAS_TIP_CAP	No	The gas priority fee cap to issue the oracle data report transactions	1	A non-zero value per gas to prioritize your data report TX to be mined.
LOG_LEVEL	No	The logging level of the oracle server	3	available levels are: 0: NoLevel, 1: Trace, 2:Debug, 3: Info, 4: Warn, 5: Error.

CLI Flags

A set of CLI flags can be used too to config and start oracle server:

$./autoracle --help
Usage of Autonity Oracle Server:
Sub commands: 
  version: print the version of the oracle server.
Flags:
  -config="": Set the oracle server configuration file path.
  -key.file="./UTC--2023-02-27T09-10-19.592765887Z--b749d3d83376276ab4ddef2d9300fb5ce70ebafe": Set oracle server key file
  -key.password="123": Set the password to decrypt oracle server key file
  -log.level=2: Set the logging level, available levels are:  0: NoLevel, 1: Trace, 2:Debug, 3: Info, 4: Warn, 5: Error
  -plugin.conf="./plugins-conf.yml": Set the plugins' configuration file
  -plugin.dir="./plugins": Set the directory of the data plugins.
  -tip=1: Set the gas priority fee cap to issue the oracle data report transactions.
  -ws="ws://127.0.0.1:8546": Set the WS-RPC server listening interface and port of the connected Autonity Client node

example to run the autonity oracle service with console flags:

$./autoracle --plugin.dir="./plugins" --key.file="../../test_data/keystore/UTC--2023-02-27T09-10-19.592765887Z--b749d3d83376276ab4ddef2d9300fb5ce70ebafe" --key.password="123" --ws="ws://127.0.0.1:8546" --plugin.conf="./plugins-conf.yml"

Configuration of plugins:

The configuration of plugins are assembled in a yaml file: A yaml file to config plugins:

# The forex data plugins are used to fetch realtime rate of currency pairs:
# EUR-USD, JPY-USD, GBP-USD, AUD-USD, CAD-USD and SEK-USD from commercial data providers.
# There are 4 implemented forex data plugins, each of them requires the end user to apply for their own service key from
# the selected data provider. The selection of the forex data plugin is on demand by the end user. The user can use anyone
# of them, or he/she can use multiple forex data plugins in the setup.
#
# The crypto data plugins are used to fetch realtime rate of crypto currency pairs:
# ATN-USDC, NTN-USDC, NTN-ATN from exchanges. For the Autonity Piccadilly Circus Games Competition Round 6 game, the data provider of these pairs is a simulated
# exchange that people can trade ATN and NTN in markets created by it. No configuration is required for
# the plugin named pcgc_cax that fetches data for these crypto currency pairs from this exchange as the default configuration is set in the plugin.

# For the forex data plugin, default
# configuration is set, so the end user just needs to configure required settings, namely `name` and `key`. The configuration settings of a plugin are:
#
# type PluginConfig struct {
#	Name               string `json:"name" yaml:"name"`         // the name of the plugin binary, it is required.
#	Key                string `json:"key" yaml:"key"`           // the API key granted by data provider, it is required.
#	Scheme             string `json:"scheme" yaml:"scheme"`     // the data service scheme, http or https, it is optional.
#	Endpoint           string `json:"endpoint" yaml:"endpoint"` // the hostname of the data service endpoint, it is optional.
#	Timeout            int    `json:"timeout" yaml:"timeout"`   // the timeout in seconds that a request last for, it is optional.
#	DataUpdateInterval int    `json:"refresh" yaml:"refresh"`   // the interval in seconds to fetch data due to the rate limit from the provider.
#}

# As an example, to set the configuration of the plugin `forex_currencyfreaks`, only the required field are needed,
# however you can configure the optional fields on demand to fit your service quality from the rate provider.
#  - name: forex_currencyfreaks              # required, it is the plugin file name in the plugin directory.
#    key: 575aab9e47e54790bf6d502c48407c10   # required, visit https://currencyfreaks.com to get your key, and replace it.
#    scheme: https                           # optional, https or http, default value is https.
#    endpoint: api.currencyfreaks.com        # optional, default value is api.currencyfreaks.com
#    timeout: 10                             # optional, default value is 10.
#    refresh: 30                             # optional, default value is 30, that is 30 seconds to fetch data from data source.

# Un-comment below lines to enable your forex data plugin's configuration on demand. Your production configurations starts from below:

#  - name: forex_currencyfreaks              # required, it is the plugin file name in the plugin directory.
#    key: 575aab9e47e54790bf6d502c48407c10   # required, visit https://currencyfreaks.com to get your key, and replace it.

#  - name: forex_openexchangerate            # required, it is the plugin file name in the plugin directory.
#    key: 0be02ca33c4843ee968c4cedd2686f01   # required, visit https://openexchangerates.org to get your key, and replace it.

#  - name: forex_currencylayer               # required, it is the plugin file name in the plugin directory.
#    key: 705af082ac7f7d150c87303d4e2f049e   # required, visit https://currencylayer.com  to get your key, and replace it.

#  - name: forex_exchangerate                # required, it is the plugin file name in the plugin directory.
#    key: 411f04e4775bb86c20296530           # required, visit https://www.exchangerate-api.com to get your key, and replace it.

Available configuration fields:

There are multiple configuration fields can be used, it is not required to config each field of them, it depends on your plugin implementation.

// PluginConfig carry the configuration of plugins.
type PluginConfig struct {
	Name               string `json:"name" yaml:"name"`         // the name of the plugin binary.
	Key                string `json:"key" yaml:"key"`           // the API key granted by your data provider to access their data API.
	Scheme             string `json:"scheme" yaml:"scheme"`     // the data service scheme, http or https.
	Endpoint           string `json:"endpoint" yaml:"endpoint"` // the data service endpoint url of the data provider.
	Timeout            int    `json:"timeout" yaml:"timeout"`   // the timeout period that an API request is lasting for.
	DataUpdateInterval int    `json:"refresh" yaml:"refresh"`   // reserved for rate limited provider's plugin, limit the request rate.
}

In the last configuration file, all the forex data vendors need a service key to access their data, thus a key is expected for the corresponding plugins.

Deployment

Oracle Client Private Key generation

Download the Autonity client to generate the private key from console, and set the password to encode the key file, the
key file path will display, and remember the password that encrypted the key file.

$./autonity --datadir ./keys/ account new  
Your new account is locked with a password. Please give a password. Do not forget this password.  
Password:xxxxxx  
Repeat password:xxxxxx

Your new key was generated

Public address of the key: 0x7C785Fe9404574AaC7daf2FF30637546493900d1  
Path of the secret key file: key-data/keystore/UTC--2023-02-28T11-40-15.383709761Z--7c785fe9404574aac7daf2ff30637546493900d1

- You can share your public address with anyone. Others need it to interact with you.
- You must NEVER share the secret key with anyone! The key controls access to your funds!
- You must BACKUP your key file! Without the key, it's impossible to access account funds!
- You must REMEMBER your password! Without the password, it's impossible to decrypt the key!

Start up the service from shell console

Prepare the plugin binaries, and save them into the plugins directory. To start the service, set the system environment variables and run the binary:

$export PLUGIN_DIR="./plugins"
$export KEY_FILE="./test_data/keystore/UTC--2023-02-27T09-10-19.592765887Z--b749d3d83376276ab4ddef2d9300fb5ce70ebafe"
$export KEY_PASSWORD="your passord to the key file"
$export AUTONITY_WS="ws://127.0.0.1:8546"
$export PLUGIN_CONF="./plugins-conf.yml"
$.~/src/autonity-oracle/build/bin/autoracle

or configure it by using console flags and run the binary:

$./autoracle --plugin.dir="./plugins" --key.file="./test_data/keystore/UTC--2023-02-27T09-10-19.592765887Z--b749d3d83376276ab4ddef2d9300fb5ce70ebafe" --key.password="123" --ws="ws://127.0.0.1:8546"

or configure it by a configuration file: oracle-server.config:

$./autoracle --config="./oracle-server.config"

Deploy via linux system daemon

Preparations

Prepare the configurations via system environment variables and the corresponding plugin binaries. Create a service registration file under your service discovery DIR of the system daemon, for example "/etc/systemd/system/" in Ubuntu Linux.
Here I create a service registration file called "/etc/systemd/system/autoracle.service" with content:

[Unit]  
Description=Clearmatics Autonity Oracle Server  
After=syslog.target network.target  
[Service]  
Type=simple  
ExecStart=/home/test/src/autonity-oracle/build/bin/autoracle -plugin.dir="/home/test/src/autonity-oracle/build/bin/plugins" -plugin.conf="/home/test/src/autonity-oracle/build/bin/plugins/plugins-conf.yml"
KillMode=process  
KillSignal=SIGINT  
TimeoutStopSec=5  
Restart=on-failure  
RestartSec=5  
[Install]  
Alias=autoracle.service  
WantedBy=multi-user.target  

Start the service

sudo systemctl start autoracle.service

Stop the service

sudo systemctl stop autoracle.service
● autoracle.service - Clearmatics Autonity Oracle Server  
Loaded: loaded (/etc/systemd/system/autoracle.service; disabled; vendor preset: enabled)  
Active: inactive (dead)  
  
Jan 19 03:03:45 systemd[1]: Stopping Clearmatics Autonity Oracle Server...  
Jan 19 03:03:45 autoracle[14568]: 2023-01-19T03:03:45.233Z [INFO] *oracleserver.OracleServer: the jobTicker jobs of oracle service is stopped  
Jan 19 03:03:45 autoracle[14568]: 2023-01-19T03:03:45.233Z [DEBUG] binance.binance: 2023/01/19 03:03:45 [DEBUG] plugin: plugin server: accept unix /tmp/plugin3024381010: use of closed network connection  
Jan 19 03:03:45 autoracle[14568]: 2023-01-19T03:03:45.235Z [INFO] binance: plugin process exited: path=/home/user/src/autonity-oracle/build/bin/plugins/binance pid=14577  
Jan 19 03:03:45 autoracle[14568]: 2023-01-19T03:03:45.235Z [DEBUG] binance: plugin exited  
Jan 19 03:03:45 autoracle[14568]: 2023-01-19T03:03:45.236Z [DEBUG] fakeplugin.fakeplugin: 2023/01/19 03:03:45 [DEBUG] plugin: plugin server: accept unix /tmp/plugin2424636505: use of closed network connection  
Jan 19 03:03:45 autoracle[14568]: 2023-01-19T03:03:45.237Z [INFO] fakeplugin: plugin process exited: path=/home/user/src/autonity-oracle/build/bin/plugins/fakeplugin pid=14586  
Jan 19 03:03:45 autoracle[14568]: 2023-01-19T03:03:45.237Z [DEBUG] fakeplugin: plugin exited  
Jan 19 03:03:45 systemd[1]: autoracle.service: Succeeded.  
Jan 19 03:03:45 systemd[1]: Stopped Clearmatics Autonity Oracle Server.  
  

Check the runtime status

sudo systemctl status autoracle.service
  
● autoracle.service - Clearmatics Autonity Oracle Server  
Loaded: loaded (/etc/systemd/system/autoracle.service; disabled; vendor preset: enabled)  
Active: active (running) since Thu 2023-01-19 02:42:19 GMT; 15min ago  
Main PID: 14568 (autoracle)  
Tasks: 34 (limit: 18690)  
Memory: 25.4M  
CGroup: /system.slice/autoracle.service  
├─14568 /home/user/src/autonity-oracle/build/bin/autoracle -plugin.dir=/home/user/src/autonity-oracle/build/bin/plugins
├─14577 /home/user/src/autonity-oracle/build/bin/plugins/binance  
└─14586 /home/user/src/autonity-oracle/build/bin/plugins/fakeplugin  
  
Jan 19 02:57:39 autoracle[14568]: 2023-01-19T02:57:39.155Z [DEBUG] fakeplugin.fakeplugin: receive request from oracle service, send data response: timestamp=2023-01-19T02:57:39.154Z  
Jan 19 02:57:59 autoracle[14568]: 2023-01-19T02:57:59.156Z [DEBUG] fakeplugin.fakeplugin: receive request from oracle service, send data response: timestamp=2023-01-19T02:57:59.156Z  
  

Collect system logs

sudo journalctl -u autoracle.service -b

-- Logs begin at Sat 2022-11-26 11:54:00 GMT, end at Thu 2023-01-19 02:59:51 GMT. --  
Jan 19 02:42:19 systemd[1]: Started Clearmatics Autonity Oracle Server.  
Jan 19 02:42:19 autoracle[14568]: 2023/01/19 02:42:19  
Jan 19 02:42:19 autoracle[14568]: Running autonity oracle service at port: 30311, with plugin diretory: /home/user/src/autonity-oracle/build/bin/plugins
Jan 19 02:42:19 autoracle[14568]: 2023-01-19T02:42:19.152Z [WARN] binance: plugin configured with a nil SecureConfig  
Jan 19 02:42:19 autoracle[14568]: 2023-01-19T02:42:19.152Z [DEBUG] binance: starting plugin: path=/home/user/src/autonity-oracle/build/bin/plugins/binance args=[/home/uesr/src/autonity-oracle/build/bin/plugins/binance]  
Jan 19 02:42:19 autoracle[14568]: 2023-01-19T02:42:19.152Z [DEBUG] binance: plugin started: path=/home/user/src/autonity-oracle/build/bin/plugins/binance pid=14577  
Jan 19 02:42:19 autoracle[14568]: 2023-01-19T02:42:19.152Z [DEBUG] binance: waiting for RPC address: path=/home/user/src/autonity-oracle/build/bin/plugins/binance  
Jan 19 02:42:19 autoracle[14568]: 2023-01-19T02:42:19.159Z [DEBUG] binance.binance: plugin address: network=unix address=/tmp/plugin3024381010 timestamp=2023-01-19T02:42:19.159Z  
Jan 19 02:42:19 autoracle[14568]: 2023-01-19T02:42:19.159Z [DEBUG] binance: using plugin: version=1  
Jan 19 02:42:19 autoracle[14568]: 2023-01-19T02:42:19.161Z [INFO] binance: plugin initialized: binance=v0.0.1  
Jan 19 02:42:19 autoracle[14568]: 2023-01-19T02:42:19.161Z [WARN] fakeplugin: plugin configured with a nil SecureConfig  
Jan 19 02:42:19 autoracle[14568]: 2023-01-19T02:42:19.161Z [DEBUG] fakeplugin: starting plugin: path=/home/user/src/autonity-oracle/build/bin/plugins/fakeplugin args=[/home/user/src/autonity-oracle/build/bin/plugins/fakeplugin]  
Jan 19 02:42:19 autoracle[14568]: 2023-01-19T02:42:19.161Z [DEBUG] fakeplugin: plugin started: path=/home/user/src/autonity-oracle/build/bin/plugins/fakeplugin pid=14586  
Jan 19 02:42:19 autoracle[14568]: 2023-01-19T02:42:19.161Z [DEBUG] fakeplugin: waiting for RPC address: path=/home/user/src/autonity-oracle/build/bin/plugins/fakeplugin  
Jan 19 02:42:19 autoracle[14568]: 2023-01-19T02:42:19.168Z [DEBUG] fakeplugin.fakeplugin: plugin address: address=/tmp/plugin2424636505 network=unix timestamp=2023-01-19T02:42:19.167Z  
Jan 19 02:42:19 autoracle[14568]: 2023-01-19T02:42:19.168Z [DEBUG] fakeplugin: using plugin: version=1  
Jan 19 02:42:19 autoracle[14568]: 2023-01-19T02:42:19.170Z [INFO] fakeplugin: plugin initialized: fakeplugin=v0.0.1  
Jan 19 02:42:29 autoracle[14568]: 2023-01-19T02:42:29.156Z [DEBUG] fakeplugin.fakeplugin: receive request from oracle service, send data response: timestamp=2023-01-19T02:42:29.156Z  
Jan 19 02:43:19 autoracle[14568]: 2023-01-19T02:43:19.156Z [DEBUG] fakeplugin.fakeplugin: receive request from oracle service, send data response: timestamp=2023-01-19T02:43:19.156Z  

Runtime plugin management

Adding new plugins

To add a new data source, just put the new plugin into the service's plugins directory. The oracle service auto discovers and manages it. There are no other operations required from the operator.

Replace running plugins

To replace running plugins with new ones, just replace the binary in the plugins directory. The oracle service auto discovers it by checking the modification time of the binary and does the plugin replacement itself. There are no other operations required from the operator.

Development

Build for Bakerloo net

make autoracle-bakerloo

Build for Piccadilly net

make autoracle

Other build helpers

To build the data source simulator run

make simulator

To run e2e test use

make e2e-test

To run all tests use

make test

To lint code run

make lint

To generate mocks for unit test

make mock

To build docker image

make build-docker-image

To build a plugin, please refer to How to build a plugin

Built binaries are presented at: ./build/bin under which there is a plugins directory for the built plugins as well.

autonity-oracle's Issues

Games R4 feedback

I have setup a oracle server running along a validator node. I have some feedback and some questions on the oracle. @Jason-Zhangxin-Chen

questions

I launched my oracle server, however from the logs I cannot understand if it is actually fetching the prices or if it is stuck. Does the node print some logs if it is fetching the price correctly?

When I try to terminate the process with CTRL+C, the oracle is unresponsive, thus I believe there is some issue going on.

oracle version: built from commit 84ec4ed9c58aca34e10638200ce5ccfa3161d049

command: ./autoracle -oracle_key_file="${HOME}/.autonity/keystore/UTC--2023-11-20T12-51-55.981678000Z--ffcfb6811b81eef05c86dcbf99136bbf3d4f656b" -oracle_key_password="" -oracle_autonity_ws_url="ws://127.0.0.1:8546" -oracle_plugin_conf="${HOME}/plugins/plugins-conf.yml" -oracle_plugin_dir="${HOME}/plugins" 2>&1 | tee oracle.log

log (taken around 17:13, so 10 minutes passed from the last log print): oracle.log

Edit: I was using the wrong version of the autonity oracle, with the v0.12.1 version the price reporting works.

Following feedback is still valid.

feedback

If launched without any arguments, the oracle client panics with error:

panic: invalid key file: ./test_data/keystore/UTC--2023-02-27T09-10-19.592765887Z--b749d3d83376276ab4ddef2d9300fb5ce70ebafe

It would be better to handle failure gracefully / print the help text.

Some feedback on the oracle options (e.g. -oracle_key_file, -oracle_key_password, etc.)

• it would be better to have the -- prefix for long names and the - for the abbreviations (or do not support abbreviations).
• I think that theoracle prefix to all the options can be removed, it is redundant.
• we should probably substitute the _ in the options with . (that's the convention we use in the autonity client) and try to keep the options keywords as concise as possible, to facilitate usage.
• I think that the default values might have potential for improvement. Right now they make sense if you compiled from source (e.g. default value for -oracle_plugin_dir="./build/bin/plugins"). However I think that we have to consider how people will get the oracle and the plugins binary, and set the default values accordingly. Do we expect them to just download the pre-compiled binaries from the release? Then I would set the default values according to the folder structure of the autonity-oracle.tgz attached to the release.

Some feedback on the sample config file:

• Are those api keys real?
• is there a reason for setting the default refresh period to 3600s?

Additionally, It would be nice to have:

• a --version option to retrieve the oracle version
• a verbosity option to set the verbosity of the logger

README - error in system env variable names - `GAS_TIP_CAP` or `TIP`?

Summary

The CLI flag name for the tip config property is tip. The section https://github.com/autonity/autonity-oracle?tab=readme-ov-file#system-environment-variables has GAS_TIP_CAP.

Should that be TIP, or is this just inconsistent naming?

Build tags for dev-net and Piccadilly and refine the default settings.

• Replace the service key in the example of plugin-conf.yml
• Replace the default update interval from 3600s to 30s for plugins.
• Introduce build tags for CAX plugin for dev net and Piccadilly network.

Add Package for Docker image

Is your feature request related to a problem? Please describe.
There is a Package for the AGC Docker at https://github.com/orgs/autonity/packages?repo_name=autonity, and statistics for downloads can be seen at https://github.com/autonity/autonity/pkgs/container/autonity

Describe the solution you'd like
An equivalen Package for the AOS Docker.

Describe alternatives you've considered
None.
Additional context
None

Remove the `key.password` option

Passing account keyfile passwords as command line options is poor security practice. And while it is true that a user can avoid this by storing the pwd in an environment variable (eg key.password=$(ORACLE_PWD)), the readme and --help document the poor practice of passing plain text pwds on the command line.

I recommend removing key.password and have the autoracle look for the password in a specific environment variable. If no such variable is found, prompt user for the pwd at runtime (using a terminal library that doesn't echo the user input).

Override default refresh rate to 1 hour in example config

Default values for the oracle have now been set to a lower refresh period which are too low for some of the free tiers in the forex APIs we are supplying.

I propose that in the example config (that most people will likely use as the Docs links to it) we include an entry to override the default refresh rate to 1 hour, in order to allow validators to use the free tiers without hitting their API limits.

I.e. in this file https://github.com/autonity/autonity-oracle/blob/master/config/plugins-conf.yml the entries follow a pattern something like:

#  - name: forex_currencyfreaks              # required, it is the plugin file name in the plugin directory.
#    key: 175aab9e47e54790bf6d502c48407c10   # required, visit https://currencyfreaks.com to get your key, and replace it.
#    refresh: 3600                           # optional, recommended for testnets in order to not exceed the free tier API limits.

Logging improvements

Suggestions to improve the logs output, these are not critical but very helpful for the end-user. I've been working on that in the autonity client actually as it wasn't great...

1. Set default mode to INFO. The DEBUG mode is reserved for developers and troubleshooting only. If a DEBUG log is useful in normal mode operations to the end-user, promote it then to INFO.
2. Remove duplicated/redundant logs:

   2023-12-07-07T13:16:18.666Z [ERROR] forex_exchangerate: fetch prices: error="data is not available"
   2023-12-07-07T13:16:18.666Z [ERROR] forex_exchangerate: fetch price routine: error="data is not available"
   

Seems like those two log outputs can be combined into one.

3. Remove unnecessary logs

   2023-12-07-07T13:16:18.594Z [WARN] pcgc_cax: find bad symbols: : EXTRA_VALUE_AT_END=[AUD-USD, CAD-USD, EUR-USD, GBP-USD, JPY-USD, SEK-USD]
   

   This seems to be expected that pgcg is not capable of returning forex symbols so there is nothing to alert the end-user. If you think that this log is still useful then I suggest demoting it to DEBUG.
4. Make sure user-facing logs are minimally intelligible.
   Going back to the previous example: "find bad symbols" is weird. First, it would be much better to say "found bad symbols" or "bad symbols found" and secondly there is no such thing as a "bad" symbol. We want to mention that we couldn't retrieve the prices for those symbols, so then some better formulations would be "prices can't be retrieved", "price feed not available", "symbols not supported" etc... @cmjc actually your input would be great here. Also EXTRA_VALUE_AT_END is not understandable and using variable names in the log output should only be at the DEBUG level.
5. Make sure to use the appropriate log level. Error will always raise concerns and Warn is always very suspicious. Use them correctly only when needed.

Password error oracle.key the password contained a character

Error after starting oracle service

systemctl restart antoracle && journalctl -u antoracle -f -o cat
-plugin.dir="./plugins": Set the directory of the data plugins.
-symbols="AUD-USD,CAD-USD,EUR-USD,GBP-USD,JPY-USD,SEK-USD,ATN-USD,NTN-USD,NTN-ATN": Set the symbols string separated by comma
-tip=1: Set the gas priority fee cap to issue the oracle data report transactions.
-ws="ws://127.0.0.1:8546": Set the WS-RPC server listening interface and port of the connected Autonity Client node
antoracle.service: Main process exited, code=exited, status=1/FAILURE
antoracle.service: Failed with result 'exit-code'.
antoracle.service: Specifier '%R' used in unit configuration, which is deprecated. Please update your unit file, as it does not work as intended. antoracle.service: Specifier '%R' used in unit configuration, which is deprecated. Please update your unit file, as it does not work as intended.
Stopped Autonity Oracle Server.
Started Autonity Oracle Server.
2023/12/22 11:05:53 Cannot decrypt keyfile: /root/.autonity/keystore/oracle.key, with the provided password!
Usage of Autonity Oracle Server:
Sub commands:
version: print the version of the oracle server.
Flags:
-key.file="./UTC--2023-02-27T09-10-19.592765887Z--b749d3d83376276ab4ddef2d9300fb5ce70ebafe": Set oracle server key file

The oracle.key password contains the % symbol; systemd does not parse it correctly; you need to specify only letters and numbers for the password.

Password is correct after command I enter password and can see ethkey check --private /root/.autonity/keystore/oracle.key Public_key and Private_key.

systemd parses password incorrectly

System:

*Ubuntu 22.04.3 LTS
*aut version 0.3.0.dev4

Discover and launch the plugins of configured ones in the plugin-confs.yml

Currently, forex plugins that are missing a service key from the plugin-confs.yml are still being started by the oracle server. However, these plugins do not work until the user configures the key granted by the data provider. Therefore, it would be preferable to have a new feature that can start those plugins only when the required key is provided.

The best option is, to let the plugin to state the requirement of a key since not all of the plugins require service key granted by their data source, some data providers like Binance provide free data. Thus, in the oracle server side, it can decide to launch those plugins with configured key that is stated as required.

Reports with commitment hash and no data

Summary

When the oracle service starts, plugins are loaded properly followed by the following logs:

2024-01-26T14:59:51.630Z [INFO]  *oracleserver.OracleServer: handle new round: round=134372 required sampling TS=1706281191 height=4031130 round period=30
2024-01-26T14:59:51.636Z [INFO]  *oracleserver.OracleServer: cannot find last round's data, reports with commitment hash and no data

With debug logs:

2024-01-26T15:03:46.301Z [DEBUG] *oracleserver.OracleServer: latest round price: round=134377 symbol=NTN-ATN price=10.1256281 status=0
2024-01-26T15:03:46.301Z [INFO]  *oracleserver.OracleServer: cannot find last round's data, reports with commitment hash and no data
2024-01-26T15:03:46.301Z [DEBUG] *oracleserver.OracleServer: skip reporting since client is no long a voter, and have no last round data.

These continues on loop round-by-round with no other data.

Preconditions

List all the relevant information about the runtime environment. This can include:

Ubuntu 22
Autonity Oracle v0.1.5
go version go1.20.1 linux/amd64

Steps to Reproduce

Install Autonity Oracle from source, setup systemd service file, start.

Expected Result

Logs that indicate the opposite of no data for last round.

Actual Result

Refer to Summary above.

Additional Information

Configuration options - should `SYMBOLS` be deprecated?

Context

At present the SYMBOLS for a testnet are set in the client in the config, e.g.

https://github.com/autonity/autonity/blob/4504e8acccca8ab65f8aba63a2674f51bc6250e6/params/protocol_contracts.go#L23C1-L23C1

OracleInitialSymbols       = []string{"AUD-USD", "CAD-USD", "EUR-USD", "GBP-USD", "JPY-USD", "SEK-USD", "ATN-USD", "NTN-USD", "NTN-ATN"}

This default oracle config is then picked up by the oracle server, which is why it does not need to be set in your config when you are running the oracle server. I.E.: it is not required, which is clear in the README under Configuration:

Values that can be configured by using environment variables:

Env Variable	Required?	Meaning	Default Value	Valid Options
SYMBOLS	No	The symbols that the oracle component collects data points for	"AUD-USD,CAD-USD,EUR-USD,GBP-USD,JPY-USD,SEK-USD,ATN-USD,NTN-USD,NTN-ATN"

Question

• What is the scenario for needing to set SYMBOLS in config? What is or are the scenario(s) where an oracle server operator would want to set symbols in their local config? The symbol set is auto-detected by the oracle server, so it's not needed for connection to a Public TestNet.
• When would symbols be different? As a validator you supply all not some of the symbol set.
• Local Testnet. To set your own Symbols and ASM config in a local testnet makes sense, but is there really a scenario where this would be realistic?