cogment / cogment-verse

Research platform for Human-in-the-loop learning (HILL) & Multi-Agent Reinforcement Learning (MARL)

cogment

Cogment enables AI researchers and engineers to build, train, and operate AI agents in simulated or real environments shared with humans. Developed by AI Redefined, Cogment is the first open source platform designed to address the challenges of continuously training humans and AI together. For the full user documentation visit https://docs.cogment.ai

This repository includes the the main Cogment module, a multi-platform stand alone CLI including:

• The orchestrator service, the heart of Cogment, it executes the trials involving actors and environments by orchestrating the different user implemented services.
• The trial datastore service, that is able to store and make available the data generated by the trials.
• The model registry service, that let's user store AI models and make them available to actor implementations during training and in production.
• The init tool to bootstrap cogment project locally (Deprecated).
• The run tool to define and run commands within a cogment project (Deprecated).

Installation

Standalone binary (preferred)

Download the install script and make sure you can run it using

curl --silent -L https://raw.githubusercontent.com/cogment/cogment/main/install.sh --output install-cogment.sh
chmod +x install-cogment.sh

Download and install the latest final version using

sudo ./install-cogment.sh

Other installation options are available using ./install-cogment.sh --help

For futher installation options please refer to cogment's installation guide.

Docker image

docker pull cogment/cogment

Developers

This repository is organized in 3 different packages grouped the packages folder:

• grpc_api is the gRPC api, consisting of protocol buffers .proto definition files,
• orchestrator is the orchestrator library, developed in C++,
• cli is the host executable, developed in Go, it integrates the orchestrator and includes the other services and tools.

A CMake based systems glues the build of both together

Prerequisites

• Fully working c++ build toolchain.
• Fully working go setup (1.20), as described in the official documentation.
• Cmake (>= 3.10), the core of the build system, it should be installed as described in the official documentation.
• Make, most flavor should work fine.
• Optional, golangci-lint (v1.53.3) used to run lint on the golang codebase, installed as described in the official documentation.
• Optional, clang-format (10) used to run lint on the c++ codebase, it can be installed on most linux flavor from https://apt.llvm.org and on mac using brew install clang-format.
• Optional, a docker installation to be able to build the docker image.

Build

At the root of the repository you'll find the following scripts:

• build_docker.sh builds the docker image.
• build_linux.sh builds the linux amd64 binary.
• build_macos.sh builds the macos amd64 & arm64 binaries.
• build_windows.bat builds the windows amd64 binary.
• build_all_no_orchestrator.sh builds the binary for all supported platforms (linux/amd64, macos/amd64, macos/arm64 and windows/amd64) without the embedded orchestrator.

The build results are store in the ./install directory per platform.

Those scripts run CMake and create a build directory, per platform, in the ./build directory. Further build target are available there, especially for testing of code formatting.

Formatting & coding style

• orchestrator_lint and orchestrator_fix_lint respectively check and fix the code formatting of the c++ orchestrator codebase using clang-format.
• cli_lint and cli_fix_lint respectively check and fix the code formatting of the go cli codebase using golangci-lint

./build_linux.sh lint or ./build_macos.sh lint can be used as a shortcut to run both linters.

Test

• cli_test runs a suite of tests over the go cli codebase.

./build_linux.sh test or ./build_macos.sh test can be used as a shortcut to run tests.

Benchmark

• cli_benchmark runs a suite of benchmarks over the go cli codebase.

Release process

People having maintainers rights of the repository can follow these steps to release a version MAJOR.MINOR.PATCH. The versioning scheme follows Semantic Versioning.

1. Run ./scripts/create_release_branch.sh MAJOR.MINOR.PATCH to create the release branch and update the version of the package,
2. On the release branch, check and update the changelog if needed,
3. Make sure everything's fine on CI,
4. Run ./scripts/tag_release.sh MAJOR.MINOR.PATCH to create the specific version section in the changelog, merge the release branch in main, create the release tag and update the develop branch with those.

The rest, publishing the package to dockerhub and updating the mirror repositories, is handled directly by the CI.

cogment-verse's Issues

Integrate agents from Dopamine

Objective

• Integrate new agent implementation relying on dopamine / flax

Introduce nightly run end-to-end testing

Objective

• Have the ability to detect regressions by periodically launch run and compare result with reference value

Add documentation for how to write your own agent / agent adapter

Objective

• Simple documentation describing how to implement a working agent relying on either pytorch or tensorflow (potentially flax)

Migrate to cogment 2.2

TD3

Integrate new version of Hive

Objective

• Be up to date with latest development on Hive

Asymmetric Self Play with pybullet driving environment

Integrate Pybullet Driving Environment
Implement ASP with one Alice and Bob agent
Test ASP implementation

Implement µ0

Objective

• Have a fully operational implementation of µ0 that properly leverages GPU

Fix multiprocessing issue with linux and torch

Support additional environment in the web-client

Objective

• We should fully support at least Lunar Lander (done), Cartpole and the gym atari games

Support discrete actions with PPO

Invalid actions received from webclient

Possible bug discovered in muzero branch where we receive actions from the teacher/observer even though the player override is not set.

Update to Cogment 2.0

Web client support for games with one human as a player

Objective

• Ability to play multiplayer game (e.g. connect 4) having one player being a human.

No trial found by trial datastore

Traceback

-4a2d-9c8f-1eeae30d90fb\" found" grpc.code=Unknown grpc.method=RetrieveSamples grpc.service=cogment.TrialDatastoreSP grpc.start_time="2021-1
1-22T16:13:10Z" grpc.time_ms=0.085 peer.address="172.19.0.11:48948" span.kind=server system=grpc                                            
torch_agents_1     | ERROR:asyncio:Task exception was never retrieved                                                                       
torch_agents_1     | future: <Task finished coro=<RunSession._start_trials.<locals>.trials_samples_listener() done, defined at /base_python/
cogment_verse/run/run_session.py:94> exception=<AioRpcError of RPC that terminated with:                                                    
torch_agents_1     |    status = StatusCode.UNKNOWN                                                                                         
torch_agents_1     |    details = "no trial "075acedd-0687-4a2d-9c8f-1eeae30d90fb" found"                                                   
torch_agents_1     |    debug_error_string = "{"created":"@1637597590.904118764","description":"Error received from peer ipv4:172.19.0.3:900
1","file":"src/core/lib/surface/call.cc","file_line":1066,"grpc_message":"no trial "075acedd-0687-4a2d-9c8f-1eeae30d90fb" found","grpc_statu
s":2}"                                                                                                                                      
torch_agents_1     | >>                                                                                                                     
torch_agents_1     | Traceback (most recent call last):                                                                                     
torch_agents_1     |   File "/base_python/cogment_verse/run/run_session.py", line 115, in trials_samples_listener                           
torch_agents_1     |     async for sample in sample_generator():                                                                            
torch_agents_1     |   File "/base_python/cogment_verse/trial_datastore_client.py", line 38, in sample_generator                            
torch_agents_1     |     async for rep_msg in rep_stream:                                                                                   
torch_agents_1     |   File "/usr/local/lib/python3.7/site-packages/grpc/aio/_call.py", line 321, in _fetch_stream_responses                
torch_agents_1     |     await self._raise_for_status()                                                                                     
torch_agents_1     |   File "/usr/local/lib/python3.7/site-packages/grpc/aio/_call.py", line 232, in _raise_for_status                      
torch_agents_1     |     self._cython_call.status())                                                                                        
torch_agents_1     | grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:                                                  
torch_agents_1     |    status = StatusCode.UNKNOWN                                                                                         
torch_agents_1     |    details = "no trial "075acedd-0687-4a2d-9c8f-1eeae30d90fb" found"                                                   
torch_agents_1     |    debug_error_string = "{"created":"@1637597590.904118764","description":"Error received from peer ipv4:172.19.0.3:900
1","file":"src/core/lib/surface/call.cc","file_line":1066,"grpc_message":"no trial "075acedd-0687-4a2d-9c8f-1eeae30d90fb" found","grpc_statu
s":2}"                                                                                                                                      
torch_agents_1     | >   

Seems to be a race condition or an eviction issue in the trial datastore, and is exacerbated by increasing the number of parallel trials.

The option keyboard

Implement the option keyboard

Ability to start/list/terminate/"join" runs from the web client

Objectives

• Reproduce the features of the client in the web_client -> start/list/terminate runs
• Have the ability to select from which run you want to join trials

Usage

• Introduce a "welcome" screen that lists ongoing run
• Each listed run can be terminated
• There is a button to start a run that allows you to pick a params from the list defined in params.yaml (perhaps through an intermediate screen/popin/modal
• Each run having pending trials involving human players be joined, it opens the existing screen, only trials from this run are player

Robosuite environment integration

Option-critic architecture

Implement and potentially extend it to attention option critic or related ideas

Split the `run_params.yaml`

Add support for CNN on µzero

• Add model card and basic training results

Polish the web client UI

Objectives

Prettier UI and better code

Streamline the process orchestration to provide clear entry point for algorithm implementation

Validate run params hyperparameters for convergence

Objective

• Validate hyperparameters for the run params in the repository

Targeted environment are:

• Cartpole
• Pendulum
• Lunar Lander
• Lunar Lander Continuous
• One environment on MinAtar (Breakout)
• One environment on Atari (Breakout)
• Connect 4

Integrate HuggingFace / SB3 agents

Objective is to have SB3 agents that can be started from saved model on the hugging face hub cf. https://github.com/huggingface/huggingface_sb3/blob/main/Stable_Baselines_3_and_Hugging_Face_🤗_tutorial.ipynb

Showcase how to do validation in a run implementation

Implement DAGGER imitation learning algorithm

Implement a visualization run impl

Objective

• Ability to launch that uses any agent using an already trained model and visualize what it does in the web-client

Support mixed discrete / continuous actions with PPO

Showcase using a priority replay buffer

Port REINFORCE to next

Implement semantic data in grpc API for web

Implement further HILL techniques

Objective

• Propose several standard HILL training techniques out of the box

Use viztracer in a preferred way

Hi,

I'm the author of VizTracer and I happened to see your project :) I'm very glad that you are using VizTracer to profile the project and recommend it to the project developers.

I saw that you are using html as the example output format, which was kind of deprecated at this time. The html standalone front-end, catapult, is deprecated by Google, and the official front-end of VizTracer has moved to Perfetto, the latest tracing UI with much better features. Due to many reasons, Perfetto does not support standalone html output, so the recommended way to use VizTracer is:

viztracer -o /path/to/report.json my_script.py 
vizviewer /path/to/report.json

I hope that you can consider using this updated way as the example, and please let me know if you have any thoughts/recommendations/ideas of VizTracer.

Tian

Add documentation and tooling for Singularity deployments

Add simple smoke test

Add off policy for continuous actions (SAC)

next branch issues running on Ubuntu 20.04

The +experiment=simple_dqn/cartpole crashes with the following error on Ubuntu:

[2022-08-29 12:55:38,295][cogment_verse.app][INFO] - Run starting...
[2022-08-29 12:55:38,328][cogment_verse.processes.run][INFO] - Starting run [pedantic_hofstadter_0] from [actors.simple_dqn.SimpleDQNTraining]
[2022-08-29 12:55:38,727][cogment_verse.mlflow_experiment_tracker][INFO] - Experiment with name '/actors.simple_dqn.SimpleDQNTraining' not found. Creating it.
[2022-08-29 12:55:41,056][cogment_verse.processes.run][ERROR] - Error while executing run [pedantic_hofstadter_0] from [actors.simple_dqn.SimpleDQNTraining]
Traceback (most recent call last):
File "/home/vahid/Desktop/cogment-verse/dev_dagger/cogment_verse/processes/run.py", line 64, in run_main_async
await run.impl(run_session)
File "/home/vahid/Desktop/cogment-verse/dev_dagger/actors/simple_dqn.py", line 367, in impl
run_session.log_metrics(trial_idx, total_reward=total_reward)
File "/home/vahid/Desktop/cogment-verse/dev_dagger/cogment_verse/run/run_session.py", line 84, in log_metrics
self._xp_tracker.log_metrics(step_timestamp=int(time.time() * 1000), step_idx=self._step_idx, *args, **kwargs)
TypeError: log_metrics() got multiple values for argument 'step_timestamp'

I have included the changes from pytorch_multiproc_fix branch.

I also see inconsistent behaviour with the +experiment=simple_a2c/cartpole. It sometimes runs fine, and sometimes stalls after running a bunch of epochs with no error.

Multiprocessing problem on ubuntu when there is two sets of trial in a training run

Linked to #72

Document how to add a custom "gym" environment

Objective

• Document how anyone can add an environment following the gym api

Make lunar lander continuous work in the web client

PPO implementation with continuous action space

Implement a TD3 agent

Objective

• Have a TD3 implementation fully operational and tested on several environments.

Define and document good hyper parameters for SimpleDQN and SimpleA2C

Add a transition screen between trials

The objective is to have some time between each trial so that the human player notices the change

Converge module error

RUN_PARAMS=minatar_breakout_rainbow cogment run test_environment

test_environment_1 | /usr/local/lib/python3.7/site-packages/coverage/report.py:87: CoverageWarning: Couldn't parse '/environment/config-3.7.py': No source for code: '/environment/config-3.7.py'. (couldnt-parse)
test_environment_1 | coverage._warn(msg, slug="couldnt-parse")
test_environment_1 | /usr/local/lib/python3.7/site-packages/coverage/report.py:87: CoverageWarning: Couldn't parse '/environment/config.py': No source for code: '/environment/config.py'. (couldnt-parse)
test_environment_1 | coverage._warn(msg, slug="couldnt-parse")
test_environment_1 | Wrote HTML report to htmlcov/index.html
test_environment_1 | /usr/local/lib/python3.7/site-packages/coverage/report.py:87: CoverageWarning: Couldn't parse '/environment/config-3.7.py': No source for code: '/environment/config-3.7.py'. (couldnt-parse)
test_environment_1 | coverage._warn(msg, slug="couldnt-parse")
test_environment_1 | /usr/local/lib/python3.7/site-packages/coverage/report.py:87: CoverageWarning: Couldn't parse '/environment/config.py': No source for code: '/environment/config.py'. (couldnt-parse)
test_environment_1 | coverage._warn(msg, slug="couldnt-parse")

R2D2

We need R2D2 for Hanabi agent and current implementations in cogment do not have recurrent networks in RL. We should reproduce the results of R2D2: https://openreview.net/pdf?id=r1lyTjAqYX

Add model card with description and some results for ASP.

Port µZero to next

Cannot restart a trial if the Mlflow project is deleted

If you delete an Mlflow project and try to recommit a trial with the same name, you get some errors from Mlflow and the trial will not launch. You either need to rename the trial or delete the data/mlfow/mlflow.db database to fix this.