microsoft / strategically_efficient_rl

More efficient exploration for reinforcement learning in two-player, zero-sum game

Strategically Efficient Exploration in Competitive Multi-agent Reinforcement Learning

This repository contains all code and hyperparameter configurations needed to replicate the results in Strategically Efficient Exploration in Competitive Multi-agent Reinforcement Learning (UAI 2021)

In addition to finite Markov games, this project also supports experiments with curiosity in deep multi-agent reinforcement learning.

Getting Started

This code has only been tested with Python 3.7.11 on Ubuntu 18.04 and 20.04 in the Windows Subsystem for Linux (WSL).

Dependencies an be installed via PIP:

pip install -r requirements.txt

This will install all the dependencies needed to reproduce published results. Some deep RL experiment configurations us environments implemented in the OpenSpiel or PettingZoo projects, which must be installed separately. Please refer to these projects for complete installation instructions.

Reproducing UAI Results

Results in Figures 3 and 4 can be generated using the script "finite_games/learn_extensive_form.py" to run the appropriate training configurations:

cd finite_games
python learn_extensive_form.py \
    -f configs/decoy_deep_sea/strategic_ulcb.yaml \
    -f configs/decoy_deep_sea/optimistic_ulcb.yaml \
    -f configs/decoy_deep_sea/strategic_nash_q.yaml \
    -f configs/decoy_deep_sea/optimistic_nash_q.yaml

Experiment configurations can be run separately if preferred. Results for Figure 5 can be generated using:

python learn_extensive_form.py \
    -f configs/alpha_beta/strategic_ulcb.yaml \
    -f configs/alpha_beta/optimistic_ulcb.yaml \
    -f configs/alpha_beta/strategic_nash_q.yaml \
    -f configs/alpha_beta/optimistic_nash_q.yaml

Figures can be generated using the "finite_games/plot_runs.py" script. Note that this script requires

Example:

python plot_runs.py \
    "Strategic ULCB" results/debug/decoy_deep_sea_strategic_ulcb/decoy_deep_sea_strategic_ulcb_decoy_games=50,decoy_size=20 \
    "Optimistic ULCB" results/debug/decoy_deep_sea_optimistic_ulcb/decoy_deep_sea_optimistic_ulcb_decoy_games=50,decoy_size=20,exploit=True

Deep RL Experiments

Deep RL experiments use RLLib 0.8.3 and Tensorflow 2.4.2, both installed by "requirements.txt". Experiments with deep multi-agent RL can be run with the "train_multiagent.py" script.

Example:

python3 train_multiagent.py -f experiment_configs/roshambo/ppo_hybrid_bandit.yaml --nash-conv

This will train PPO in self-play in a simple two-player matrix game. This project currently supports two intrinsic reward mechansims with multi-agent PPO, Random Network Distillation and the Intrinsic Curiosity Module.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.