v1ta111 / dice_rl

DICE: The DIstribution Correction Estimation Library

This library unifies the distribution correction estimation algorithms for off-policy evaluation, including:

• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• GenDICE: Generalized Offline Estimation of Stationary Values
• Reinforcement Learning via Fenchel-Rockafellar Duality Please cite these work accordingly upon using this library.

Summary

Existing DICE algorithms are the results of particular regularization choices in the Lagrangian of the Q-LP and d-LP policy values. Choices of regularization (colored) in the Lagrangian.

These choices navigate the trade-offs between optimization stability and estimation bias. Estimation bias given the choices of regularization.

Install

Navigate to the root of project, and perform:

pip3 install -e .

To run taxi, download the pretrained policies and place them under policies/taxi:

git clone https://github.com/zt95/infinite-horizon-off-policy-estimation.git
cp -r infinite-horizon-off-policy-estimation/taxi/taxi-policy policies/taxi

Run DICE Algorithms

First, create datasets using the policy trained above:

for alpha in {0.0,1.0}; do python3 scripts/create_dataset.py --save_dir=./tests/testdata --load_dir=./tests/testdata/CartPole-v0 --env_name=cartpole --num_trajectory=400 --max_trajectory_length=250 --alpha=$alpha --tabular_obs=0; done

Run DICE estimator:

python3 scripts/run_neural_dice.py --save_dir=./tests/testdata --load_dir=./tests/testdata --env_name=cartpole --num_trajectory=400 --max_trajectory_length=250 --alpha=0.0 --tabular_obs=0

To recover DualDICE, append the following to the above python command:

--primal_regularizer=0. --dual_regularizer=1. --zero_reward=1 --norm_regularizer=0. --zeta_pos=0

To recover GenDICE, append the following to the above python command:

--primal_regularizer=1. --dual_regularizer=0. --zero_reward=1 --norm_regularizer=1. --zeta_pos=1

The configuration below generally works the best:

--primal_regularizer=0. --dual_regularizer=1. --zero_reward=0 --norm_regularizer=1. --zeta_pos=1