uidilr / deepirl_chainer

Implementation of GAIL and AIRL using chinerrl

Deep Inverse Reinforcement Learning

Contents

Chainer implementation of Adversarial Inverse Reinforcement Learning (AIRL) and Generative Adversarial Imitation Learning (GAIL). The code heavily depend on the reinforcement learning package Chainerrl.

Commands

Train and sample expert trajectory

python train_gym.py ppo --gpu $gpu_id --env CartPole-v0 --arch FFSoftmax --steps 50000 

Run GAIL

python train_gym.py gail --gpu $gpu_id --env CartPole-v0 --arch FFSoftmax --steps 100000 \
                    --load_demo ${PathOfDemonstrationNpzFile} --update-interval 128 --entropy-coef 0.01

Run AIRL

python train_gym.py airl --gpu $gpu_id --env CartPole-v0 --arch FFSoftmax --steps 100000 \
                    --load_demo ${PathOfDemonstrationNpzFile} --update-interval 128 --entropy-coef 0.01

LICENSE

MIT

deepirl_chainer's Issues

Incorrect AIRL reward

The paper suggests that the reward is given by: f(s, a, s') - \pi(a | s) (which is the same as logD - log(1-D)) but the reward in the repo is g(s, a).
Why is this discrepancy?

where is the demonstration data?

If I have to make demonstration, how do I compose the dimension of demonstration?
For example,
state = [batch, sequence, *observation_shape]
action = [batch, sequence, *action_shape]

I trained "halfcheetah-v4" for above shape of demonstration, but gail and airl does not be learnt at all...

I does not know what I mistake...

and For binary cross entropy loss, why softmax function is more stable??

discrete index action is not converted to one-hot vector in AIRL

License

I think it is good work.
Which is a license of these codes?