mome36 / branchingdqn

BranchingDQN

Branching DQN

This repo holds an implementation of a PyTorch version of Branching Double Deep Q-Learning. It is based on my other repo for Double DQN: https://github.com/mome36/DoubleDQN and on the paper https://arxiv.org/pdf/1711.08946.pdf

Branching DQN allows a Q-Learning agent to select multiple actions simultaneously. It can be used for continuous control, as I've demonstrated in BipedalWalker-v2 or for controlling a video game character (yielding actions sur as Jump and Hit instead of Jump or Hit).

How to use:

To train an agent, run:

python dueling_dqn.py --env BipedalWalker-v2

To see the agent perform:

python enjoy.py --env BipedalWalker-v2

Performances

branchingdqn's Issues

The calculation of loss function is inconsistent with the paper

loss = F.mse_loss(expected_q_vals, current_q_values)
In this line of code, current_q_values are the mean of all branches Q values, the same with equation 6 in the paper, y.
However, the calculation of loss function in equation 7 uses y_d, whitch means Q values of each branch.

Moreover, the size of expected_q_vals is [128, 1], but the size of current_q_values is [128, 4]. This cause a warning
UserWarning: Using a target size (torch.Size([128, 4])) that is different to the input size (torch.Size([128, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.

I didn't understand the concept of Branching DQN yet.

Is that true that can select multiple actions at a time. But there is an argmax ?

Bugs in model.py?

Thanks for your contribution!
I guess ac_dim and n should swap positions here?

BranchingDQN/model.py

Line 48 in 3eb4df0

self.adv_heads = nn.ModuleList([nn.Linear(128, n) for i in range(ac_dim)])