cycraig / mp-dqn Goto Github PK

Hello. I have a few questions.
1, What the effect of variable "shrink" in the class "SarsaLambdaAgent"? And Can I use other basis instead, like the polynomial basis?
2, Why do you scale step size of SARSA agent? Can I use a fixed one instead?

/home/tj/anaconda3/envs/p-dqn/bin/python3 /snap/pycharm-professional/171/helpers/pydev/pydevd.py --multiproc --qt-support=auto --client 127.0.0.1 --port 40821 --file /home/tj/PycharmProjects/MP-DQN/run_soccer_qpamdp.py
pydev debugger: process 7349 is connecting

Connected to pydev debugger (build 192.7142.56)
/home/tj/anaconda3/envs/p-dqn/lib/python3.6/site-packages/hfo_py/bin/HFO
Starting server with command: /home/tj/anaconda3/envs/p-dqn/lib/python3.6/site-packages/hfo_py/bin/HFO --headless --frames-per-trial 500 --offense-agents 1 --defense-agents 0 --offense-npcs 0 --defense-npcs 0 --port 47115 --offense-on-ball 0 --seed -1 --ball-x-min 0.000000 --ball-x-max 0.200000 --log-dir log --fullstate --no-logging
usage: HFO [-h] [--headless] [--trials NUMTRIALS] [--frames NUMFRAMES]
           [--frames-per-trial MAXFRAMESPERTRIAL]
           [--untouched-time MAXUNTOUCHEDTIME]
           [--offense-agents OFFENSEAGENTS] [--defense-agents DEFENSEAGENTS]
           [--offense-npcs OFFENSENPCS] [--defense-npcs DEFENSENPCS]
           [--agent-play-goalie] [--offense-team OFFENSETEAM]
           [--defense-team DEFENSETEAM] [--no-sync] [--port PORT]
           [--no-logging] [--hfo-logging] [--log-dir LOGDIR] [--record]
           [--offense-on-ball OFFENSEONBALL] [--fullstate] [--seed SEED]
           [--message-size MESSAGESIZE] [--ball-x-min MIN_BALL_X]
           [--ball-x-max MAX_BALL_X] [--ball-y-min MIN_BALL_Y]
           [--ball-y-max MAX_BALL_Y] [--verbose] [--deterministic]
HFO: error: argument --frames-per-trial: invalid int value: '/snap/pycharm-professional/171/helpers/pydev/pydevd.py'
******************************************************************
 librcsc 4.1.0
 Copyright 2000 - 2007. Hidehisa Akiyama.
 Copyright 2007 - 2011. Hidehisa Akiyama and Hiroki Shimora
 All rights reserved.
******************************************************************
[ConnectToServer] Server Down!
 -1: waited 5 seconds. server down??

Process finished with exit code 1

If it runs run_psoccer_pdqn.py , the code will work properly, but debugging will result in the above error. How can I debug correctly?

Hi!

For my Master's thesis, I'm looking to use the MP-DQN algorithm in order to solve a problem. However, I'd like to use a dueling network to speed up convergence. With the regular P-DQN algorithm, this was quite easily accomplished, but it is not as intuitive with the MP-DQN algorithm. For the advantage function, the diagonal of the output of the multi-pass works as expected since it refers to the numbers of actions, but it does not make sense for the value function, since that is not dependent on the action parameters, but just a single value representing the value of the state.

There are a few things that could be done. One thing is to do as the P-DQN and have the value function part of the dueling network be dependent on all action parameters. Another thing is to have it be dependent on no action parameters, but this requires the dueling network to be split up more (since the input dimensions for the first fully connected layer would be different for the advantage function and the value function). Finally, one could still perform the multi-pass on the value part, and take the mean of the outputs or something similar.

I don't really know what the smartest choice here is. Intuitively, it feels as if the value function should have nothing to do with the action parameters. Given your knowledge with the MP-DQN, do you have any ideas as to what might be a smart solution?

Thank you! (Also thank you very much for making your code open-sourced!)

When does the update of the the discrete action parameters happen? There is no usage of '_action_update' function in 'learn' function of the file 'qpamdp.py'.
Thank you.

I have trained a model for "soccer pdqn", and I want to finetune a new work based on the trained model, what should I do?