cycraig / mp-dqn Goto Github PK
View Code? Open in Web Editor NEWSource code for the dissertation: "Multi-Pass Deep Q-Networks for Reinforcement Learning with Parameterised Action Spaces"
License: MIT License
Source code for the dissertation: "Multi-Pass Deep Q-Networks for Reinforcement Learning with Parameterised Action Spaces"
License: MIT License
Hello. I have a few questions.
1, What the effect of variable "shrink" in the class "SarsaLambdaAgent"? And Can I use other basis instead, like the polynomial basis?
2, Why do you scale step size of SARSA agent? Can I use a fixed one instead?
/home/tj/anaconda3/envs/p-dqn/bin/python3 /snap/pycharm-professional/171/helpers/pydev/pydevd.py --multiproc --qt-support=auto --client 127.0.0.1 --port 40821 --file /home/tj/PycharmProjects/MP-DQN/run_soccer_qpamdp.py
pydev debugger: process 7349 is connecting
Connected to pydev debugger (build 192.7142.56)
/home/tj/anaconda3/envs/p-dqn/lib/python3.6/site-packages/hfo_py/bin/HFO
Starting server with command: /home/tj/anaconda3/envs/p-dqn/lib/python3.6/site-packages/hfo_py/bin/HFO --headless --frames-per-trial 500 --offense-agents 1 --defense-agents 0 --offense-npcs 0 --defense-npcs 0 --port 47115 --offense-on-ball 0 --seed -1 --ball-x-min 0.000000 --ball-x-max 0.200000 --log-dir log --fullstate --no-logging
usage: HFO [-h] [--headless] [--trials NUMTRIALS] [--frames NUMFRAMES]
[--frames-per-trial MAXFRAMESPERTRIAL]
[--untouched-time MAXUNTOUCHEDTIME]
[--offense-agents OFFENSEAGENTS] [--defense-agents DEFENSEAGENTS]
[--offense-npcs OFFENSENPCS] [--defense-npcs DEFENSENPCS]
[--agent-play-goalie] [--offense-team OFFENSETEAM]
[--defense-team DEFENSETEAM] [--no-sync] [--port PORT]
[--no-logging] [--hfo-logging] [--log-dir LOGDIR] [--record]
[--offense-on-ball OFFENSEONBALL] [--fullstate] [--seed SEED]
[--message-size MESSAGESIZE] [--ball-x-min MIN_BALL_X]
[--ball-x-max MAX_BALL_X] [--ball-y-min MIN_BALL_Y]
[--ball-y-max MAX_BALL_Y] [--verbose] [--deterministic]
HFO: error: argument --frames-per-trial: invalid int value: '/snap/pycharm-professional/171/helpers/pydev/pydevd.py'
******************************************************************
librcsc 4.1.0
Copyright 2000 - 2007. Hidehisa Akiyama.
Copyright 2007 - 2011. Hidehisa Akiyama and Hiroki Shimora
All rights reserved.
******************************************************************
[ConnectToServer] Server Down!
-1: waited 5 seconds. server down??
Process finished with exit code 1
If it runs run_psoccer_pdqn.py , the code will work properly, but debugging will result in the above error. How can I debug correctly?
Hi!
For my Master's thesis, I'm looking to use the MP-DQN algorithm in order to solve a problem. However, I'd like to use a dueling network to speed up convergence. With the regular P-DQN algorithm, this was quite easily accomplished, but it is not as intuitive with the MP-DQN algorithm. For the advantage function, the diagonal of the output of the multi-pass works as expected since it refers to the numbers of actions, but it does not make sense for the value function, since that is not dependent on the action parameters, but just a single value representing the value of the state.
There are a few things that could be done. One thing is to do as the P-DQN and have the value function part of the dueling network be dependent on all action parameters. Another thing is to have it be dependent on no action parameters, but this requires the dueling network to be split up more (since the input dimensions for the first fully connected layer would be different for the advantage function and the value function). Finally, one could still perform the multi-pass on the value part, and take the mean of the outputs or something similar.
I don't really know what the smartest choice here is. Intuitively, it feels as if the value function should have nothing to do with the action parameters. Given your knowledge with the MP-DQN, do you have any ideas as to what might be a smart solution?
Thank you! (Also thank you very much for making your code open-sourced!)
When does the update of the the discrete action parameters happen? There is no usage of '_action_update' function in 'learn' function of the file 'qpamdp.py'.
Thank you.
I have trained a model for "soccer pdqn", and I want to finetune a new work based on the trained model, what should I do?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.