cun-bjy / gym-ddpg-keras

Keras Implementation of DDPG(Deep Deterministic Policy Gradient) with PER(Prioritized Experience Replay) option on OpenAI gym framework

Junyeob Baek

gym-ddpg-keras's Issues

Simulation is too slow

Simulation Step getting slower.. (using CPU)

---step28(EP0)---
predict:  0.0423583984375 sec
update_network:  0.05187201499938965 sec
total:  0.12126398086547852 sec

---step467(EP2)---
predict:  0.16730880737304688 sec
update_network:  0.2372143268585205 sec
total:  0.4585869312286377 sec

IS weight on PER

Importance-Sampling(IS) Weights

5 out of the last 5 calls to <function Model.make_train_function.<locals>.train_function at 0x7f627418ac80> triggered tf.function retracing.

When the train.py module run, some warning happen on the tensorflow.
and the speed is critically slow..

I cannot sure if it depends on GPU using or not.

WARNING:tensorflow:5 out of the last 5 calls to <function Model.make_train_function.<locals>.train_function at 0x7f627418ac80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.
WARNING:tensorflow:6 out of the last 6 calls to <function Model.make_train_function.<locals>.train_function at 0x7f627418ac80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.
WARNING:tensorflow:7 out of the last 7 calls to <function Model.make_train_function.<locals>.train_function at 0x7f627418ac80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.
WARNING:tensorflow:8 out of the last 8 calls to <function Model.make_train_function.<locals>.train_function at 0x7f627418ac80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.
WARNING:tensorflow:9 out of the last 9 calls to <function Model.make_train_function.<locals>.train_function at 0x7f627418ac80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.
WARNING:tensorflow:10 out of the last 10 calls to <function Model.make_train_function.<locals>.train_function at 0x7f627418ac80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.
WARNING:tensorflow:11 out of the last 11 calls to <function Model.make_train_function.<locals>.train_function at 0x7f627418ac80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.
WARNING:tensorflow:11 out of the last 11 calls to <function Model.make_train_function.<locals>.train_function at 0x7f627418ac80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.

hyperparameter setting on config file

for grace code.

libpcre16.so.3: cannot open shared object file: No such file or directory

apt-get install libpcre3-dev
reference:
openai/roboschool#193

support various model

• Ant
• HalfCheetah
  etc.

look like not being trained

after 20 episodes, the reward 's so poor.

=========EPISODE # 16 ==========
100%|########################################9| 499/500 [00:53<00:00,  8.67it/s]Episode#16, steps:499, rewards:-311.104067
100%|########################################9| 499/500 [00:53<00:00,  9.28it/s]
=========EPISODE # 17 ==========
100%|########################################8| 498/500 [00:51<00:00, 10.16it/s]Episode#17, steps:499, rewards:-211.220928
100%|########################################9| 499/500 [00:51<00:00,  9.67it/s]
=========EPISODE # 18 ==========
100%|########################################8| 498/500 [00:55<00:00,  9.79it/s]Episode#18, steps:499, rewards:-243.705719
100%|########################################9| 499/500 [00:56<00:00,  8.90it/s]
=========EPISODE # 19 ==========
100%|########################################9| 499/500 [00:55<00:00,  7.42it/s]Episode#19, steps:499, rewards:-332.657685
100%|########################################9| 499/500 [00:56<00:00,  8.90it/s]
=========EPISODE # 20 ==========
100%|########################################9| 499/500 [00:55<00:00,  9.19it/s]Episode#20, steps:499, rewards:-231.899928

UserWarning: Matplotlib is currently using agg, which is a non-GUI backend, so cannot show the figure.

train.py:84: UserWarning: Matplotlib is currently using agg, which is a non-GUI backend, so cannot show the figure.
  plt.pause(0.01)
Segmentation fault (core dumped)

memorize and replay with PER

Experience Priorities can help for training.

reference link:
https://mrsyee.github.io/rl/2019/01/25/PER-sumtree/

cartpole learning(continuous) : RoboschoolInvertedPendulum-v1

reinforcement learning
in RoboschoolInvertedPendulum-v1 environment

auto-save

train by cuda

with optional usage

learning status reporting for analysis

• basic analysis tool using seaborn
• log save & load for bootstrap

cartpole learning(discrete)

20.12.29

• poor performance even in cartpole..
• get back to the original. something wrong..

easy environment making

for every computers

DDPG w/ :RoboschoolHopper-v1

load pre-trained weights