uber-research / ape-x Goto Github PK
View Code? Open in Web Editor NEWThis repo replicates the results Horgan et al obtained in "Distributed Prioritized Experience Replay"
License: Apache License 2.0
This repo replicates the results Horgan et al obtained in "Distributed Prioritized Experience Replay"
License: Apache License 2.0
After I run make under gym_tensorflow, I got the flowing error messages
.//atari/tf_atari.cpp:137:16: error: ‘std::string AtariEnvironment::DebugString()’ marked ‘override’, but does not override
string DebugString() override { return "AtariEnvironment"; }
.//atari/tf_atari.cpp:155:64: error: invalid new-expression of abstract class type ‘AtariEnvironment’
AtariEnvironment* env = new AtariEnvironment(batch_size);
What's the problem?
Hi! Managed to install this using python 3.6 and tensorflow 1.13.1 (make crushed with tensorflow-gpu so I deactivated the gpu).
Running "python apex.py --env video_pinball --num-timesteps 1000000000 --logdir=/tmp/agent" crashes after a few seconds with:
INFO:tensorflow:Saving checkpoints for 768 into /tmp/agent/model.ckpt.
Failed to unreference resources
Failed to unreference resources
Failed to unreference resources
Failed to unreference resources
Traceback (most recent call last):
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
return fn(*args)
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: FIFOQueue '_5_training_queue/prefetch_queue/fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
[[{{node fifo_queue_Dequeue_1}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "apex.py", line 379, in learn
lambda step_context: step_context.session.run([update_target, stage_op]))
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 737, in run_step_fn
return self._sess.run_step_fn(step_fn, self._tf_sess(), run_with_hooks=None)
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1191, in run_step_fn
return self._sess.run_step_fn(step_fn, raw_session, run_with_hooks)
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1098, in run_step_fn
return step_fn(_MonitoredSession.StepContext(raw_session, run_with_hooks))
File "apex.py", line 379, in
lambda step_context: step_context.session.run([update_target, stage_op]))
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: FIFOQueue '_5_training_queue/prefetch_queue/fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
[[node fifo_queue_Dequeue_1 (defined at apex.py:336) ]]
Caused by op 'fifo_queue_Dequeue_1', defined at:
File "apex.py", line 437, in
cli()
File "apex.py", line 433, in cli
main(env=args.env, num_timesteps=args.num_timesteps)
File "apex.py", line 48, in main
**kwargs
File "apex.py", line 336, in learn
train_dequeue = training_fifo.dequeue()
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/ops/data_flow_ops.py", line 445, in dequeue
self._queue_ref, self._dtypes, name=name)
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 3908, in queue_dequeue_v2
timeout_ms=timeout_ms, name=name)
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
op_def=op_def)
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in init
self._traceback = tf_stack.extract_stack()
OutOfRangeError (see above for traceback): FIFOQueue '_5_training_queue/prefetch_queue/fifo_queue' is closed and has insufficient elements (requested 1, current size 0)
[[node fifo_queue_Dequeue_1 (defined at apex.py:336) ]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "apex.py", line 437, in
cli()
File "apex.py", line 433, in cli
main(env=args.env, num_timesteps=args.num_timesteps)
File "apex.py", line 48, in main
**kwargs
File "apex.py", line 386, in learn
sess.run_step_fn(lambda step_context: step_context.session.run([update_target]))
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 788, in exit
self._close_internal(exception_type)
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 826, in _close_internal
self._sess.close()
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1082, in close
self._sess.close()
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1244, in close
ignore_live_threads=True)
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/six.py", line 719, in reraise
raise value
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/training/queue_runner_impl.py", line 257, in _run
enqueue_callable()
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1257, in _single_operation_run
self._call_tf_sessionrun(None, {}, [], target_list, None)
File "/home/elena/anaconda3/envs/tf_old/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.UnimplementedError: Generic conv implementation only supports NHWC tensor format for now.
[[{{node deepq/read_q_func/convnet/Conv/Conv2D}}]]
when i run apex.py . apex_tensorflow.so: cannot open shared object file: No such file or directory
There are three operation related replaybuffer
Actor
: enqueue
Leaner
: sample_proportional_from_buffer()
sample transitions from replay buffer
Learner
: replay_buffer.assign(idxs, new_priorities)
update priorities
I wonder that: How does replaybuffer
ensure multithread safety?
idxs may have been replaced since the actor
performs enqueue operations
During make
inside ape-x/gym_tensorflow
, we get the following error.
.//atari/tf_atari.cpp:18:29: fatal error: ale_interface.hpp: No such file or directory
#include <ale_interface.hpp>
^
compilation terminated.
make: *** [gym_tensorflow.so] Error 1
The ALE header file seems to be like this or this, but it requires additional steps for installation. README should mention this.
Makefile reads:
ALE := $(shell pwd)/atari-py/atari_py/ale_interface
FLAGS += -I$(ALE)/src -I$(ALE)/src/controllers -I$(ALE)/src/os_dependent -I$(ALE)/src/environment -I$(ALE)/src/external -L$(ALE)/build
Therefore a simple solution is to have atari-py
directory cloned (which is gitignored) inside gym_tensorflow
, and build (make
) it first. Doing this additional procedure made build successful.
When I run tf_env.py, I got the following error.
/ape-x/gym_tensorflow/atari-py/atari_py/ale_interface/src/emucore/TIA.cxx:1147: void TIA::computePlayerPositionResetWhenTable(): Assertion `(s1 % 4 == 0) && (s2 % 8 == 0)' failed.
This is the same problem which was discussed in Farama-Foundation/Arcade-Learning-Environment#86.
So I made tf_atari.cpp not load_rom concurrently by the following code, then I resolved the above issue.
const int num_threads = std::min(thread_pool->num_threads, batch_size);
std::mutex atari_mutex;
auto f = [&](int thread_id) {
for(int b =thread_id; b < batch_size;b+=num_threads)
{
atari_mutex.lock();
env->load_rom(m_game, b);
atari_mutex.unlock();
}
};
Is this problem a common problem? Othewise is there any problem in my environment?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.