simple_q_agent example about tensorforce

[michaelschaarschmidt]

This example should also be included in new unit tests, normally was not tested because it's only for API demo purposes

[krfricke]

Thanks for the info, I fixed the example code and included the new code in the docs with commit 7d4ead3. The simple q agent should now start.

I got a different error this time when running the script:

ValueError: Variable simpleqnetwork/linear1/weights already exists, disallowed. Did you mean to set reuse=True in VarScope? Originally defined at:

  File "/Users/aidanrocke/anaconda/envs/py3k/lib/python3.6/site-packages/tensorflow/contrib/framework/python/ops/variables.py", line 217, in variable
    use_resource=use_resource)
  File "/Users/aidanrocke/anaconda/envs/py3k/lib/python3.6/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 181, in func_with_args
    return func(*args, **current_args)
  File "/Users/aidanrocke/anaconda/envs/py3k/lib/python3.6/site-packages/tensorflow/contrib/framework/python/ops/variables.py", line 262, in model_variable
    use_resource=use_resource)

I'm using a conda virtual environment(py3k) and Python 3.6

There appears to be a conflict with tensorflow.

[krfricke]

Hmm, I got yet another error with python 3.5. I'll check for python 3.x compatibility and possibly look into anaconda compatibility some time this week - probably won't make it today or tomorrow, however.

[krfricke]

I fixed the other issue with python 3.x in 9c6bee4, but that was unrelated to your error.

[krfricke]

Could not replicate the error using anaconda, everything works for me:

python -V
Python 3.6.0 :: Anaconda 4.3.1 (64-bit)

python tensorforce/examples/simple_q_agent.py 
[2017-04-24 14:25:30,683] Making new env: CartPole-v0
2017-04-24 14:25:30.690554: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-04-24 14:25:30.690602: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-04-24 14:25:30.690616: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-04-24 14:25:30.690629: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
[2017-04-24 14:25:30,816] Starting SimpleQAgent for Environment 'CartPole-v0'
[2017-04-24 14:25:30,857] Finished episode 1 after 24 timesteps
[2017-04-24 14:25:30,857] Episode reward: 24.0
[2017-04-24 14:25:30,858] Average of last 10 rewards: 24.0
[2017-04-24 14:25:31,447] Finished episode 11 after 28 timesteps
[2017-04-24 14:25:31,447] Episode reward: 28.0
[2017-04-24 14:25:31,447] Average of last 10 rewards: 35.5

I'm not familiar with anaconda, so I can only guess. Which TensorFlow version do you use? Do you use a special configuration? Could you try replicating the bug again with a fresh environment?

I setup a fresh environment this morning using these instructions and I also keras+tensorflow with tensorflow version 1.1.0

Moreover, to install tensorflow I simply used:

pip install tensorflow # Python 2.7; CPU support (no GPU support)

I followed the instructions on the Mac OSX page.

Here's the error I get this time:

runfile('/Users/aidanrocke/Desktop/tensorforce/tensorforce/examples/simple_q_agent.py', wdir='/Users/aidanrocke/Desktop/tensorforce/tensorforce/examples')
[2017-04-24 23:22:24,847] Making new env: CartPole-v0
[2017-04-24 23:22:25,238] Starting SimpleQAgent for Environment 'CartPole-v0'
Traceback (most recent call last):

  File "<ipython-input-1-4e18e599c233>", line 1, in <module>
    runfile('/Users/aidanrocke/Desktop/tensorforce/tensorforce/examples/simple_q_agent.py', wdir='/Users/aidanrocke/Desktop/tensorforce/tensorforce/examples')

  File "/Users/aidanrocke/anaconda/envs/py3k/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 866, in runfile
    execfile(filename, namespace)

  File "/Users/aidanrocke/anaconda/envs/py3k/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 102, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "/Users/aidanrocke/Desktop/tensorforce/tensorforce/examples/simple_q_agent.py", line 217, in <module>
    main()

  File "/Users/aidanrocke/Desktop/tensorforce/tensorforce/examples/simple_q_agent.py", line 213, in main
    runner.run(max_episodes, max_timesteps, episode_finished=episode_finished)

  File "/Users/aidanrocke/Desktop/tensorforce/tensorforce/execution/runner.py", line 62, in run
    self.agent.add_observation(processed_state, action, result['reward'], result['terminal_state'])

  File "/Users/aidanrocke/Desktop/tensorforce/tensorforce/agents/memory_agent.py", line 112, in add_observation
    if self.step_count >= self.min_replay_size and self.step_count % self.update_steps == 0:

TypeError: '>=' not supported between instances of 'int' and 'NoneType'

It's not clear to me why self.step_count is of type NoneType at some point but this is the error I'm getting.

[michaelschaarschmidt]

solved in gitter