Agent reset() not called before starting new episodes in SingleThreadedWorker about rlgraph

[michaelschaarschmidt]

Hey, thanks for raising this! Will take care of shortly

[michaelschaarschmidt]

Irrespective of the issue just a short piece of info on why the preprocessor code is even there:

The reason the preprocessors are done separately in Python is because we noticed an issue around image resizing in TensorFlow (https://hackernoon.com/how-tensorflows-tf-image-resize-stole-60-days-of-my-life-aba5eb093f35), so to do any benchmarks around images, we realised we would need to have the option to do the preprocessing with CV2 out-of-graph.

[michaelschaarschmidt]

Cont.:

The heart of the issue is then that agent.reset() is meant to reset the preprocessor, but because of this separation as a consequence of this critical TF bug, it is not being called currently.

In any case, the name reset() is maybe misleading because users could reasonably expect that reset() fully resets the internal state (i.e. reinitalises all variables in particular).

We should hence maybe have a reset() method to fully reset agent state, and an episode_reset() which resets preprocessor state.

[dwang9302]

Since agent.reset() is not being called because of an external bug, I assume the worker(s) logic would remain the same until the TF bug is fixed? The reason I asked is that I put some logic into my modified agent and an end-of-episode agent.reset() call becomes critical in training.
Thanks for the reply and clarification!

[michaelschaarschmidt]

Yes since preprocessing is being handled via the python/numpy preprocessor stack implementations by default now, the tf preprocessor does not need resetting (even thought it would not hurt). So you could add a reset call if your agent needed some specific extra resetting.