lopespm / agent-trainer

Using tensorflow, this agent can autonomously train itself to play Out Run and potentially be modified to play other games or perform tasks other than gaming

Agent Trainer

Through Deep Reinforcement Learning, this agent can autonomously train itself to play Out Run and potentially be modified to play other games or perform tasks other than gaming.

More details about the training process in this blogpost.

Built with Python and Tensorflow.

Setup

Build cannonball's (Out Run game emulator) SO file

1. Clone the custom Cannonball fork, which contains the changes needed to access the emulator externally.

2. Build Cannonball:

   # On Cannonball's root folder do:
   $ mkdir build
   $ cd build
   # Building on OSX / MacOS
   $ cmake -G "Unix Makefiles" -DTARGET:STRING=sdl2_macosx ../cmake
   # If building on Linux, execute this commented line instead:
   # $ cmake -G 'Unix Makefiles' -DTARGET:STRING=sdl2 ../cmake
   $ make

3. Copy the built shared object file (libcannonball.so) to agent-trainer's lib folder:

   $ cp <cannonball-folder>/build/libcannonball.so <agent-trainer-folder>/lib/libcannonball.so

Out Run Game Roms

Copy your game roms to <agent-trainer-folder>/roms/. More details here.

Set the training results folder

Set the train_results_root_folder parameter in config.py. This will be the default folder used to write and read training results (can be overriden, see below)

Install python dependencies

On the root folder, run:

$ make

This installs the depencies via pip. Feel free to use virtual env wrapper (for example) to contain these.

Dependencies on a GPU enabled Linux machine

If you are running a Linux machine and want to make use of its GPU for training, use the following flag when installing dependencies:

$ USE_GPU=true make

Usage

You can use the trainer via make tasks:

# Run all the tests
$ make test

# Start new training session
$ make train-new

# Resume training session
$ SESSION_ID="201609272034" make train-resume

# Play a previously trained session
$ SESSION_ID="201609272034" make play

# Create a t-Distributed Stochastic Neighbor Embedding (t-SNE) visualization, placed on `<train-results-root-folder>/<session-id>/visualizations/t-SNE.png`
# Example output: https://github.com/lopespm/agent-trainer-results/blob/master/201609171218_175eps/visualizations/t-SNE_no_time_mode.png
$ SESSION_ID="201609272034" make visualize-tsne

# Export the metrics of the given session, as PNG images files. These will be placed on `<train-results-root-folder>/<session-id>/metrics-session/`
# Example output: https://github.com/lopespm/agent-trainer-results/tree/master/201609171218_175eps/metrics-session
$ SESSION_ID="201609272034" make metrics-export

# Show the session's metrics on-screen
$ SESSION_ID="201609272034" make metrics-show

For finer control, you can run the library module as a script. For example:

python -m agent play --ec2spot --resultspath /example/alternative-results-folder -s 201609261533

Actions: train-new, train-resume, play, visualize-tsne, metrics-show or metrics-export

Options:

• -s: define the session ID
• --ec2spot: when used, it will check on which episode if the spot instance is scheduled for termination, acting accordingly by saving the current session and halting training
• --resultspath: overrides the default train_results_root_folder parameter

Deploy to Remote Machine / AWS EC2 Instance

You can deploy the agent to a generic remote Linux machine or to an AWS EC2 instance with the help of agent-trainer-deployer

agent-trainer's Issues

How to play

Hi!
After training, I try to let the code to play the game by typing
SESSION_ID="201609272034" make play

However, although the code outputs turn left or turn right. The interface of the game seems stuck, and the car in it does not move at all.

Could you please give me some comments to solve this issue?

fatal error: 'iostream' file not found

Zachs-Mac:build zach$ make
Scanning dependencies of target cannonball
[ 1%] Building CXX object CMakeFiles/cannonball.dir/Users/zach/Downloads/cannonball-master/src/main/romloader.cpp.o
clang: warning: no such sysroot directory: '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk'
/Users/zach/Downloads/cannonball-master/src/main/romloader.cpp:11:10: fatal error: 'iostream' file not
found
#include
^
1 error generated.
make[2]: *** [CMakeFiles/cannonball.dir/Users/zach/Downloads/cannonball-master/src/main/romloader.cpp.o] Error 1
make[1]: *** [CMakeFiles/cannonball.dir/all] Error 2
make: *** [all] Error 2

Train model in debian server

Hi

I cloned the code base in my debian server and installed deps according the README guide. make test run very well, and I tried make train-new, it produced the output:

$ make train-new
python -m agent train-new
20161007|17:50:01|INFO: Session id 201610071749: training
20161007|17:50:01|INFO: Session 201610071749 at episode 0: q network ref<agent.trainer.q_network.QNetwork object at 0x7f90900b4550>; replay memories len 0; final iteration: 0
SDL Initialization Failed: No available video device
Makefile:26: recipe for target 'train-new' failed
make: *** [train-new] Segmentation fault

How can I fix the error?
Thanks!