This project is for applying the idea of elastic time step with reinforcement learning under tmrl framework.
This repo is for the paper Reinforcement Learning with Elastic Time Step. About the
The tmrl framework can establish a communication link between game Trackmania to your local machine though OpenPlant.
The tmrl framework is a powerful Multi-threaded data transfer framework, allowing users to train multi-threaded. However, Trackmania is not supposed to run multi-threaded. In other words, one machine can only launch one game at the same time.
Therefore, our code will be used for single-thread training on a Windows machine.
- Windows 10 / 11
- Python >= 3.7
- A NVIDIA GPU (required only if you plan to train your own AI, better latest the Ampere framework to enable the latest driver and Cuda version.)
We recommend installing the conda version of pywin32
:
conda install pywin32
The following instructions are for installing tmrl
with support for the TrackMania 2023 video game.
You will first need to install TrackMania 2023 and also a small community-supported utility called Openplanet for TrackMania (the Gymnasium environment needs this utility to compute the reward).
Ensure you install the Visual C++ runtime
before installing the OpenPlanet. You can download it
here for 64bits versions of Windows.
Then you can install tmrl
framework easily by:
pip install tmrl
Once you have installed the tmrl
framework, you should set up your own Cuda environment.
You need to install the driver by checking the official website for the latest driver. Then, install cuda toolkit with its related cuDNN.
After that, you need to install pytorch
with the guidance of their official website.
We tested our code with these hardware and software conditions:
- Nvidia RTX 4070
- Driver version: 535.98
- Cuda version: 11.8
- cuDNN version: 8.7.0
- Pytorch version: 2.1.0+cu118
Our code should work with a higher version. Those conditions are just for reference.
Once you have the development environment prepared, you can download our code though:
git clone https://github.com/alpaficia/TMRL_MOSEAC
Then copy the config.json
file to the tmrl config folder. It should locate at C:\user\your_name\TmrlData\config
.
Load the tmrl-test track into your TrackMania game:
- Navigate to your home folder
(C:\Users\username\)
, and openTmrlData\resources
- Copy the
tmrl-test.Map.Gbx
file into...\Documents\Trackmania\Maps\My Maps
.
Next, you need to install the requirement package by:
cd Path_To_Your_Code pip install -r requirement.txt
-
Launch TrackMania 2023
-
In case the OpenPlanet menu is showing in the top part of the screen, hide it using the
f3
key -
Launch the
tmrl-test
track. This can be done by selectingcreate > map editor > edit a map >
tmrl-test,` selecting a map, and hitting the green flag. -
Set the game in windowed mode. To do this, bring the cursor to the top of the screen, and a drop-down menu will show Hit the window icon.
-
Bring the TrackMania window to the top-left corner of the screen. On Windows 10 / 11, it should automatically fit
a quarter of the screen (NB: the window will automatically snap to the top-left corner and get sized properly when you start the training). -
Hide the ghost by pressing the g key.
Once you start the game, you can launch a new terminal and cd to your code path. Then:
python MOSEAC_main.py
The training process is based on real-time data. So, performance is highly related to your PC's hardware products.
If you don't see the log outputs many "time out," which means your PC is good enough to process with the current time parameters setting.
If you see many "time out" warnings during this training, it means your CPU IPC or RAM is not good enough for this training. Our control rate is default set within [5, 30] Hz. You can change them by:
python3 MOSEAC_main.py --min_time=YOUR_VALUES --max_time=YOUR_VALUES
`
To meet your hardware productions' performance.
Our code was tested on the CPU: i5-13600K, RAM: DDR4 3200 MHz, and GPU: Nvidia RTX 4070. If you keep the default parameters, you probably see the agent is able to pass the first curve around 8 hours.
The Trackmania game also provides lidar data. If you would like to do more, please refer to the repo here.
Name | Value | Annotation |
---|---|---|
Total steps | 2e7 | |
0.95 | Discount factor | |
Net shape | (256, 256) | |
batch_size | 256 | |
a_lr | 2e-4 | Learning rate of Actor Network |
c_lr | 3e-4 | Learning rate of Critic Network |
max_steps | 2000 | Maximum steps for one episode |
0.01 | ||
-3 | Refer to SAC | |
min_time | 0.02 | Minimum control duration, in seconds |
max_time | 0.5 | Maximum control duration, in seconds |
10.0 | Maximum value for |
|
1.0 | Init value of |
|
1e-4 | Monotonically increasing H-parameter | |
Optimizer | Adam | Refer to Adam |
environment steps | 1 | |
Replaybuffer size | 1e5 | |
Number of samples before training start | 5 * max_steps | |
Number of critics | 2 |
The training graphs are here:
The graph of consumption of steps is here:
The graph of consumption of time is here:
The video is available here.
MIT
Author: Dong Wang ([email protected]), Giovanni Beltrame ([email protected])
You are also welcome to contact MISTLAB for more fun and practical robotics and AI-related projects and collaborations. :)