idreesshaikh / autonomous-driving-in-carla-using-deep-reinforcement-learning

Deep Reinforcement Learning (PPO) in Autonomous Driving (Carla) [from scratch]

Implementing a Deep Reinforcement Learning Model for Autonomous Driving

Artificial Intelligence (AI) is growing extraordinarily in almost every area of technology, and research into self-driving cars is one of them. In this work, we will take the liberty to utilize state-of-the-art methods to train our agent to drive autonomously using the Deep Reinforcement Learning (DRL) approach. We will use an open-source simulator, CARLA, to conduct our experiment, providing a hyper-realistic urban simulation environment to train our models. We cannot use our raw algorithms in the real world because they come with many risks and moral questions, so we use these simulators to help us test them.

Moreover, DRL has shown promising results in learning complex decision-making tasks, from strategic games to challenging puzzles. Here, we will look at how an on-policy DRL algorithm called Proximal Policy Optimization (PPO) will be used in a simulated driving environment to learn to navigate on a predetermined route. The primary goal of this work is to investigate how a DRL model can train an agent on a continuous state and action space. Our main contribution is a PPO-based agent that can learn to drive reliably in our CARLA-based environment. In addition, we also implemented a Variational Autoencoder (VAE) that compresses high-dimensional observations into a potentially easier-to-learn low-dimensional latent space that can help our agent learn faster.

About the Project

This work aims to develop an end-to-end solution for autonomous driving that can send commands to the vehicle to help it drive in the right direction and avoid crashes as much as possible, and is divided in the following components:

1. CARLA Environment setup.
2. Variational Autoencoder.
3. Proximal Policy Optimization.

We have used CALRA (version 0.9.8) as our environment (Urban Simulator). We have also summarized some results and analyses to simplify this problem further.

Find the documented work here to better understand this whole project.

Prerequisites

We're using CARLA (0.9.8) + Additional Maps. We're mainly focused on two towns which are Town 2 and Town 7, therefore we'd advice you download Additional Maps file alongside the CARLA server. You can copy/paste the maps from Additional Maps directory to Main CARLA directory to ensure everything is seemless.

Moving forth we'd advice you to setup your project on Windows or Linux as these are the two OSs supported by CARLA at the moment.

Project Setup (Installations)

In order to setup this project you're advised to clone this repository and make sure you have Python v3.7.+ (64bit) version installed. After clonding this repository we can create a python virtual environment for this project 💥 let's call it venv python -m venv venv. You can call it something else if you want :) Now we can activate our virtual env source venv/Script/activate, and don't forget to do so before installing any of the dependencies. Moving forward we can install the dependencies with pip with the following command pip install -r requirements.txt. We're not only using pip as our depency manager but poetry as well, therefore execute the following command cd poetry/ && poetry update in the repo. This will install all the other dependencies now with poetry. Once everything is setup up we're nearly there!

Download the CARLA server (0.9.8) + Additional Maps, and make sure you've read the Prerequisites of this repo. Once the server is up and running, we can start our client with python continuous_driver.py --exp-name=ppo --train=False command. Don't forget to start the Carla server beforehand. Yey!!!

Built With

• Python - Programming language
• PyTorch - Open source machine learning framework
• CARLA - An urban driving simulator
• Poetry - Packaging and dependency manager
• Tensorboard - Visualization toolkit

Methodology

Architectural layout encapsulating the three most essential components:

1. CARLA Simulation.
2. VAE.
3. PPO Agent.

Architectural Methodology

How to Run

Running a Trained Agent

With the project, we provide you two pretrained PPO agents, one for each town (Town 02 & Town 07). The preTrained serialized files for this model are placed in preTrained_models/PPO/<town> folder.

python continuous_driver.py --exp-name ppo --train False

By deafult we are on Town 07 but we can changed it to Town 02 with the following argument addition:

python continuous_driver.py --exp-name ppo --train False --town Town02

Training a New Agent

In order to train a new agent use the following command:

python continuous_driver.py --exp-name ppo

This will start training an agent with the default parameters, and checkpoints will be written to checkpoints/PPO/<town>/ and the other metrics will be logged into logs/PPO/<town>/. Same as above, by default we're training on Town07 but we can change it to Town02 with this argument addition --town Town02.

How our Training looks like.

Town 7

Town 2

Variational AutoEncoder

The Variational Autoencoder (VAE) training process starts by driving around automatically and manually, collecting 12,000 160x80 semantically segmented images we will be using for training. Then, we will use the SS image as the input to the variational autoencoder (h ∗ 𝑤 ∗ 𝑐 = 38400 input units). VAE’s weights are frozen while our DRL network trains.

Variational Autoencoder

Once we have trained our VAE, we can use the following command to check out the reconstructed images:

cd autoencoder && python reconstructor.py

Original Image to Reconstructed Image

Project Architecture Pipeline (Encoder to PPO)

The folloing diagram depicts the VAE+PPO training pipeline. Note: all the variable names are missing the subscript 𝑡.

VAE + PPO training pipeline

File Overview

File	Description
continuous_driver.py	Script for training/testing our continuous agent e.g. PPO agent
discrete_driver.py	Script for training our discrete agent e.g. Duelling DQN agent (work in progress)
encoder_init.py	script that uses the trained Encoder to turn the incoming images (states) into latent space
parameters.py	Contains the hyper-paramters of the Project
simulation/connection.py	Carla Environment class that makes the connection with the CARLA server
simulation/environment.py	CARLA Environment class that contains most of the Environment setup functionality (gym inspired class structure)
simulation/sensors.py	Carla Environment file that contains all the agent's sensor classes (setup)
simulation/settings.py	Carla Environement file that contains environment setup parameters
runs/	Folder containing Tensorboard plots/graphs
preTrained_models/ppo	Folder containing pre-trained models' serialized files
networks/on_policy/agent.py	Contains code of our PPO agent
networks/on_policy/ppo.py	Contains code of our PPO network
logs/	Folder containing the logged metrics of our agent while training
info/	Folder containing figures, gifs, diagrams, & documentation of the project
checkpints/	Folder containing serialized parameters of our agent saved while training
carla/	Folder containing CARL egg file, that is used in order to make connection with the server
autoencoder/	Folder containing the code for our Variational Autoencoder (VAE)

To view the training progress/plots in the Tensorboard:

tensorboard --logdir runs/

Authors

Idrees Razak - GitHub, LinkedIn

License

This project is licensed under the MIT License - see the LICENSE.md file for details

Acknowledgments

I thank, Dr. Toka László, for his exceptional leadership and unwavering support during this work.

autonomous-driving-in-carla-using-deep-reinforcement-learning's Issues

Encoder could not be initialized

pygame 2.1.2 (SDL 2.0.18, Python 3.7.15)
Hello from the pygame community. https://www.pygame.org/contribute.html
Encoder could not be initialized.

Exit

Messed up video in pygame

Thank you so much for your work, I have successfully run the code using the instruction python continuous_driver.py --exp-name=ppo --train=False, but there is an issue that is confusing me, as you can see in the image below, I'm not sure why the video inside the pygame window looks like this.
I would be very grateful if you can give me some tips!

ValueError: Expected parameter loc (Tensor of shape (10516, 2)) of distribution MultivariateNormal(loc: torch.Size([10516, 2]), covariance_matrix: torch.Size([10516, 2, 2])) to satisfy the constraint IndependentConstraint(Real(), 1), but found invalid values: tensor([[nan, nan], [nan, nan], [nan, nan], ..., [nan, nan], [nan, nan], [nan, nan]], grad_fn=<ExpandBackward0>)

Hello, I met this bug when training a ppo agent in Town02. I can't fix it.
Can u help me? Thanks a lot!

Not able to run discrete or continous file

Hi,
When I run the continous file I got this error:

##################################################################################################
python continuous_driver.py --exp-name ppo --town Town07
##################################################################################################
Couldn't import Carla egg properly
pygame 2.1.2 (SDL 2.0.16, Python 3.8.10)
Hello from the pygame community. https://www.pygame.org/contribute.html
ERROR:root:Connection has been refused by the server.

Exit
Traceback (most recent call last):
File "continuous_driver.py", line 330, in
runner()
File "continuous_driver.py", line 146, in runner
env = CarlaEnvironment(client, world,town)
UnboundLocalError: local variable 'client' referenced before assignment

##################################################################################################

As I am getting error related to egg file. I also add these lines in the continous flie:
try:
import pygame
except ImportError:
raise RuntimeError('cannot import pygame, make sure pygame package is installed')

try:
import numpy as np
except ImportError:
raise RuntimeError('cannot import numpy, make sure numpy package is installed')
except ImportError:
raise RuntimeError('cannot import pygame, make sure pygame package is installed')

Calra module

try:
sys.path.append(glob.glob('../carla/dist/carla-*%d.%d-%s.egg' % (
sys.version_info.major,
sys.version_info.minor,
'win-amd64' if os.name == 'nt' else 'linux-x86_64'))[0])
except IndexError:
pass

##################################################################################################

But still I am getting the error related to the egg flie. My other files are working perfectly in CARLA.

When I run the file for discrete file. I got this error.

python3 discrete_driver.py
Couldn't import Carla egg properly
pygame 2.1.2 (SDL 2.0.16, Python 3.8.10)
Hello from the pygame community. https://www.pygame.org/contribute.html
Traceback (most recent call last):
File "discrete_driver.py", line 16, in
from networks.off_policy.ddqn.agent import DQNAgent
File "/home/Autonomous-Driving-in-Carla-using-Deep-Reinforcement-Learning-main/networks/off_policy/ddqn/agent.py", line 4, in
from networks.off_policy.ddqn.dueling_dqn import DuelingDQnetwork
File "/home/Autonomous-Driving-in-Carla-using-Deep-Reinforcement-Learning-main/networks/off_policy/ddqn/dueling_dqn.py", line 5, in
from parameters import DQN_LEARNING_RATE, DQN_CHECKPOINT_DIR, TOWN7
ImportError: cannot import name 'TOWN7' from 'parameters' (/home/ryzen/Autonomous-Driving-in-Carla-using-Deep-Reinforcement-Learning-main/parameters.py)

So I add this line in parameters.py file
TOWN7 = "Town07"

And then I got this error:

python3 discrete_driver.py
Couldn't import Carla egg properly
pygame 2.1.2 (SDL 2.0.16, Python 3.8.10)
Hello from the pygame community. https://www.pygame.org/contribute.html

Exit
Traceback (most recent call last):
File "discrete_driver.py", line 212, in
runner()
File "discrete_driver.py", line 57, in runner
writer = SummaryWriter(f"runs/{run_name}/{town}")
UnboundLocalError: local variable 'run_name' referenced before assignment

How to resolve both these errors?

Why can't agents train using gpu?

I see the torch. device ('cpu ') in your code. I tried to modify it to torch. device ('cuda: 0'), but it doesn't seem to work.
my torch.cuda.is_ available() returns true.

Training exits before the while loop criteria is met

Hello Idree,

I am running a new training and the code is exiting before meeting the criteria, why is this happening and how can I have a complete training without being stopped too frequently?

Thank you.

pygame show

The image displayed on my pygame is incorrect. Is there a problem with yours

pygame.error: Unable to make GL context current

Thank you so much for your work.But when I run the code using the instruction python continuous_driver.py --exp-name=ppo --train=False, there is an error as follows.

Traceback (most recent call last):
File "/home/**/Autonomous-Driving-in-Carla-using-Deep-Reinforcement-Learning-main/simulation/sensors.py", line 63, in self.sensor.listen(lambda image: CameraSensorEnv._get_third_person_camera(weak_self, image)) File "/home//Autonomous-Driving-in-Carla-using-Deep-Reinforcement-Learning-main/simulation/sensors.py", line 87, in _get_third_person_camera pygame.display.flip()
pygame.error: Unable to make GL context current

I would be very grateful if you can give me some tips!

Hardwares

Hi, i just want to know what hardwares did you run the project on? Specially cpu modal, gpu modal and memory, ram memory, motherboard modal?

checkpoint is not optimal

I ran continuous_drive.py in town02 using checkpoint_ppo_12.pickle , but the car's behavior with the provided checkpoint network parameters is not optimal, if you have any updated or improved checkpoint files available that might enhance the self-driving capabilities of the code.

bug

hi, when i run the code, i find that it will stop because logprobs is the same as old_logprobs in agent.py

[org.freedesktop.DBus.Error.TimedOut] ("Failed to activate service 'org.freedesktop.secrets': timed out (service_start_timeout=120000ms)",)

While following the instructions in Read.md, I encountered an error after running cd poetry and poetry update. I am not sure how to resolve the error. I am using CARLA 0.9.8 version on Linux.

There may be some errors in your code

The 42nd line of code in the sensors.py file under the simulation folder,it seems like reshape((image.height, image.width, 4)) instead of reshape((image.width, iamge.height, 4)).If this error is true, your preTrained_models may need to be updated with the correct version.

The console works, but the client displays a faulty screen!

The console works, but the client displays a faulty screen!
The console works, but the client displays a faulty screen!
The client's GUI display screen, not fully displayed, is a pair of horizontal stripes! I don't know what is the reason for this. I don't know what's causing this, so I'm asking for your help! Please give me a suggestion!
The interface of the client is full of horizontal stripes, there is no way to display my car and planning level properly, the interface is full of horizontal stripes!

Encoder could not be initialized.

Couldn't import Carla egg properly
pygame 2.1.2 (SDL 2.0.18, Python 3.8.10)
Hello from the pygame community. https://www.pygame.org/contribute.html
INFO: Found the required file in cache! Carla/Maps/Nav/Town07.bin
Encoder could not be initialized.

Exit

Couldn't inport Carla egg properly

I tried to run this code on a Linux system, and replaced the corresponding libraries (such as torch, torchvision and tensorboard) with the Linux version in the project setup phase. It passed the compilation during the poetry update, but an error occurred when running the continuous_driver.py file:

(venv) ......$ python continuous_driver.py --exp-name=Ppo --train=False
Couldn't inport Carla egg properly
pygame 2.1.2(SDL 2.0.16,Python 3.7.13)
Hello fron the pygane coRnunity. https://www. pygane.org/contribute.html
Failed to nake a connection with the server: module 'carla' has no attribute 'client'
ERROR : root:connection has been refused by the server.
Exit
Traceback (most recent call last):
File "continuous_driver.py", line 297, in <module>runner()
File "continuous_driver.py" , line 115, in runner
env = CarlaEnvironment(client, world,town,checkpoint_frequency=None)UnboundLocalError: local variable 'client' referenced before assignnent

I have opened CarlaUE4Editor and started running before running the code. I opened the carla folder of the code and found only the carla-0.9.8-py3.7-win-amd64.egg file. This is not compatible with the Linux system. How can I make the code run correctly on the Linux system？

Training does not work properly and agent does not beyond first traffic light in TOWN02

Hello Idree,

I was going through your code, and I had a question about a specific part of it. I was training a new agent from scratch in TOWN02. The training was going well (the average reward is increasing every timestep), however, out of a sudden at any point the training stopped and then “exit” is printed in the command window prompt. Basically, the training is not finished, and it does not get to the end of the training (the print “Terminating the run” does never appear). I run the training several times and still doing the same situation but at different timesteps. I am making sure that in the while loop timestep < TOTAL_TIMESETPS, does not break because of this condition. I have attached a screenshot of the situation when the training stops and jumps to the printed exit. I don’t know if run into a similar issue.

Another issue is that the agent does not go beyond the first traffic light corner. I run the code until 2M timesteps but every time the agent restart and stops in that specific point. I don't not if this issue is related to training is stopped unexpected.

Thank you,
Oscar

How long training takes?

Hello,

I am currently training the PPO AV. I am using town 2 for the training process. However, the training does not seem to improve a lot every minute. It started in average reward 97. After 330 episodes, the average reward is 144. How long does it take to train a proper and safe AV? And how many episodes are required to complete a "full" training?

Thank you,
-Oscar

Description issues

From the code, it is more like a DDPG algorithms but not PPO algorithms. Besides, for the auto-encoder part, this code is not use auto-encoder to extract features of input image, which looks use encoder as the CNNs. Why not learn and update the weight of these CNNs with actor and critics network? Why need to train these CNNs separately?

Couldn't import Carla egg properly

Couldn't import Carla egg properly

You MPDF seems like the same as two single PDF ?

Hi,
I really appreciate this work! However, I think the Multivariate(steer, throttle) Probability Distribution Function in this project works as the same way like Two Single Gaussian PDF for you don't consider the Rho between with steer and throttle ?

Codes in PPO in line 19:
self.cov_var = torch.full((self.action_dim, ), action_std_init)
self.cov_mat = torch.diag(self.cov_var).unsqueeze(dim=0). It's a diag matrix which consider the Rho is 0 ? And what the difference if you use two single Gaussian PDF for sampling the actions? Any ablation experiments ?

Best wishes !