A demo of Lane Departure Warning System: a monocular camera is used for detecting current lane, tracking the vehicle position and estimating the front road status.
Video demo (Click to see the full video)
(Note: the hyperlinks only works if you are on the homepage of this GitHub reop, and if you are viewing it in "github.io" you can be redirected by clicking the View the Project on GitHub on the top)
- calibration.py contains the script to calibrate camera and save the calibration results
- main.py is the main code for 2 demos
- lane.py contains the lane class
- camera_cal folder contains the images used for camera calibration and calibration results
- examples folder contains the sample images and videos
- README.md summarizing the results
Anaconda is used for managing my dependencies.
- OpenCV3, Python3.5
- OS: Ubuntu 16.04 (should work on other platform too)
If you want to run the demo, you can simply run:
python main.py
If you want to use the code to calibration your own camera and test the video, save the image into camera_cal and calibrate them with:
python calibration.py
-
0.1.2
- Update the README.md
- Date 19 April 2017
-
0.1.1
- Edit the visulization flag in lane.py
- Date 3 March 2017
-
0.1.0
- The first proper release
- Date 3 March 2017
- Gather a set of chessboard images with camera
- Compute the camera calibration matrix and distortion coefficients with given a set images.
- Apply a distortion correction to raw images.
- Use color transforms, gradients, etc., to create a thresholded binary image.
- Apply a perspective transform to rectify binary image ("birds-eye view").
- Detect / track lane pixels and fit to find the lane boundary.
- Determine the curvature of the lane
- Compute the vehicle position with respect to center.
- Warp the detected lane boundaries back onto the original image.
- Print the road status into image
The code for this step is contained calibration.py. It start by preparing "object points", which will be the (x, y, z) coordinates of the chessboard corners in the world. Here I am assuming the chessboard is fixed on the (x, y) plane at z=0, such that the object points are the same for each calibration image. Thus, objp
is just a replicated array of coordinates, and objpoints
will be appended with a copy of it every time I successfully detect all chessboard corners in a test image. imgpoints
will be appended with the (x, y) pixel position of each of the corners in the image plane with each successful chessboard detection.
I then used the output objpoints
and imgpoints
to compute the camera calibration and distortion coefficients using the cv2.calibrateCamera()
function. I applied this distortion correction to the test image using the cv2.undistort()
function and obtained this result:
Function process_frame()
in lane.py
contains every step described below.
In this step, cv2.undistort
is used with the previous calibration results (see lane.py line 580). One of example for test images is as follows:
I used a combination of color and gradient (thresholds to generate a binary image (see function find_edges
in lane.py
line 187).
The image is firstly converted to HLS space, and I use the S channel to filter the image, by which is more robust to locate yellow and white lanes even under different lighting conditions.
Then a sobel filter (along x direction) and gradient direction filter is used to filter out most of the horizontal lines. Finally, to handle the cases where more than two candidate lanes are found,
I assgin the twice the weights to the S channel filter than the gradient filter. As a result, the yellow lane should be more visible than the edge of the road. Here's an example of my output for this step:
The code for my perspective transform includes a function called warper()
(in lane.py
line 214). Firstly, I choose a
straight line driving image, and select 4 source points which form a trapezoidal along two lane lines, then I defined another 4 destination points,
so that I could get a perspective transform with cv2.getPerspectiveTransform
function (see the two images below). The hardcoded source points and destination points are
listed as follows.
This resulted in the following source and destination points:
Source | Destination |
---|---|
194, 719 | 290, 719 |
1117, 719 | 990, 719 |
705, 461 | 990, 0 |
575, 461 | 290, 0 |
I verified that my perspective transform was working as expected by drawing the src
and dst
points onto a test image and its warped counterpart to verify that the lines appear parallel in the warped image.
As we could probably noticed that the image are not in the right aspect ratio (flattened lanes), but it will have no effect on the final augmented image if we use the right inverse transform to warp the image back.
If we want to see the image in a more right scale, we could also select the destination points according to the right ratio of lane width and length in the trapezoidal. One example could be as follows:
After applying calibration, thresholding, and a perspective transform to a road image, we should have a binary image where the lane lines stand out clearly.
However, we still need to decide explicitly which pixels are part of the lines and which belong to the left line and which belong to the right line.
If the input is single image or first frame of the video, a function named detector
(line 548 in lane.py) is used. It processes the image as follows
-
I first take a histogram along all the columns in the lower half of the image. The two most prominent peaks in this histogram will be good indicators of the x-position of the base of the lane lines. I can use that as a starting point for where to search for the lines. as a reuslt I will get similar figure as follows:
-
From that point, I use a sliding window, placed around the line centers, to find and follow the lines. It starts at the bottom, and search up to the top of the frame. The idea is visualized as the following figure.
-
Finally, I fit my lane lines with a 2nd order polynomial by
np.polyfit
like inlane.py
line 293:
- If the input is video, the first frame can be treated the same as the individual image. Then we will have the estimated position of left and right lanes,
so it will be wise to take their positions as initiates for finding the lanes in the next frame. Therefore, a function named
tracker
inlane.py
is used for this case. - In order to get smooth lanes among a sequence of frame, I use a frame buffer to save the lanes' positions in previous N frame (line 25 in
lane.py
). - If tracking fail (e.g. high standard deviation of the distance between left and right lanes), either keep previous lane positions or start a new detection.
5. Calculate the radius of curvature of the lane and the position of the vehicle with respect to center.
The radius of curvature of the curve at a particular point is defined as the radius of the approximating circle. You can see the reference here.
Function measure_lane_curvature
in lane.py
is implemented for this purpose.
We assume the camera is mounted roughly at the center of the front window, and we have computed positions of the left and right lanes. So we could just take the bottom position of the left and right lanes, and compare with the middle of the image frame. Also, we could utilize the prior knowledge of the width of between left and right lanes (e.g. 3.7 m in US) to estimate the distance in real scale.
A function named off_center
in lane.py
is implemented for this purpose.
Finally, we could warp the detected lane region back to the original input frame. If the vehicle is driving too close to the lane edge, we could
warn the driver by changing the color from green to red (see function create_output_frame
in lane.py
).
In order to be useful, driver assistance systems should reach very low error rates. For a warning system like LDW, the false alarm rate should be very low, as high rates irritate drivers and lead to system rejection. The exact amount of false alarms acceptable by drivers is still a subject for research [1]. Also, there are various of scenarios diversity which road and lane detection should cope with, and Lane and road appearance can change dramatically as the following figure.
Current implementation is purely based on monocular camera with computer vision techniques, it involves hard-code perspective transform and many manually tuned parameters for lane-line filtering. This maybe sufficient for good condition road driving, e.g. highway, daytime, clear lane marks, but it may fail with vairous of hard scenarios. In the future, we could consider two approaches:
- Use more sensing modalities other then only monocular camera when possible, e.g. Stereo camera, lidar, radar.
- Adopt machine learning techniques.
Reference: [1] Recent Progress in Road and Lane Detection - A survey