wvangansbeke / lanedetection_end2end Goto Github PK
View Code? Open in Web Editor NEWEnd-to-end Lane Detection for Self-Driving Cars (ICCV 2019 Workshop)
Home Page: https://arxiv.org/pdf/1902.00293.pdf
License: Other
End-to-end Lane Detection for Self-Driving Cars (ICCV 2019 Workshop)
Home Page: https://arxiv.org/pdf/1902.00293.pdf
License: Other
Hi Wouter,
It is an impressing work, and I hope to try other ideas based on your work. I will use it in academic purpose. May I know if the code is available at this moment? If yes, could you please share the code with me? Many thanks.
My email is [email protected]
Hi, @wvangansbeke
Thanks for your great work. The result seems impressive.
Well, I am wondering if you compare the End2End method with other lane detection methods, https://arxiv.org/abs/1712.06080 and https://arxiv.org/abs/1802.05591?
Our ENet-Label-Torch has been released. More details can be found in my repo.
Key features:
(1) ENet-label is a light-weight lane detection model based on ENet and adopts self attention distillation (more details can be found in our paper which will be published soon).
(2) It has 20 × fewer parameters and runs 10 × faster compared to the state-of-the-art SCNN, and achieves 72.0 (F1-measure) on CULane testing set (better than SCNN which achieves 71.6).
(Do not hesitate to try our model!!!)
Performance on CULane testing set (F1-measure):
Category | SCNN-Torch | SCNN-Tensorflow | ENet-Label-Torch |
---|---|---|---|
Normal | 90.6 | 90.2 | 90.7 |
Crowded | 69.7 | 71.9 | 70.8 |
Night | 66.1 | 64.6 | 65.9 |
No line | 43.4 | 45.8 | 44.7 |
Shadow | 66.9 | 73.8 | 70.6 |
Arrow | 84.1 | 83.8 | 85.8 |
Dazzle light | 58.5 | 59.5 | 64.4 |
Curve | 64.4 | 63.4 | 65.4 |
Crossroad | 1990 | 4137 | 2729 |
Total | 71.6 | 71.3 | 72.0 |
Runtime(ms) | 133.5 | -- | 13.4 |
Parameter(M) | 20.72 | -- | 0.98 |
Why the images are all black in the ground truth dataset?I downloaded it in the link provided by the author.
Hi,
first of all, thanks for making this project public, some good work and ideas here! To my issue, I'm trying to run the model in the conventional mode (not end2end) with:
python3 -W ignore main.py --image_dir /mount/ssd_middle/TuSimple_subset/images/ --gt_dir /mount/ssd_middle/TuSimple_subset/ground_truth/ --end_to_end False --nclasses 4
but get the following error message:
0it [00:00, ?it/s]Batch with idx [2236 879 2282 2079 262 1271 2015 1018] skipped due to singular matrix The size of tensor a (4) must match the size of tensor b (8) at non-singleton dimension 0
commenting out the try / catch block I'm getting the following error:
RuntimeError: The size of tensor a (4) must match the size of tensor b (8) at non-singleton dimension 0
does anybody know what might be the reason for this? Is any quick fix possible? For the moment, I'm interested in getting segmentation maps for lanes without modeling.
Many thanks in advance!
Best,
Alexey
Hello,
I am trying to perform inference in a single image as well.
Here's what I got; it's not working yet.
### Open Image
from PIL import Image
image = Image.open("10.jpg")
### 1 : Load the Model
model = Net(args)
checkpoint = torch.load("model.pth.tar")
model.load_state_dict(checkpoint['state_dict'])
### 2 : Preprocessing / Transformations
w,h = image.size
totensor = transforms.ToTensor()
resize = args.resize ### 256
image = F.crop(image, h-640, 0, 640, w)
image = F.resize(image, size=(resize, 2*resize), interpolation=Image.BILINEAR)
image = totensor(image).float()
## Pass it to the model
with torch.no_grad():
beta0, beta1, beta2, beta3, weightmap_zeros, M, output_net, outputs_line, outputs_hozizon = model(image, True)
### I also tried = model.forward(image, True)
And I have an error :
RuntimeError: Expected 4-dimensional input for 4-dimensional weight [13, 3, 3, 3], but got 3-dimensional input of size [3, 256, 512] instead
The pipeline should be usable by anyone who wants to test once fixed. Can anybody help ?
Hi
I would need to detect lanes and borders from a dataset which is LIDAR only. Is it possible ?
Thanks
Can you share file for test one image?
Hi Wouter,
Now I want to convert your model to ONNX. So I change some code in Birds_Eye_View_Loss/main.py-save_checkpoint()
def save_checkpoint(state, to_copy, epoch):
batch_size = 1
filepath = os.path.join(args.save_path, 'checkpoint_model_epoch_{}.pth.tar'.format(epoch))
torch.save(state, filepath)
dummy_input = torch.randn(batch_size, 3, args.resize, args.resize*2, requires_grad=True)
model_onnx_path = "super_resolution.onnx"
model = Net(args)
model.load_state_dict(state['state_dict'])
model.train(False)
model.cuda()
torch.onnx.export(model,
dummy_input,
"epoch_" + str(epoch) + model_onnx_path,
verbose=False
)
print("onnx well done!")
...
and I input
python main.py --image_dir ... --gt_dir ... --end_to_end fasle
BUT
Traceback (most recent call last):
File "/tmp/pycharm_project_633/venv/Scripts/pytorch2onnx.py", line 42, in <module>
verbose=False
File "/work/dependence/anaconda3/lib/python3.6/site-packages/torch/onnx/__init__.py", line 27, in export
return utils.export(*args, **kwargs)
File "/work/dependence/anaconda3/lib/python3.6/site-packages/torch/onnx/utils.py", line 104, in export
operator_export_type=operator_export_type)
File "/work/dependence/anaconda3/lib/python3.6/site-packages/torch/onnx/utils.py", line 281, in _export
example_outputs, propagate)
File "/work/dependence/anaconda3/lib/python3.6/site-packages/torch/onnx/utils.py", line 224, in _model_to_graph
graph, torch_out = _trace_and_get_graph_from_model(model, args, training)
File "/work/dependence/anaconda3/lib/python3.6/site-packages/torch/onnx/utils.py", line 192, in _trace_and_get_graph_from_model
trace, torch_out = torch.jit.get_trace_graph(model, args, _force_outplace=True)
File "/work/dependence/anaconda3/lib/python3.6/site-packages/torch/jit/__init__.py", line 197, in get_trace_graph
return LegacyTracedModule(f, _force_outplace)(*args, **kwargs)
File "/work/dependence/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
result = self.forward(*input, **kwargs)
File "/work/dependence/anaconda3/lib/python3.6/site-packages/torch/jit/__init__.py", line 252, in forward
out = self.inner(*trace_inputs)
File "/work/dependence/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 487, in __call__
result = self._slow_forward(*input, **kwargs)
File "/work/dependence/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in _slow_forward
result = self.forward(*input, **kwargs)
TypeError: forward() missing 1 required positional argument: 'end_to_end'
How can I input this required argument?
Thank you!
Student Lulu
Hello:
The dataset you provide in the link include 2664 images rather than 3626images. Where can I download the latest dataset?
By the way, when you do training in BackProjection_Loss, you set clas = 1 so it needs the test images, how can I get it and its' json file?
I set clas=0 so that the training can continue, but the training record has something wrong although it doesn't matter.
Best regards
Max
Please update the README to include instructions for testing the source code on custom dataset with pretrained weights.
Thanks and Regards,
Divye
sudo python3 main.py --image_dir /home/rossfisher/LaneDetection_End2End/DATASET/images --gt_dir /home/rossfisher/LaneDetection_End2End/DATASET/ground_truth --end_to_end True
Traceback (most recent call last):
File "main.py", line 20, in <module>
from Dataloader.Load_Data_new import get_loader, get_homography, \
ImportError: cannot import name 'write_lsq_results' from 'Dataloader.Load_Data_new' (/home/rossfisher/LaneDetection_End2End/Birds_Eye_View_Loss/Dataloader/Load_Data_new.py)
Hi Wouter, could you please provide me with the pre-trained model weights; or even share it? It would be a good hot-start. Thank you in advance!
hi Wouter:
1、in the code,Net.forward has self.net(input, end_to_end*self.pretrained),the input image is normal image not a bird's view image?
2、the net output “masked” is showed in “save_weightmap”. I get the "wm0_zeros" , the weighted picture is not matched with the detect line.as show in the picture.
i think the left ego_lines in the wm0_zeros should be show, but cannot find left ego lines infor in the wm0_zeros.
thanks you work!
I try to run model on the test set of tusimple ,I get good result!!!
but I can't show gt on image , I don't know how to get beta params of gt lane.
I tried to convert the lane coordinates of the test set to the top view, then fit the lane on the top view and get the curve parameters twice, but when I use the obtained parameters to draw the line onto the picture, I get the wrong result, may I ask? Do I need a line to normalize the coordinates and then fit?
def draw_fitted_line(img, params, resize, color=(255,0,0)):
params = params.data.cpu().tolist()
y_stop = 0.7
y_prime = np.linspace(0, y_stop, 20)
params = [0] * (4 - len(params)) + params
d, a, b, c = [*params]
x_pred = d*(y_prime**3) + a*(y_prime)**2 + b*(y_prime) + c
x_pred = x_pred*(2*resize-1)
y_prime = (1-y_prime)*(resize-1)
lane = [(xcord, ycord) for (xcord, ycord) in zip(x_pred, y_prime)]
img = cv2.polylines(img, [np.int32(lane)], isClosed = False, color = color,thickness = 1)
return img, lane
the y_prime is normalize the coordinates ? so beta parms of model output is normalize the coordinates ?
best!!
thanks
hi, Is the end2end suitable for unclassified lanes?
I downloaded the linked dataset you created, then
python3 main.py --image_dir /home/rossfisher/LaneDetection_End2End/DATASET/images --gt_dir /home/rossfisher/LaneDetection_End2End/DATASET/ground_truth --end_to_end True
size train loader is 2901
size valid loader is 725
Traceback (most recent call last):
File "main.py", line 548, in <module>
main()
File "main.py", line 89, in main
split_percentage=args.split_percentage)
File "/home/rossfisher/LaneDetection_End2End/Backprojection_Loss/Dataloader/Load_Data_new.py", line 280, in get_loader
nclasses=nclasses)
File "/home/rossfisher/LaneDetection_End2End/Backprojection_Loss/Dataloader/Load_Data_new.py", line 95, in __init__
assert len(self.rgb_lst) == len(self.gt_lst) == 3626
AssertionError
Hi,
The tusimple evaluation takes as input h samples and lane points with the size of 720 x 1280 however if we train a network on smaller size then there are two ways to go about it.
a) resize the network output image to 720 x 1280.
b) resize the points to smaller size and generate y samples/lane points pair.
Which one do you think is best and if this is considered in your code ?
hi, I tried to run you model by:
python main.py --image_dir {mypath}/tusimple/DATASET/images/ --gt_dir {mypath}/tusimple/DATASET/ground_truth/ --end_to_end True
but got the error like this:
0it [00:00, ?it/s]Batch with idx [1375 536 1493 1366 750 1664 1031 1017] skipped due to singular matrix
tensor should be 2 dimensional
The Dataset is downloaded form your link. Appropriate if you could help by this.
hi wvangansbeke:
I have some questions about "label_new.json"
label_new.json
{"position": [0, 0, 1, 1, 1, 1, 1, 0], "lines": [-1, -1, 1, 0, 0, 0, 0, 1, -1, -1], "raw_file": "clips/0313-1/6040/20.jpg"}
in the Curve_parameters.json
"lanes": [[-2, -2,...,307, 299],[-2, -2,...,1250, 1265, -2, -2, -2, -2, -2],[-2, -2,...,241, 212,...-2, -2],[-2, -2,...,1229, 1269,...-2, -2], "raw_file": "clips/0313-1/6040/20.jpg", "run_time": 20, "poly_params": [[-0.00074, 0.0480, 0.466], [-0.000110, -0.020, 0.569], [-0.00282, 0.127, 0.3673], [0.00101, -0.10, 0.69]]}
question1:how to get the position [0, 0, 1, 1, 1, 1, 1, 0],
question2:how to get the lines": [-1, -1, 1, 0, 0, 0, 0, 1, -1, -1],
in the prediscuss I found "A 1 means the line is a continuous line, a 0 means it is a dashed line and a -1 means the line is not present. The latter is really important, because the network will always output 2 or 4 lane lines." but i still can not get the right mean.
Hi,
Thank you very much for your meaningful work. I've probably reproduced the results of your paper, so I want to extend this work to different data sets.
so I need your help:
How to generate the parameters of 'poly_params' and 'position'?
I made them differently from what you provided. Would you give me a detailed description?
Have you verified in other datasets?How does the result of your paper perform in rainy days, nights and other complex weather conditions?
about train
could i ask some question?
tusimple dataset only include some images and *.json files .what is --image_dir and --gt_dir?(run code python main.py --image_dir /path/to/image/folder --gt_dir /path/to/ground_truth/folder --end_to_end True)
Hello,
I'm a hobbiest software dev who has only gotten into ML topics. OpenPilot is an open source project to add L2 autonomy to most cars 2017 and newer. The code repo is here:
https://github.com/commaai/openpilot
The community is currently constrained to Qualcomm based hardware due to the provided models being in a snpe dlc format. I instead bought a Nvidia Jetson Nano and many others are working on custom hardware solutions.
I realize that you are in academia and your intents are to further the field, so this may be out of your scope. But..
It would be fantastic if I could use this model in a real world application. The outputs from a model would be consumed here (which are used by Pathplanner/MPC to generate desired steering angles) https://github.com/commaai/openpilot/blob/devel/selfdrive/controls/lib/model_parser.py
An example of output from the existing model is here (check model file):
https://www.dropbox.com/sh/7ncns9yuk7lx910/AADWlF2y9nple5in6YXtv06ba?dl=0
We can ignore path, lead as they aren't currently used. Only left, right lane is important (I think).
An example:
I see that the existing model outputs are contained in the lane object, just the scale is wrong for this use case it seems.
It would mean a great deal to many users of OpenPilot to have an alternative model to use. One community member has offered $5,000 USD for a workable solution (we are rather passionate users).
I'm training a model now using your codebase and will continue tinkering to see if I can get something workable for OpenPilot. I am only a novice, so any assistance you could provide would be wonderful (although, I have no expectations).
Regardless, thank you for sharing your work!
Update: The 0.6 model puts out polys instead of X Y. I also have a network architecture diagram of their existing network (it's all OSS, just in dlc format that is unfriendly). I can collect any information you'd request if you'd take this on.
Hi,
Congrats on your great work. I am being kind of confused on a point in your paper.
You mentioned that there are two maps for the x, y coordinates, then, are they just representing the normalized pixel coordinates of every pixel in the image? e.g. The first colomn of X-map would be 0,0,0,0,0,0,0...? I don't see the point of having these maps if they are just pointing to themselves.
In this case, if you choose the first way of backprop, which is to update the coordinates, then do you mean that only the values in the X, Y-features maps are updating?
One last point, is there any correlation between the pixel and the location they are pointing to? Such as they might be pointing to a pixel of the same lane line?
Hey there, is there a way to find the number of lanes? I understand that there is a max of 4 lanes that the model can detect, but is there a way to find the number of lanes?
Is the final loss function derived from the lane geometry model? as shown in the paper?
Hello, I am trying to train the network on my dataset, but I do not know the details on how to do the normalization , can you please show me more? Thanks!
Hi, I am wondering when the license issue can be fixed so that we can check out the code.
Thanks!
If I train this network with another dataset, how to prepare the dataset and the code for training should be modified?
thanks.
In your paper, the transformation matrix H is kownn, but I don't know how to get the matrix, is it fixed to every image in the tusimple dataset according to the camera parameters, or it's dynamic due to different image? Thanks!
hi,
thanks for your great job, but when I read the paper and the github code, all the experiment is about ego-lane, but readme gif in the github is multi lane, I wonder how to reach this result as you, please help, thanks a lot.
Hi. I am able to train the network on the 20% TuSimple data (~100 epochs), but the result is not very good when testing with images from other sources. I think the network need to be trained more thoroughly. Is there a pre-trained weight of the network available? Could you share it with us? Thank you!
Hello, in the source code, function test_projective_transform tranforms normalized M to image size M. I cannot understand why M_scaledup
is set like that.
I'm asking why M_scaledup[1, 0]
and M_scaledup[2, 0]
are both 0? I think it ought to be: M_scaledup[1, 0] = M[1, 0] / 2
and M_scaledup[2, 0] = M[2, 0] / (2*resize-1)
Here's my deduction. Perspective transform does following things:
(where a_{ij}
is item of normalized M
)
So transforming normalized M to image size M basically does following things:
Thus M_scaledup
should be:
So every item matches your code except M_scaledup[1, 0]
and M_scaledup[2, 0]
.
Hi, wvangansbeke:
I have transplant the pytorch code to caffe and compare their precision. The output of the network is nearly the same. But when I test the network my own dataset use my caffe code, the detection result is not ideal. What should I pay attention to when I test my own dataset? And what should I do if I want to improve the performance?
Hello, I am experimenting with your model, but the result is not good, I guess there is a problem with the curve parameters that I generated from my own dataset. So I first use the M-inv in your code to make a perspective transformation of the TuSimple data and then get the curve parameters, but unlike the file Curve_parameters.json, can you tell me how to generate the details of the curve parameters?
Hi,I am very emotional about the work you have done.I want to run the model with my own Dataset.What parameters do I need to generate the in .json file? Is it just the "poly_params" parameter used?I only generated this parameter and can run the code, but the final prediction result is not ideal. Do you have any suggestions?
Hi @wvangansbeke , Could you please introduce how to generate curve parameter? Thanks.
Excuse me, I have several questions as below:
1. Did you generate the coefficient label on the 256x512 image or on the original 720x1280 image?
2. Why did you do the (1-y_prime) operation on y_prime when draw the lines? (in utils.py: def draw_fitted_line)
hi Wouter:
in the issues i found “ I left some images out for training were the car is switching lanes, since you'll need an instance segmentation to handle those cases”
why these switching lanes cannot detect by the method.
"I used a second branch in decoder which told me which one is present in the scene" i can not find the second branch in the code? is it not released?
i can find“if self.nclasses > 3” support 4 lines.
"maximum number of lane lines is a hyperparameter"means I used n output maps in order to detect a maximum of n lane lines.(n is 5 or biger), is it right?
Hi , I used the Tusimple dataset to generate the training label (the value of β) according to the description in your paper. There are several questions I would like to ask you:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.