koyeongmin / pinet_new

Update

2020-11-06: New trained model for TuSimple ("0_tensor(0.5242)_lane_detection_network") (Acc: 96.81%, FP: 0.0387, FN: 0.0245, threshold = 0.36)

key points estimation and point instance segmentation approach for lane detection

• Yeongmin Ko, Younkwan Lee, Shoaib Azam, Farzeen Munir, Moongu Jeon, Witold Pedrycz
• Abstract: Perception techniques for autonomous driving should be adaptive to various environments. In the case of traffic line detection, an essential perception module, many condition should be considered, such as number of traffic lines and computing power of the target system. To address these problems, in this paper, we propose a traffic line detection method called Point Instance Network (PINet); the method is based on the key points estimation and instance segmentation approach. The PINet includes several stacked hourglass networks that are trained simultaneously. Therefore the size of the trained models can be chosen according to the computing power of the target environment. We cast a clustering problem of the predicted key points as an instance segmentation problem; the PINet can be trained regardless of the number of the traffic lines. The PINet achieves competitive accuracy and false positive on the TuSimple and Culane datasets, popular public datasets for lane detection.
• link: https://arxiv.org/abs/2002.06604

Dependency

• python ( We tested on python3, python2 is also work, but spline fitting is supported only on python3 )
• pytorch ( We tested on pytorch 1.1.0 with GPU(RTX2080ti))
• opencv
• numpy
• visdom (for visualization)
• sklearn (for evaluation)
• ujon (for evaluation)
• csaps (for spline fitting)

Dataset (TuSimple)

You can download the dataset from TuSimple/tusimple-benchmark#3. We recommand to make below structure.

dataset
  |
  |----train_set/               # training root 
  |------|
  |------|----clips/            # video clips, 3626 clips
  |------|------|
  |------|------|----some_clip/
  |------|------|----...
  |
  |------|----label_data_0313.json      # Label data for lanes
  |------|----label_data_0531.json      # Label data for lanes
  |------|----label_data_0601.json      # Label data for lanes
  |
  |----test_set/               # testing root 
  |------|
  |------|----clips/
  |------|------|
  |------|------|----some_clip/
  |------|------|----...
  |
  |------|----test_label.json           # Test Submission Template
  |------|----test_tasks_0627.json      # Test Submission Template

Next, you need to change "train_root_url" and "test_root_url" to your "train_set" and "test_set" directory path in "parameters.py". For example,

# In "parameters.py"
line 54 : train_root_url="<tuSimple_dataset_path>/train_set/"
line 55 : test_root_url="<tuSimple_dataset_path>/test_set/"

Finally, you can run "fix_dataset.py", and it will generate dataset according to the number of lanes and save dataset in "dataset" directory. (We have uploaded dataset. You can use them.)

Dataset (CULane)

You can download the dataset from https://xingangpan.github.io/projects/CULane.html.

If you download the dataset from the link, you can find some files and we recommand to make below structure.

dataset
  |
  |----train_set/               # training root 
  |------|
  |------|----driver_23_30frame/
  |------|----driver_161_90frame/
  |------|----driver_182_30frame/
  |
  |----test_set/               # testing root 
  |------|
  |------|----driver_37_30frame/
  |------|----driver_100_30frame/
  |------|----driver_193_90frame/
  |
  |----list/               # testing root 
  |------|
  |------|----test_split/
  |------|----test.txt
  |------|----train.txt
  |------|----train_gt.txt
  |------|----val.txt
  |------|----val_gt.txt

Test

We provide trained model, and it is saved in "savefile" directory. You can run "test.py" for testing, and it has some mode like following functions

• mode 0 : Visualize results on test set
• mode 1 : Run the model on the given video. If you want to use this mode, enter your video path at line 63 in "test.py"
• mode 2 : Run the model on the given image. If you want to use this mode, enter your image path at line 82 in "test.py"
• mode 3 : Test the model on whole test set, and save result as json file.

You can change mode at line 22 in "parameters.py".

If you want to use other trained model, just change following 2 lines.

# In "parameters.py"
line 13 : model_path = "<your model path>/"
# In "test.py"
line 42 : lane_agent.load_weights(<>, "tensor(<>)")

• TuSimple If you run "test.py" by mode 3, it generates "test_result.json" file. You can evaluate it by running just "evaluation.py".

• CULane The evaluation code is forked from https://github.com/harryhan618/SCNN_Pytorch. The repository ported official evaluation code and provide the extra CMakeLists.txt.

cd evaluation_code/
mkdir build && cd build
(remove default build directory, it is my mistake)
cmake ..
make

If you run "test.py" by mode 3, it generates result files in the defined path (the path is defined by test.py). The generated file can be evaluated by the following:

./evaluation_code/Run.sh <file_name>

Before running it, you should ckech path in Run.sh

Train

If you want to train from scratch, make line 13 blank in "parameters.py", and run "train.py"

# In "parameters.py"
line 13 : model_path = ""

"train.py" will save sample result images(in "test_result/"), trained model(in "savefile/").

If you want to train from a trained model, just change following 2 lines.

# In "parameters.py"
line 13 : model_path = "<your model path>/"
# In "train.py"
line 54 : lane_agent.load_weights(<>, "tensor(<>)")

Network Clipping

PINet is made of several hourglass modules; these hourglass modules are train by the same loss function.

We can make ligher model without addtional training by clipping some hourglass modules.

# In "hourglass_network.py"
self.layer1 = hourglass_block(128, 128)
self.layer2 = hourglass_block(128, 128)
#self.layer3 = hourglass_block(128, 128)
#self.layer4 = hourglass_block(128, 128) some layers can be commentted 

Result

You can find more detail results at our paper.

4 horglass modules is run about 25fps on RTX2080ti

• TuSimple (4 hourglass modules)

Accuracy	FP	FN
96.75%	0.0310	0.0250

• CULane (4 hourglass modules)

Category	F1-measure
Normal	90.3
Crowded	72.3
HLight	66.3
Shadow	68.4
No line	49.8
Arrow	83.7
Curve	65.6
Crossroad	1427 （FP measure）
Night	67.7
Total	74.4

pinet_new's Issues

Something Wrong in the "hardloss"?

` hard_loss = 0

    for i in range(real_batch_size):
        confidance_gt = ground_truth_point[i, 0, :, :]
        confidance_gt = confidance_gt.view(1, self.p.grid_y, self.p.grid_x)
        hard_loss =  hard_loss +\
            torch.sum( (1-confidance[i][confidance_gt==1])**2 )/\
            (torch.sum(confidance_gt==1)+1)

        target = confidance[i][confidance_gt==0]
        hard_loss =  hard_loss +\
			torch.sum( ( target[target>0.01] )**2 )/\
			(torch.sum(target>0.01)+1)

        node = hard_sampling.sampling_node(loss = hard_loss.cpu().data, data = data_list[i], previous_node = None, next_node = None)
        self.hard_sampling.insert(node)

you add the hardloss along batches, and insert every single node use the added loss.
Is it right? Maybe should not use the added loss

generate result problem

why do you remove eliminate_out function in post-processing?

I found that post processing for eliminating outliers is removed, can you tell me what's wrong with this function, thank you

2 questions about “make_dense_x”

Thanks for your update again！
I am confused about the function ““make_dense_x”” in "data_loader.py".
1: Why need make x coordinate more dense？
2: When I test the function,
If l=[500,400,300,200,100], h = [100,200,300,400,500], a very simple line.
But The out_x will be [400 400 410 420 430 440 450 460 470 480 490 300 300 310 320 330 340 350
360 370 380 390 200 200 210 220 230 240 250 260 270 280 290 100 100 110
120 130 140 150 160 170 180 190],
out_y = [200. 0. 10. 20. 30. 40. 50. 60. 70. 80. 90. 300. 100. 110. 120. 130. 140. 150. 160. 170. 180. 190. 400. 200. 210. 220. 230. 240.
250. 260. 270. 280. 290. 500. 300. 310. 320. 330. 340. 350. 360. 370.
380. 390.]
It occurs many bad points.

what's new? what's updates? More detail?

Would u post more detail? Comparison with previous model? Comparison with UltrafastLaneDetection?

culane evaluation

Hello, I encountered a problem. When running test.py in mode 3, the progress bar automatically stopped running when it reached 33%, and no error was reported. What is the reason?

Score calculation

i

Running PINet_new using only CPU?

Hello all,

I am trying to run CULane code just by using CPU (my machine does not have a dedicated GPU). I changed maplocation = torch.device('cpu') both in serialization.py (line 486) and agent.py (line 378). But I am getting the following error:

(tdata2) C:\Users\...\PINet_new_cpu\CULane>python test.py
Testing
Get agent
model parameters:
4056849
Setup GPU mode
Testing loop
Traceback (most recent call last): File "test.py", line 382, in <module> Testing() File "test.py", line 96, in Testing _, _, ti = test(lane_agent, np.array([test_image])) File "test.py", line 277, in test result = lane_agent.predict_lanes_test(test_images) File "C:\Users\...\PINet_new_cpu\CULane\agent.py", line 346, in predict_lanes_test inputs = Variable(inputs).cuda() File "C:\Users\...\Anaconda3\envs\tdata2\lib\site-packages\torch\cuda\__init__.py", line 186, in _lazy_init _check_driver() File "C:\Users\...\Anaconda3\envs\tdata2\lib\site-packages\torch\cuda\__init__.py", line 61, in _check_driver raise AssertionError("Torch not compiled with CUDA enabled") AssertionError: Torch not compiled with CUDA enabled

Is possible to run this code without a dedicated GPU? If so, which are the changes I should do?

Translation and Rotate

Hi, I have some question about Translation and Rotate in data_loader.py.
https://github.com/koyeongmin/PINet_new/blob/fead6baf0210a11444bf02d8fda7b3a33cec8e8c/TuSimple/data_loader.py#L384
Why don't you process target_h when you translate and rotate the image in case that target_h is out of [0, y_size-1]?

Training with CULane

about training

Hello, when I used Tusimple to retrain the model according to the parameters in the project, I found that the accuracy was only 80%. Could you tell me the specific training parameters adjustment details? Thank you very much!

learn rate question

I have some question for trim. I would appreciate it if you could answer me.
If I comment out trim of line 297 for Tusample/agent.py
How do I handle those trim of 300, 304, 306......
I think the learning rate and decay value has been given in parameter.py. Can I set trim=0 in line 297 or remove other lines trim directly.
Because when I did not comment out trim on line 297. I got a different 804_tensor result. My accuracy is not high.
If I comment out trim in line 297 as authors changed, I'll get some bugs about "not finding trim value".
I would be grateful if you could answer my question.
Thanks.

Refer to parameters

Hello, author, I see you provide the training process, when I use Tusimple training, do I need to divide a part of the validation set additionally? I don't seem to see the validation set in your file structure, so how do I do the tuning, do I look at the test results during training to adjust the hyperparameters? Very much looking forward to your reply, I would appreciate it!

Support for Distributed training

Hi,

I'm trying to reimplement PINet and making it compatible with PyTorch Distributed Data-Parallel (DPP) training.
I'm having trouble solving some errors and I was wondering if you can provide the code you used for distributed training.

This would greatly help me in comparing the code and solving the issue.

Add entropy loss

How can I lightweight the PINet model?

I have some questions. Is there any solution to lightweight the PINet model?
It seems good accuracy of detecting lanes, but the fps is quite low, so I wonder if there's any solution.

Thank you.

A question about multi-gpu training

Hello,how could i set a multi-gpu training mode,where should i modify the code?

about c++ implementation

Thank you very much for the author's work, can you provide tensorrt support？

Is the accuracy evaluation method of tusimple data set correct?

500 epochs were trained, and the accuracy rate of writing to the file has been 0.0000000000000x
for example:
epoch : 499 loss : 1.0181945085525513 [{"name":"Accuracy","value":0.0003594536304816679,"order":"desc"},{"name":"FP","value":0.00017972681524083394,"order":"asc"},{"name":"FN","value":0.9996405463695184,"order":"asc"}]

[BUG] l2 norm for feature instance, match metric `threshold_instance`

PINet_new/TuSimple/test.py

Line 356 in 59aa2b8

dis = np.linalg.norm((feature[i] - j)**2)

How to evaluate on CULane dataset

In the command for evaluation of CULane dataset, what is meant by "file name". There is no path defines for mode 3 in test.py.

save as Jason file problem

Hey :),

first thanks for the nice approach.

I'm facing one problem when saving the results on a Jason file for the test data. In line 122 of test.py there is result_data = write_result(x_, y_, path) but this method does not return any result_data. I've looked at the other PINet GitHub code and there you had a specific method result_data = write_result_json(result_data, x, y, testset_index) which produced this result data. Could you clarify on this ? :)

Thanks!

test result matching

hello, i used the model provided in the directory of savefile, and inference on CULane dataset.
the actual results are as follow:
normal: 89.9
crowd: 71.9
hlight: 64.1
shadow: 66.8
noline: 47.1
arrow 83.7
curve: 63.6
cross: 1033
night: 66.1

Is there anything wrong with the evaluation settings?

hope for your reply, thanks.

Frame rate of the video

When I test the model on mode 1, the frame rate of the output video is significantly low as compared to the input video. What changes do I need to do to increase the frame rate of the output videos or for real-time testing on the road.

Question about instance loss calculation

Hi,
Thanks for making this great work public.

I have a question regarding the calculation of instance loss for PINet of CULane dataset. could you please explain what exactly is happening in the following lines?

compute loss for similarity

feature_map = feature.view(real_batch_size, self.p.feature_size, 1, self.p.grid_y*self.p.grid_x)
feature_map = feature_map.expand(real_batch_size, self.p.feature_size, self.p.grid_y*self.p.grid_x, 
                                                          self.p.grid_y*self.p.grid_x)#.detach()

point_feature = feature.view(real_batch_size, self.p.feature_size, self.p.grid_y*self.p.grid_x,1)
point_feature = point_feature.expand(real_batch_size, self.p.feature_size, self.p.grid_y*self.p.grid_x, 
                                                            self.p.grid_y*self.p.grid_x)#.detach()

distance_map = (feature_map-point_feature)**2 
distance_map = torch.sum( distance_map, dim=1 ).view(real_batch_size, 1, self.p.grid_y*self.p.grid_x, 
                                             self.p.grid_y*self.p.grid_x)

As per my understanding (please correct me if I'm wrong):

The feature_map and point_feature are derived from the output of the feature branch of PINet by doing some resizing and expanding operations. and then these feature_map and point_feature are used to calculate the distance_map.

My exact question is:

what is the difference between feature_map and point_feature?? and what is the significance of feature_map and point_feature?

Thanks :)

About the setting of the lane number.

In the CULane folder, which parameter in the file 'parameters.py' represents the max lane number of the dataset? Thanks for your reply!

About data augmentation ‘Change_intensity’?

In the 'Change_intensity' data augmentation, you only change the v channel values in hsv color space. Could you please tell me how useful is this operation for the performance?

Where can I find the total score of Culane data set?

Where can I find the total score of Culane data set? The result file after my evaluation only has the evaluation results from out0_NORMAL to out8_NIGHT, but I can't see the overall "total score".The results are shown below:

file: /home/user/CarProject/Code/pinet_new/CULane/evaluation_outputs/out0_normal.txt
tp: 26040 fp: 4548 fn: 6737
precision: 0.851314
recall: 0.79446
Fmeasure: 0.821905

file: /home/user/CarProject/Code/pinet_new/CULane/evaluation_outputs/out1_crowd.txt
tp: 14915 fp: 5548 fn: 13088
precision: 0.728877
recall: 0.532622
Fmeasure: 0.615483

file: /home/user/CarProject/Code/pinet_new/CULane/evaluation_outputs/out2_hlight.txt
tp: 751 fp: 267 fn: 934
precision: 0.737721
recall: 0.445697
Fmeasure: 0.555679

file: /home/user/CarProject/Code/pinet_new/CULane/evaluation_outputs/out3_shadow.txt
tp: 1045 fp: 651 fn: 1831
precision: 0.616156
recall: 0.363352
Fmeasure: 0.45713

file: /home/user/CarProject/Code/pinet_new/CULane/evaluation_outputs/out4_noline.txt
tp: 3766 fp: 1814 fn: 10255
precision: 0.67491
recall: 0.268597
Fmeasure: 0.384266

file: /home/user/CarProject/Code/pinet_new/CULane/evaluation_outputs/out5_arrow.txt
tp: 2033 fp: 522 fn: 1149
precision: 0.795695
recall: 0.638906
Fmeasure: 0.708733

file: /home/user/CarProject/Code/pinet_new/CULane/evaluation_outputs/out6_curve.txt
tp: 688 fp: 323 fn: 624
precision: 0.680514
recall: 0.52439
Fmeasure: 0.592337

file: /home/user/CarProject/Code/pinet_new/CULane/evaluation_outputs/out7_cross.txt
tp: 0 fp: 781 fn: 0
precision: 0
recall: -1
Fmeasure: 0

file: /home/user/CarProject/Code/pinet_new/CULane/evaluation_outputs/out8_night.txt
tp: 10221 fp: 3348 fn: 10809
precision: 0.753261
recall: 0.48602
Fmeasure: 0.590826

class labels for the lane boundaries of the TuSimple dataset

Hi,
Is it possible to include class labels to the lanes and group instances to a specific class (category) in the embedding feature?
Does the feature size somehow refer to the different instances that are detected?

Thanks :)

For annotation

I wonder that which tool or how did you annotation the data set , it will be soooo helpful for my study!!!
Thank you.

learning rate value is a typo

agent.py in culane version
is this "trim" value reasonable?
### trim = 180
if epoch>0 and epoch%100==0 and self.current_epoch != epoch:
self.current_epoch = epoch
if epoch == 1-trim:
self.p.l_rate = 0.0005
self.setup_optimizer()
elif epoch == 2-trim:
self.p.l_rate = 0.0002
self.setup_optimizer()
elif epoch == 3-trim:
self.p.l_rate = 0.0001
self.setup_optimizer()
elif epoch == 5-trim:
self.p.l_rate = 0.00005
self.setup_optimizer()
elif epoch == 7-trim:
self.p.l_rate = 0.00002
self.setup_optimizer()

agent.py in tusimple version
is this "trim" also value reasonable?
trim = 180 #70+30+70 + 110
if epoch>0 and epoch%100==0 and self.current_epoch != epoch:
self.current_epoch = epoch
if epoch == 30-trim:
self.p.l_rate = 0.0005
self.setup_optimizer()
elif epoch == 60-trim:
self.p.l_rate = 0.0002
self.setup_optimizer()
elif epoch == 90-trim:
self.p.l_rate = 0.0001
self.setup_optimizer()
elif epoch == 100-trim:
self.p.l_rate = 0.00005
self.setup_optimizer()
elif epoch == 110-trim:
self.p.l_rate = 0.00002
self.setup_optimizer()

How to build my own dataset?

Some questions about testing and training

When I modify Parameters.py 63 line Mode=2 or 3, training and testing, progress bar end at 67%, 464/695，with no error

模式1怎么添加路径

新手，刚学，请问模式1怎么添加路径啊，代码怎么改

some trouble in train 1H

I've commented <self.layer4 = hourglass_block(128, 128)>、<self.layer3= hourglass_block(128, 128)>、<self.layer2 = hourglass_block(128, 128)> and <result4, out, feature4 = self.layer4(out)>、<result3, out, feature3 = self.layer3(out)>、<result2, out, feature2 = self.layer2(out)> , also I modify the <return [result1, result2, result3, result4], [feature1, feature2, feature3, feature4]> to <return [result1], [feature1]>, and I changed "index=[3]" to "index=[0]",

but there was a problem with the evaluation，The error is displayed as follows “'int' object has no attribute 'data'” .
I would appreciate it if the author could tell me how to solve this problem.

l2 norm calcuate error

PINet_new/CurveLanes/test.py

Line 439 in fead6ba

dis = np.linalg.norm((feature[i] - j)**2)

Is it should be dis = np.linalg.norm((feature[i] - j))?

why set 4 of 'feature_size' in parameters.py

Why set the
feature_size = 4 ?
Can I set other values ？

distributed training?

It seems that your code doesn't support distributed training. I’m trying to translate your code to mmdetection? I have some doubts:

1. How much does the hard sampling improve the final performance?
2. The scheduler in your code is quite strange. Can I use another one such as CosineAnnealing or MultiStep?

Looking forward to your reply. Thanks!

Unreachable code!!! Doesn't perform the intended operations defined in the exit loop

PINet_new/TuSimple/agent.py

Line 99 in 59aa2b8

while False:

Questions about data normalization

hi @koyeongmin thanks for your execellent work, it's really impressive!
I noticed that you didn't do data normalization like data=(data-mean)/stddev. Is it better to normalize data or do batchnorm at the input layer?

About Training Strategy

Hello, could you please tell me what is the initial learning rate and learning rate schedule to train from scratch on TuSimple dataset? Thanks a lot.

Does it have switch lane problem as UltraFastLaneDetection?

UltraFastLaneDetection this model has switch lane mess up problem, does PINet have this issue?

ConnectionRefusedError

During the train process, also show this error:
Traceback (most recent call last):
File "/home/lin/anaconda3/envs/PINet/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/home/lin/anaconda3/envs/PINet/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/home/lin/anaconda3/envs/PINet/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
Does it matter? what this error for?

data_loader.py

line419    x[j][y[j]<0] = -2
line420    x[j][y[j]>=self.p.y_size] = -2

why x ? maybe:

y[j][y[j]<0] = -2
y[j][y[j]>=self.p.y_size] = -2

???

and in agent.py
line 99 while False ???
line 127-129

if temp[i]>0:
         gt_one[temp==temp[i]] = 1            # same instance
         if temp[i] == 0:

???

where is the function get_angle_two_points quoted in util.py

so confusing

about solid and dashed lines

hi, can PINet distinguish between dashed and solid lines? If I want to detect dashed lines, solid lines and diversion lines, and distinguish them, how should I label the data? thanks

After the evaluation of Culane dataset, only the value of FP was obtained, and all other values were 0

Evaluating the results...
tp: 0 fp: 7575 fn: 0
no ground truth positive
finished process file
precision: 0
recall: -1
Fmeasure: 0

How train on custom data

Hi, just wonder, if training on custom data, do we need keypoints annotation or simple using label.png which contains lane index segment in pixel values like CULane does?

Any instructions would be appreciated!

Pre-trained

연구원님 논문 잘봤습니다. 결과가 정말 훌륭하네요. ^^

논문 결과를 보고싶어서 학습완료된 모델을 찾아봤는데, 테스트에 사용된 모델은 없더군요.( 그 부분은 공개가 힘드신 부분인가요?)

제공해주신 코드로 학습을 해야하나요?

About CULane Evaluation

when I run "make", I met the following problem:

Scanning dependencies of target evaluate
[ 20%] Building CXX object CMakeFiles/evaluate.dir/src/evaluate.cpp.o
[ 40%] Building CXX object CMakeFiles/evaluate.dir/src/counter.cpp.o
[ 60%] Building CXX object CMakeFiles/evaluate.dir/src/lane_compare.cpp.o
[ 80%] Building CXX object CMakeFiles/evaluate.dir/src/spline.cpp.o
[100%] Linking CXX executable ../evaluate
/lib/x86_64-linux-gnu/libm.so.6: undefined reference to `__strtof128_nan@GLIBC_PRIVATE'
collect2: error: ld returned 1 exit status
CMakeFiles/evaluate.dir/build.make:190: recipe for target '../evaluate' failed
make[2]: *** [../evaluate] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/evaluate.dir/all' failed
make[1]: *** [CMakeFiles/evaluate.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

The same problem as: https://stackoverflow.com/questions/68480926/lib-x86-64-linux-gnu-libm-so-6-undefined-reference-to-strtof128-nanglibc-p

Can you help me? Thanks!

Add dataset of CurvedLanes training?

Is this dataset able to work with PINet?

https://github.com/xbjxh/CurveLanes