ziliharvey / smart-annotation-pointrcnn

A PointRCNN version of SAnE, which is a web-based semi-automatic annotation tool for point cloud data.

smart-annotation-pointrcnn

SANE-PointRCNN, a browser-based 3D bounding boxes annotation tool assisted by PointRCNN. This works is based on H. A. Arief's paper and code

Environment

Tested on Debian 9.9, Cuda: 10.0, Python: 3.6, Pytorch: 1.2.0 with Anaconda

Installation

git clone --recursive https://github.com/ziliHarvey/smart-annotation-pointrcnn.git
cd app/PointCNN/
sh build_and_install.sh

Also install all necessary libraries using conda, such as flask, easydict,tqdm, tensorboardX, etc.

Usage

cd app
python app.py

Open your browser and then go to http://0.0.0.0:7772. The first time loading will be relatively slow and the rest will be very fast.
For detailed instructions on annotating your own data, please refer to Docs.

Progress

• Reorganized the code base by running PointRCNN as backend
• Fully-Automated-Bbox click
• Segmented object points display
• One-click annotation by holding A key and click on the point
• Fix heading angle in boxes display
• Display all LiDAR points with corresponded point labels
• Modify dataLoader to run on the specied file
• Modify dataLoader to run for inference without ground truth
• JSON and KITTI-format conversion and offline visualization
• remove legacy code and files and clean
• Tracking, etc.

Contact

Zi Li
Kartik Sah

smart-annotation-pointrcnn's Issues

Program stuck

Hi, thank you for your excellent work. I have come across some problems and would like to ask you for advice. I am stuck when running a program and the browsing page keeps not showing up when I open it, have you ever encountered this? Or because my IP is mainland China, does this have any effect?

does this tool only can afford to annotate 50 files a time ?

does this tool only can afford to annotate 50 files a time ?

automation annotation is not good as readme

Hello,

thanks for sharing this project.

I have tried this tool from my side and using automation annotation feature by hold A key, but the result is not good as your demo.

Could you give more suggestion that how can I get better result? Do I need to train the model
again?

no model named iou3d_cuda

I am having this assertion error of no module named iou3d_cuda. Please how do I go about it...any help will do

Display files on the frontend for data type other than .ply

Visualize data and 3d box from the third party tool to validate correctness

Assertion error from frame_handler.py

Hi,
I have git cloned (git clone --recursive https://github.com/ziliHarvey/smart-annotation-pointrcnn.git
) and build the files in Ubuntu 18 with python 3.6, Cuda 10.1 and pytorch==1.2.0 torchvision==0.4.0 . I have got the following assertion error from frame_handler.py upon opening the tool in http://0.0.0.0:7772/ . Can you check on this?

(smartannotation) teroot@Nuvo-5000-Perception:~/Uhnder_OGM/Annotator/smart-annotation-pointrcnn/app$ python app.py
object_types_reverse {0: 'vehicle'}

• Serving Flask app "app" (lazy loading)
• Environment: development
• Debug mode: on
• Running on http://0.0.0.0:7772/ (Press CTRL+C to quit)
• Restarting with stat
  object_types_reverse {0: 'vehicle'}
• Debugger is active!
• Debugger PIN: 863-389-179
  127.0.0.1 - - [28/Jan/2021 14:34:17] "GET / HTTP/1.1" 200 -
  127.0.0.1 - - [28/Jan/2021 14:34:18] "POST /loadFrameNames HTTP/1.1" 200 -
  127.0.0.1 - - [28/Jan/2021 14:34:18] "GET /favicon.ico HTTP/1.1" 200 -
  =============================================
  =============================================
  Processing data begines......................
  /home/teroot/Uhnder_OGM/Annotator/smart-annotation-pointrcnn/app/PointCNN/tools/../lib/config.py:187: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  yaml_cfg = edict(yaml.load(f))
  2021-01-28 14:34:19,417 INFO Start logging
  2021-01-28 14:34:19,417 INFO cfg_file cfgs/argo_config_sampling_trainfull.yaml
  2021-01-28 14:34:19,418 INFO eval_mode rcnn
  2021-01-28 14:34:19,418 INFO test True
  2021-01-28 14:34:19,418 INFO rpn_ckpt checkpoint_epoch_50.pth
  2021-01-28 14:34:19,418 INFO rcnn_ckpt checkpoint_epoch_40.pth
  2021-01-28 14:34:19,418 INFO ckpt_dir None
  2021-01-28 14:34:19,418 INFO batch_size 1
  2021-01-28 14:34:19,418 INFO save_result True
  2021-01-28 14:34:19,418 INFO single_file 000
  2021-01-28 14:34:19,418 INFO cfg.TAG: argo_config_sampling_trainfull
  2021-01-28 14:34:19,418 INFO cfg.CLASSES: VEHICLE
  2021-01-28 14:34:19,418 INFO cfg.INCLUDE_SIMILAR_TYPE: False
  2021-01-28 14:34:19,418 INFO cfg.AUG_DATA: False
  2021-01-28 14:34:19,418 INFO cfg.AUG_METHOD_LIST: ['rotation', 'scaling', 'flip']
  2021-01-28 14:34:19,418 INFO cfg.AUG_METHOD_PROB: [1.0, 1.0, 0.5]
  2021-01-28 14:34:19,418 INFO cfg.AUG_ROT_RANGE: 18
  2021-01-28 14:34:19,418 INFO cfg.GT_AUG_ENABLED: False
  2021-01-28 14:34:19,418 INFO cfg.GT_EXTRA_NUM: 15
  2021-01-28 14:34:19,418 INFO cfg.GT_AUG_RAND_NUM: False
  2021-01-28 14:34:19,418 INFO cfg.GT_AUG_APPLY_PROB: 1.0
  2021-01-28 14:34:19,418 INFO cfg.GT_AUG_HARD_RATIO: 0.6
  2021-01-28 14:34:19,418 INFO cfg.PC_REDUCE_BY_RANGE: True
  2021-01-28 14:34:19,419 INFO cfg.PC_AREA_SCOPE: [[-40. 40.]
  [ -3. 3.]
  [-60. 60.]]
  2021-01-28 14:34:19,419 INFO cfg.CLS_MEAN_SIZE: [[1.5256319 1.6285675 3.8831165]]
  2021-01-28 14:34:19,419 INFO
  cfg.RPN = edict()
  2021-01-28 14:34:19,419 INFO cfg.RPN.ENABLED: True
  2021-01-28 14:34:19,419 INFO cfg.RPN.FIXED: True
  2021-01-28 14:34:19,419 INFO cfg.RPN.USE_INTENSITY: False
  2021-01-28 14:34:19,419 INFO cfg.RPN.LOC_XZ_FINE: True
  2021-01-28 14:34:19,419 INFO cfg.RPN.LOC_SCOPE: 3.0
  2021-01-28 14:34:19,419 INFO cfg.RPN.LOC_BIN_SIZE: 0.5
  2021-01-28 14:34:19,419 INFO cfg.RPN.NUM_HEAD_BIN: 12
  2021-01-28 14:34:19,419 INFO cfg.RPN.BACKBONE: pointnet2_msg
  2021-01-28 14:34:19,419 INFO cfg.RPN.USE_BN: True
  2021-01-28 14:34:19,419 INFO cfg.RPN.NUM_POINTS: 65536
  2021-01-28 14:34:19,419 INFO
  cfg.RPN.SA_CONFIG = edict()
  2021-01-28 14:34:19,419 INFO cfg.RPN.SA_CONFIG.NPOINTS: [4096, 1024, 256, 64]
  2021-01-28 14:34:19,419 INFO cfg.RPN.SA_CONFIG.RADIUS: [[0.1, 0.5], [0.5, 1.0], [1.0, 2.0], [2.0, 4.0]]
  2021-01-28 14:34:19,419 INFO cfg.RPN.SA_CONFIG.NSAMPLE: [[16, 32], [16, 32], [16, 32], [16, 32]]
  2021-01-28 14:34:19,420 INFO cfg.RPN.SA_CONFIG.MLPS: [[[16, 16, 32], [32, 32, 64]], [[64, 64, 128], [64, 96, 128]], [[128, 196, 256], [128, 196, 256]], [[256, 256, 512], [256, 384, 512]]]
  2021-01-28 14:34:19,420 INFO cfg.RPN.FP_MLPS: [[128, 128], [256, 256], [512, 512], [512, 512]]
  2021-01-28 14:34:19,420 INFO cfg.RPN.CLS_FC: [128]
  2021-01-28 14:34:19,420 INFO cfg.RPN.REG_FC: [128]
  2021-01-28 14:34:19,420 INFO cfg.RPN.DP_RATIO: 0.5
  2021-01-28 14:34:19,420 INFO cfg.RPN.LOSS_CLS: SigmoidFocalLoss
  2021-01-28 14:34:19,420 INFO cfg.RPN.FG_WEIGHT: 15
  2021-01-28 14:34:19,420 INFO cfg.RPN.FOCAL_ALPHA: [0.25, 0.75]
  2021-01-28 14:34:19,420 INFO cfg.RPN.FOCAL_GAMMA: 2.0
  2021-01-28 14:34:19,420 INFO cfg.RPN.REG_LOSS_WEIGHT: [1.0, 1.0, 1.0, 1.0]
  2021-01-28 14:34:19,420 INFO cfg.RPN.LOSS_WEIGHT: [1.0, 1.0]
  2021-01-28 14:34:19,420 INFO cfg.RPN.NMS_TYPE: normal
  2021-01-28 14:34:19,420 INFO cfg.RPN.SCORE_THRESH: 0.3
  2021-01-28 14:34:19,420 INFO
  cfg.RCNN = edict()
  2021-01-28 14:34:19,420 INFO cfg.RCNN.ENABLED: True
  2021-01-28 14:34:19,420 INFO cfg.RCNN.USE_RPN_FEATURES: True
  2021-01-28 14:34:19,420 INFO cfg.RCNN.USE_MASK: True
  2021-01-28 14:34:19,420 INFO cfg.RCNN.MASK_TYPE: seg
  2021-01-28 14:34:19,420 INFO cfg.RCNN.USE_INTENSITY: False
  2021-01-28 14:34:19,420 INFO cfg.RCNN.USE_DEPTH: True
  2021-01-28 14:34:19,420 INFO cfg.RCNN.USE_SEG_SCORE: False
  2021-01-28 14:34:19,420 INFO cfg.RCNN.ROI_SAMPLE_JIT: True
  2021-01-28 14:34:19,420 INFO cfg.RCNN.ROI_FG_AUG_TIMES: 10
  2021-01-28 14:34:19,420 INFO cfg.RCNN.REG_AUG_METHOD: multiple
  2021-01-28 14:34:19,420 INFO cfg.RCNN.POOL_EXTRA_WIDTH: 1.0
  2021-01-28 14:34:19,420 INFO cfg.RCNN.LOC_SCOPE: 1.5
  2021-01-28 14:34:19,420 INFO cfg.RCNN.LOC_BIN_SIZE: 0.5
  2021-01-28 14:34:19,420 INFO cfg.RCNN.NUM_HEAD_BIN: 9
  2021-01-28 14:34:19,421 INFO cfg.RCNN.LOC_Y_BY_BIN: False
  2021-01-28 14:34:19,421 INFO cfg.RCNN.LOC_Y_SCOPE: 0.5
  2021-01-28 14:34:19,421 INFO cfg.RCNN.LOC_Y_BIN_SIZE: 0.25
  2021-01-28 14:34:19,421 INFO cfg.RCNN.SIZE_RES_ON_ROI: False
  2021-01-28 14:34:19,421 INFO cfg.RCNN.USE_BN: False
  2021-01-28 14:34:19,421 INFO cfg.RCNN.DP_RATIO: 0.0
  2021-01-28 14:34:19,421 INFO cfg.RCNN.BACKBONE: pointnet
  2021-01-28 14:34:19,421 INFO cfg.RCNN.XYZ_UP_LAYER: [128, 128]
  2021-01-28 14:34:19,421 INFO cfg.RCNN.NUM_POINTS: 512
  2021-01-28 14:34:19,421 INFO
  cfg.RCNN.SA_CONFIG = edict()
  2021-01-28 14:34:19,421 INFO cfg.RCNN.SA_CONFIG.NPOINTS: [128, 32, -1]
  2021-01-28 14:34:19,421 INFO cfg.RCNN.SA_CONFIG.RADIUS: [0.2, 0.4, 100]
  2021-01-28 14:34:19,421 INFO cfg.RCNN.SA_CONFIG.NSAMPLE: [64, 64, 64]
  2021-01-28 14:34:19,421 INFO cfg.RCNN.SA_CONFIG.MLPS: [[128, 128, 128], [128, 128, 256], [256, 256, 512]]
  2021-01-28 14:34:19,421 INFO cfg.RCNN.CLS_FC: [256, 256]
  2021-01-28 14:34:19,421 INFO cfg.RCNN.REG_FC: [256, 256]
  2021-01-28 14:34:19,421 INFO cfg.RCNN.LOSS_CLS: BinaryCrossEntropy
  2021-01-28 14:34:19,421 INFO cfg.RCNN.FOCAL_ALPHA: [0.25, 0.75]
  2021-01-28 14:34:19,421 INFO cfg.RCNN.FOCAL_GAMMA: 2.0
  2021-01-28 14:34:19,421 INFO cfg.RCNN.CLS_WEIGHT: [1. 1. 1.]
  2021-01-28 14:34:19,422 INFO cfg.RCNN.CLS_FG_THRESH: 0.6
  2021-01-28 14:34:19,422 INFO cfg.RCNN.CLS_BG_THRESH: 0.45
  2021-01-28 14:34:19,422 INFO cfg.RCNN.CLS_BG_THRESH_LO: 0.05
  2021-01-28 14:34:19,422 INFO cfg.RCNN.REG_FG_THRESH: 0.55
  2021-01-28 14:34:19,422 INFO cfg.RCNN.FG_RATIO: 0.5
  2021-01-28 14:34:19,422 INFO cfg.RCNN.ROI_PER_IMAGE: 64
  2021-01-28 14:34:19,422 INFO cfg.RCNN.HARD_BG_RATIO: 0.8
  2021-01-28 14:34:19,422 INFO cfg.RCNN.SCORE_THRESH: 0.3
  2021-01-28 14:34:19,422 INFO cfg.RCNN.NMS_THRESH: 0.1
  2021-01-28 14:34:19,422 INFO
  cfg.TRAIN = edict()
  2021-01-28 14:34:19,422 INFO cfg.TRAIN.SPLIT: train
  2021-01-28 14:34:19,422 INFO cfg.TRAIN.VAL_SPLIT: sample
  2021-01-28 14:34:19,422 INFO cfg.TRAIN.LR: 0.002
  2021-01-28 14:34:19,422 INFO cfg.TRAIN.LR_CLIP: 1e-05
  2021-01-28 14:34:19,422 INFO cfg.TRAIN.LR_DECAY: 0.5
  2021-01-28 14:34:19,422 INFO cfg.TRAIN.DECAY_STEP_LIST: [100, 150, 180, 200]
  2021-01-28 14:34:19,422 INFO cfg.TRAIN.LR_WARMUP: True
  2021-01-28 14:34:19,422 INFO cfg.TRAIN.WARMUP_MIN: 0.0002
  2021-01-28 14:34:19,422 INFO cfg.TRAIN.WARMUP_EPOCH: 1
  2021-01-28 14:34:19,422 INFO cfg.TRAIN.BN_MOMENTUM: 0.1
  2021-01-28 14:34:19,422 INFO cfg.TRAIN.BN_DECAY: 0.5
  2021-01-28 14:34:19,422 INFO cfg.TRAIN.BNM_CLIP: 0.01
  2021-01-28 14:34:19,422 INFO cfg.TRAIN.BN_DECAY_STEP_LIST: [1000]
  2021-01-28 14:34:19,422 INFO cfg.TRAIN.OPTIMIZER: adam_onecycle
  2021-01-28 14:34:19,422 INFO cfg.TRAIN.WEIGHT_DECAY: 0.001
  2021-01-28 14:34:19,422 INFO cfg.TRAIN.MOMENTUM: 0.9
  2021-01-28 14:34:19,422 INFO cfg.TRAIN.MOMS: [0.95, 0.85]
  2021-01-28 14:34:19,423 INFO cfg.TRAIN.DIV_FACTOR: 10.0
  2021-01-28 14:34:19,423 INFO cfg.TRAIN.PCT_START: 0.4
  2021-01-28 14:34:19,423 INFO cfg.TRAIN.GRAD_NORM_CLIP: 1.0
  2021-01-28 14:34:19,423 INFO cfg.TRAIN.RPN_PRE_NMS_TOP_N: 9000
  2021-01-28 14:34:19,423 INFO cfg.TRAIN.RPN_POST_NMS_TOP_N: 512
  2021-01-28 14:34:19,423 INFO cfg.TRAIN.RPN_NMS_THRESH: 0.75
  2021-01-28 14:34:19,423 INFO cfg.TRAIN.RPN_DISTANCE_BASED_PROPOSE: True
  2021-01-28 14:34:19,423 INFO
  cfg.TEST = edict()
  2021-01-28 14:34:19,423 INFO cfg.TEST.SPLIT: sample
  2021-01-28 14:34:19,423 INFO cfg.TEST.RPN_PRE_NMS_TOP_N: 9000
  2021-01-28 14:34:19,423 INFO cfg.TEST.RPN_POST_NMS_TOP_N: 100
  2021-01-28 14:34:19,423 INFO cfg.TEST.RPN_NMS_THRESH: 0.7
  2021-01-28 14:34:19,423 INFO cfg.TEST.RPN_DISTANCE_BASED_PROPOSE: True
  THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1565272279342/work/aten/src/THC/THCGeneral.cpp line=50 error=30 : unknown error
  Traceback (most recent call last):
  File "eval_rcnn.py", line 568, in
  eval_single_ckpt(root_result_dir)
  File "eval_rcnn.py", line 507, in eval_single_ckpt
  model = PointRCNN(num_classes=2, use_xyz=True, mode='TEST')
  File "/home/teroot/Uhnder_OGM/Annotator/smart-annotation-pointrcnn/app/PointCNN/tools/../lib/net/point_rcnn.py", line 15, in init
  self.rpn = RPN(use_xyz=use_xyz, mode=mode)
  File "/home/teroot/Uhnder_OGM/Annotator/smart-annotation-pointrcnn/app/PointCNN/tools/../lib/net/rpn.py", line 64, in init
  self.proposal_layer = ProposalLayer(mode=mode)
  File "/home/teroot/Uhnder_OGM/Annotator/smart-annotation-pointrcnn/app/PointCNN/tools/../lib/rpn/proposal_layer.py", line 13, in init
  self.MEAN_SIZE = torch.from_numpy(cfg.CLS_MEAN_SIZE[0]).cuda()
  File "/home/teroot/miniconda3/envs/smartannotation/lib/python3.6/site-packages/torch/cuda/init.py", line 179, in _lazy_init
  torch._C._cuda_init()
  RuntimeError: cuda runtime error (30) : unknown error at /opt/conda/conda-bld/pytorch_1565272279342/work/aten/src/THC/THCGeneral.cpp:50
  000 finished................ enjoy!
  ==============================================
  ==============================================
  127.0.0.1 - - [28/Jan/2021 14:34:19] "POST /getFramePointCloud HTTP/1.1" 500 -
  Traceback (most recent call last):
  File "/home/teroot/miniconda3/envs/smartannotation/lib/python3.6/site-packages/flask/app.py", line 2464, in call
  return self.wsgi_app(environ, start_response)
  File "/home/teroot/miniconda3/envs/smartannotation/lib/python3.6/site-packages/flask/app.py", line 2450, in wsgi_app
  response = self.handle_exception(e)
  File "/home/teroot/miniconda3/envs/smartannotation/lib/python3.6/site-packages/flask/app.py", line 1867, in handle_exception
  reraise(exc_type, exc_value, tb)
  File "/home/teroot/miniconda3/envs/smartannotation/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
  raise value
  File "/home/teroot/miniconda3/envs/smartannotation/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
  response = self.full_dispatch_request()
  File "/home/teroot/miniconda3/envs/smartannotation/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
  rv = self.handle_user_exception(e)
  File "/home/teroot/miniconda3/envs/smartannotation/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
  reraise(exc_type, exc_value, tb)
  File "/home/teroot/miniconda3/envs/smartannotation/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
  raise value
  File "/home/teroot/miniconda3/envs/smartannotation/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
  rv = self.dispatch_request()
  File "/home/teroot/miniconda3/envs/smartannotation/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
  return self.view_functionsrule.endpoint
  File "/home/teroot/Uhnder_OGM/Annotator/smart-annotation-pointrcnn/app/app.py", line 86, in getFramePointCloud
  data_str = fh.get_pointcloud(drivename, fname, dtype=str)
  File "/home/teroot/Uhnder_OGM/Annotator/smart-annotation-pointrcnn/app/frame_handler.py", line 68, in get_pointcloud
  assert isfile(seg_file)
  AssertionError