sekunde / 3d-sis Goto Github PK

thanks for your great work!
I want to train the model on scannetv2, but I don't see any command in readme.md used for training. what should I do for training?

Greetings!

Your example.sh doesn't seem to be working. Some issues I've found:

• Line 25: you call example_result/test but it does not exist
• Line 25: you call example_result/benchmark_result but it does not exist
• Line 33: ERROR: No result files found.

can you please tell me where to download suncg dataset and how to get its instance and semantic lable for every point? thanks

I want to download , ScanNet Training Chunks(70gb) which you write in your readme.
But I click on it and nothing happens,so if i want to download ScanNet ,where to find it

oh,luckliy,i put the download url in my web， I can see the zip file, but it no speed,
so, if i want to use it dataset, i will go to download scannet dataset, however it mayby 1 t , so if i just download few of the dataset, can i use it to train directly or i should do some pre work

Thank you for the wonderful job!!. I got the output by running example demo i can see only segmented objects, but How to visualize bounding boxes on segmented objects in the output .ply file ??. Could you please clarify whether we will get 3D bounding boxes or not.

Hi，Prof.Hou，I have a question about the 3DMV train dataset.I used the 3DMV dataset to train 3DMV model，but I found that there were too many invalid samples in this dataset.more specifically，many 3D chunks have no mapping relationship with 2d images，and 3Dmv cannot be trained.

Hello author, when I run example.sh, the system will prompt that there is no example_result/test&benchmark_result folder. I don't think the code has created these two folders. What's the matter? Thank you！

Hi, In 3D-SIS, inappropriate dependency versioning constraints can cause risks.

Below are the dependencies and version constraints that the project is using

torch==0.4.1
torchnet==0.0.4
torchvision==0.2.1
easydict==1.9
tqdm==4.29.0
reprint==0.5.2
plyfile==0.7
tensorflow==1.4.0
pyyaml==3.13
ipdb==0.11
matplotlib==3.0.2
h5py==2.9.0
scikit-image==0.14.2
cffi==1.11.5
scipy==1.1.0

The version constraint == will introduce the risk of dependency conflicts because the scope of dependencies is too strict.
The version constraint No Upper Bound and * will introduce the risk of the missing API Error because the latest version of the dependencies may remove some APIs.

After further analysis, in this project,
The version constraint of dependency tqdm can be changed to >=4.36.0,<=4.64.0.
The version constraint of dependency ipdb can be changed to >=0.7,<=0.13.9.
The version constraint of dependency matplotlib can be changed to >=1.3.0,<=3.0.3.

The above modification suggestions can reduce the dependency conflicts as much as possible,
and introduce the latest version as much as possible without calling Error in the projects.

The invocation of the current project includes all the following methods.

tqdm.tqdm

ipdb.set_trace

matplotlib.use

dict.keys
shift_z.ravel.shift_y.ravel.shift_x.ravel.shift_z.ravel.shift_y.ravel.shift_x.ravel.np.vstack.transpose
os.path.abspath
torch.LongTensor.append
labels_level2.reshape.fill
rpn_bbox_inside_weights_level2.float
keep_inds.append
self._anchors_level2.cuda
i.self._gt_bbox.size
rpn_bbox_targets_level1.float.cuda
self._proposal_layer
_sample_rois
numpy.convolve
torch.Tensor
i.gt_bbox.anchors_level1.bbox_overlap.numpy.argmax
numpy.reshape
math.cos
StatefulMaxPool2d
torch.autograd.Variable
lib.layer_utils.proposal_target_layer.proposal_target_layer
rpn_bbox_outsise_weights_level2.float.cuda
valid_ind_mask.p.depth_vals.torch.abs.le.any
_ext.roi_pooling.roi_pooling_backward_cuda
redfile.replace
bbox_outside_weights.cuda
data.Variable.cuda
torch.log
dets.numpy
torch.mm
pre_nms_topN.scores.view
torch.set_num_threads
camera_to_world.repeat
DetectionMAP.compute_IoU
rpn_bbox_pred_level2.size.rpn_bbox_pred_level2.size.rpn_bbox_pred_level2.size.cfg.NUM_ANCHORS_LEVEL2.self.batch_size.rpn_level2.self.rpn_cls_score_net_level2.view.permute
box_stats.append
numpy.asarray
lib.utils.config.cfg.MASK_FG_THRESH.max_overlaps.nonzero
bbox_inside_weights_batch_level2.append
torch.lt
mask_target.float.cuda
features.size
cls.mask_ind.net._predictions.data.cpu
make_voxel_mesh
roi.item
zip
lib.utils.timer.Timer.tic
self.StatefulMaxPool2d.super.forward
numpy.min
struct.unpack
bbox_outside_weights_batch_level2.append
max
mask_target.float
rpn_bbox_pred_level1.size.rpn_bbox_pred_level1.size.rpn_bbox_pred_level1.size.cfg.NUM_ANCHORS_LEVEL1.self.batch_size.rpn_level1.self.rpn_cls_score_net_level1.view.permute
rpn_bbox_pred_level3.size.rpn_bbox_pred_level3.size.rpn_bbox_pred_level3.size.cfg.NUM_ANCHORS_LEVEL3.self.batch_size.rpn_level3.self.rpn_cls_score_net_level3.view.permute
numpy.arange.ravel
bbox_inside_weights_batch_level3.append
labels_batch.append
torch.nn.functional.binary_cross_entropy
pool_size.pool_size.pool_size.bottom1.shape.rois.shape.torch.zeros.Variable.cuda
torch.LongTensor
lib.model.trainval.train
x.size
lib.layer_utils.mask_target_layer.mask_target_layer
a.items
numpy.copy
self.fix_eval_parts
blobs.split
read_segmentation
num_ft.grad_label.data.view.index_copy_
rpn_label_level1.data.nonzero.numel
keep_inds.levelInds.contiguous
numpy.ceil
labels_level2.reshape.reshape
Lambda
i.rpn_bbox_pred_level3.view
lib.utils.config.cfg.MASK_FG_THRESH.max_overlaps.nonzero.view
grad_label.data.view
outputs.append
self._imageft.append
list.pop
load_matrix
self.writer.add_summary
compute_averages
open.write
os.path.join
numpy.maximum
bbox_outside_weights_batch_level3.append
i.self._gt_bbox.cuda
levelInds.cuda
torch.utils.ffi._wrap_function
rpn_bbox_outsise_weights_level3.float
lib.utils.bbox_transform.bbox_transform_inv
redfile.replace.append
torch.nn.functional.cross_entropy
self.resize_crop_image
self._smooth_l1_loss
grad_output.clone
self.restore
i.gt_bbox.anchors_level2.bbox_overlap.numpy
self._proposal_targets.view
_merge_a_into_b
num_label_ft.output.view.index_copy_
lib.utils.config.cfg.BG_THRESH.max_overlaps.nonzero.view
lib.datasets.dataloader.get_dataloader
argmax.cuda.cuda
self._region_classification
numpy.array
sort_index.append
self.volume_dims.voxel_bounds_max.np.minimum.cpu.float
write_ply
inspect.currentframe
t.cuda
l.__str__.startswith
print
i.rpn_cls_prob_level1.view
rpn_bbox_outsise_weights_level1.float.cuda
rpn_bbox_outsise_weights_level3.float.cuda
int.keys
importlib.import_module
data_logger.scalar_summary
bbox_outside_weights_batch.append
os.makedirs
grad_label.data.resize_
Autoencoder.cuda.train
export_ids
torch.cuda.synchronize
images.append
opt.filelist.open.readlines
label_name.pred2gt.append
world2grid.append
time.time
bbox_outside_weights_level3.reshape.reshape
rpn_label_level3.data.nonzero.numel
seg_id.seg_to_verts.append
int
Timer
numpy.expand_dims
random_shift.torch.float32.self.step.self.step.grid_dimX.torch.arange.unsqueeze.unsqueeze.k.expand
torch.ones_like
dir
ctr_x.unsqueeze
bbox_targets_level3.reshape.reshape
torch.float32.grid_dimY.torch.arange.unsqueeze.unsqueeze
bbox_inside_weights_level2.reshape.reshape
Bottleneck
torch.abs
label.view
A.K.anchors_level3.reshape.astype
torch.cat.cpu
rois.is_cuda
typeName.lower
volume_dims.volume_dims.volume_dims.num_label_ft.label.new.fill_.view
txt_file.benchmark_path.os.path.join.open.readlines
lib.datasets.dataset.Dataset.load_mapping
DetectionMAP.jaccard_mask
isinstance
read_aggregation
self.get_instance_verts
lin_ind_volume.cuda.cuda
p0.p1.astype
num_out.keep.cuda.order.contiguous
lib.utils.config.cfg.NORMALIZE_WEIGHTS.append
lib.utils.timer.Timer.toc
Autoencoder.cuda.eval
rpn_labels_level2.long.cuda
mean_average_precision.append
util.print_error
numpy.fromstring
self.train
self.pooled_length.self.pooled_height.self.pooled_width.num_channels.num_rois.torch.IntTensor.zero_
rpn_level3.self.rpn_bbox_pred_net_level3.permute.contiguous.size
self.rpn_cls_score_net_level1
numpy.clip
depth.view
K.shifts.reshape.transpose
self.rpn_net_level3
self.encode
coords.resize_.fill_
torch.zeros
numpy.empty.fill
self.load_depth
dets.size.np.arange.torch.from_numpy.long
numpy.loadtxt
numpy.modf
DetectionMAP.intersect_area_mask
widths.unsqueeze
os.path.basename
torch.stack.clamp
scores_combined_list.append
lib.layer_utils.generate_anchors.generate_anchors
self.np_paths.remove
self.net._scene_info.pred_box.clip_boxes.numpy
_SPLITTER.join
self.net._losses.item
input.data.new
sorted
torch.float32.self.step.self.step.grid_dimY.torch.arange.unsqueeze.unsqueeze
lib.utils.config.cfg.NET.net_module.getattr.init_modules
self._anchor_component
SensorData.SensorData.export_depth_images
keep_inds.labels.contiguous
StringIO.StringIO
self._mask_target_layer
torch.float32.grid_dimX.torch.arange.unsqueeze.unsqueeze.expand
pred_instance.copy
numpy.float32.f.read.struct.unpack.np.asarray.reshape
pprint.pprint
DetectionMAP.jaccard
csv.DictReader
layer.__str__
numpy.copy.astype
loss_box.mean.sum
self.Dataset.super.__init__
lib.layer_utils.nms.pth_nms.cpu_nms
self._predictions.data.cpu
img_summaries.append
lib.utils.config.cfg.NORMALIZE_WEIGHTS.torch.FloatTensor.cuda
mask_pred_batch.append
tensorflow.HistogramProto
lib.utils.config.cfg.COLOR_STD.cfg.COLOR_MEAN.transforms.Normalize
grad_output.data.contiguous.view
RGBDFrame.load
torch.zeros.to
_ext.roi_pooling.roi_pooling_forward
torch.no_grad
rpn_label_level2.data.nonzero
rpn_bbox_inside_weights_level3.float
lib.datasets.BinaryReader.BinaryReader.read
labels.append
bbox_inside_weights_level3.reshape.reshape
self.log_print
open
DetectionMAP.intersect_area
self.__normalize__
num_out.keep.cuda
self.rpn_cls_score_net_level3
txt_file.self.res_path.os.path.join.open.readlines.keys
lib.utils.config.cfg.BG_THRESH_LO.max_overlaps.cfg.BG_THRESH.max_overlaps.nonzero
sys.path.insert
self.net.state_dict
shift_z.ravel.shift_y.ravel.shift_x.ravel.shift_z.ravel.shift_y.ravel.shift_x.ravel.np.vstack.transpose.reshape
levelInd.cpu
self.net.init_modules
f.self.frames.decompress_color
json.dumps
rpn_labels_level3.long
num_bg.bg_inds.numel.num_bg.int.bg_inds.numel.np.arange.np.random.choice.torch.from_numpy.long
self.find_previous
torch.float32.grid_dimZ.torch.arange.unsqueeze
SolverWrapper
lib.utils.timer.Timer
fg_inds.cls.contiguous.view.numel
layer.__str__.startswith
net.mask_backbone.eval
represents_int
torch.optim.RMSprop
net_conv_level3.size
bbox_inside_weights.cuda
self.combine
x.split
recalls.append
grid_shape.ind2d.ind3d.ft.Projection.apply.contiguous
rpn_bbox_inside_weights_level1.float
bbox_targets_batch_level3.append
net.mask_backbone.cuda
math.floor
self._proposal_target_layer
lib.utils.evaluation.DetectionMAP
numpy.arange
scene_info.torch.ByteTensor.zero_.cuda
self.outbbox_thresh
sys.stderr.write
open.readlines
type
plyfile.PlyData.read
rpn_bbox_targets_level2.float
label_mapping.iteritems
output.append
self._anchor_target_layer
torchvision.transforms.CenterCrop
model
labels.torch.Tensor.long
gt_num.label_name.gt2pred.append
coords.torch.unsqueeze.repeat.to
torch.remainder
numpy.float32.file_handle.read.struct.unpack.np.asarray.reshape
self.Base_Backbone.super.__init__
bbox_targets_batch_level1.append
RGBDFrame
folder.split
lib.utils.config.cfg.BG_THRESH_LO.max_overlaps.cfg.BG_THRESH.max_overlaps.nonzero.view
int.items
torch.cat.size
rois.size
numpy.prod
float
input.dim
torch.nn.Conv3d
random_shift.torch.float32.self.step.self.step.grid_dimY.torch.arange.unsqueeze.unsqueeze.k.expand
f.endswith
torch.arange
voxel_bounds_min.np.maximum.cpu.float
scores_batch.append
query_boxes.t.boxes.torch.max.query_boxes.t.boxes.torch.min.clamp
self.net._predictions.data.cpu
sys.stdout.flush
txt_file.self.res_path.os.path.join.open.readlines
self.StatefulMaxPool2d.super.__init__
torch.float32.grid_dimX.torch.arange.unsqueeze.unsqueeze
numpy.ones
self._anchors_level3.cuda
list.append
bbox_outside_weights_level2.reshape.reshape
i.rpn_bbox_pred_level2.view
create_color_palette
copy.deepcopy
rpn_level2.self.rpn_cls_score_net_level2.view
voxel_bounds_min.np.maximum.cuda
numpy.floor
assign_instances_for_scan
cls.nonzero.view.numel
pickle.dump
AddCoordinates3d
lend.lstart.hend.hstart.wend.wstart.c.data.numpy.argmax
Dataset.load_mapping
tensorflow.Summary.Value
lib.utils.config.cfg.NET.net_module.getattr
keep_inds.labels.contiguous.long
self.classifier_cls_score_net
A.K.anchors_level1.reshape.astype.reshape
self.net.state_dict.update
A.get_device
self.__get_outputs
DetectionMAP.compute_TP_FP_FN
input.size
rpn_bbox_inside_weights_level2.float.cuda
KeyError
torch.optim.SGD
blobs.append
lib.utils.config.cfg.CLASS_BATCH_SIZE.fg_inds.numel.num_fg.int.fg_inds.numel.np.arange.np.random.choice.torch.from_numpy.long
numpy.append
tensorflow.Summary.Image
torch.lt.any
line.split.split
_ext.roi_pooling.roi_pooling_forward_cuda
numpy.array.append
callable
self.load_image
f.self.frames.decompress_depth
coords.torch.unsqueeze.repeat
self.CoordConvNet3d.super.__init__
torch.from_numpy
self.volume_dims.self.volume_dims.coords.long
torch.nn.BatchNorm2d
method_to_call
j.i.gt_mask.float
self.ss_paths.remove
rpn_bbox_inside_weights_level3.float.cuda
depths.append
labels.cuda
self.classifier_bbox_pred_net
self.unmold_mask
rois.append
A.K.anchors_level2.reshape.astype.reshape
lib.utils.config.cfg.CLASS_BATCH_SIZE.bg_inds.numel.num_bg.int.bg_inds.numel.np.arange.np.random.choice.torch.from_numpy.long
torch.utils.ffi.create_extension
weights.append
net._scene_info.pred_box.clip_boxes.numpy
torch.nn.Sigmoid
lib.datasets.dataset.Dataset
instance.split
file_handle.read
self.rpn_bbox_pred_net_level2
io.BytesIO.getvalue
labels_level1.reshape.reshape
join
self._initialize_weights
roi.cpu
torch.nn.Linear
torch.float32.self.step.self.step.grid_dimX.torch.arange.unsqueeze
rpn_label_level2.data.nonzero.numel
labels_level1.reshape.fill
self.net._predictions.data.cpu.numpy
self.Padding.super.__init__
self.rpn_bbox_pred_net_level3
abs
self.image_enet_fixed
PIL.Image.NEAREST.resize_width.new_image_dims.transforms.Resize
numpy.where
torch.ceil
self.classifier
bbox_targets.shape.cls.new.zero_.float
self.color
num_images.torch.nn.MaxPool1d
grid_shape.ind2d.ind3d.ft.Projection.apply.contiguous.view
sum
self.total_accumulators.append
self.LambdaBase.super.__init__
rpn_level2.self.rpn_bbox_pred_net_level2.permute.contiguous
gt_box_list.append
rotation
Autoencoder
idx.self.scenes.os.path.basename.rsplit
tensorflow.summary.FileWriter
torch.nn.MaxPool3d
query_ares.view
rpn_labels_level3.long.cuda
numpy.concatenate.t
torch.pow
numpy.not_equal
depth_image.astype.astype
self.mask_backbone
self.decompress_color_jpeg
self.Network.super.__init__
numpy.log
dets.size
labels_level3.reshape.fill
self.image_enet_fixed.eval
torch.FloatTensor
lib.utils.config.cfg.NET.net_module.getattr.forward
volume_dims.volume_dims.volume_dims.num_label_ft.label.new.fill_
collections.deque
torch.nn.ConvTranspose3d
lib.utils.config.cfg.BASE_IMAGE_PATH.endswith
rpn_bbox_targets_level2.float.cuda
fg_inds.cls.contiguous.view
self.conv2
B.A.torch.mm.t
dict
ScanNet
range
gt_mask_list.append
evaluate
rpn_level1.self.rpn_bbox_pred_net_level1.permute.contiguous
APAccumulator
data_length.data_height.data_width.num_channels.batch_size.torch.zeros.cuda
numpy.empty
precisions.append
torch.floor
self._predictions.cpu.cuda
create_enet
torch.exp
module
torch.float32.self.step.self.step.grid_dimZ.torch.arange.unsqueeze.unsqueeze
lib.utils.config.cfg.NET.net_module.getattr.state_dict
batch_size.i.batch_size.i.levelInds.nonzero
getattr
self.conv_tr_layer
self.validation
self.writer.flush
locals
self.RoIPoolFunction.super.__init__
self.images.append
torch.max
sys.exit
lib.model.trainval.benchmark
rpn_level3.self.rpn_cls_score_net_level3.view
create_cylinder_mesh
pred_class.astype.astype
self.volume_dims.voxel_bounds_max.np.minimum.cuda.float
self.rpn_bbox_pred_net_level1
img.scipy.misc.toimage.save
self.net.load_state_dict
super
torch.zeros_like
lib.model.trainval.test
ValueError
self.geometry
keep_inds.scores.contiguous
scipy.misc.imread
anchor.strip.split
utils.read_mesh_vertices
self.relu
utils.get_instances
list
voxel_bounds_min.np.maximum.cuda.float
yaml.dump
proj3d.cpu.cpu
Benchmark_reader
valid_ind_mask.p.depth_vals.torch.abs.le
torch.nonzero
enumerate
layer
self.Dropout2d.super.forward
os.path.splitext
torch.nn.Sequential
c.cuda
rpn_bbox_inside_weights_level1.float.cuda
rpn_label_level3.data.nonzero
self._backbone
self.coord_adder
self.net.forward
get_bbox_verts
numpy.matmul
numpy.average
imageio.imread
skimage.transform.resize
filename.open.read
imageft.view.cpu
lib.utils.overlap.bbox_overlap.max
rpn_level1.self.rpn_bbox_pred_net_level1.permute
self._gt_bbox.long
bbox_targets_batch_level2.append
mask_pred.append
numpy.eye
self.mapping.items
rpn_labels_level1.long
torch.float32.grid_dimY.torch.arange.unsqueeze.unsqueeze.expand
pool5.size
lib.utils.config.cfg.FG_THRESH.max_overlaps.nonzero
fg_inds.cls.contiguous.view.new
Autoencoder.cuda.state_dict
yaml.load
self.save_mat_to_file
cls.mask_ind.net._predictions.data.cpu.numpy
inst.label_id.id2label.instances.append
instance_key.instances.nonzero
torchvision.transforms.ToTensor
compute_length_vec3
self.RoIPool.super.__init__
self.net.delete_intermediate_states
gt_class.astype.astype
rpn_labels_level1.long.cuda
A.K.anchors_level3.reshape.astype.reshape
rpn_level3.self.rpn_bbox_pred_net_level3.permute
gt_classes.astype.astype
labels_level3.reshape.reshape
self._region_proposal
rpn_bbox_targets_level3.float.cuda
net._predictions.data.cpu
json.load
gt_files.append
alignment.split.a_.a_.float.np.array.reshape
net_conv_level1.size
LambdaMap
self.conv3
self.optimizer.zero_grad
utils.read_instance_prediction_file
poses.append
dimZ.dimY.dimX.mask_data.np.reshape.astype
self.compute_precision_recall_
hist.bucket_limit.append
numpy.random.randint
argparse.ArgumentParser.parse_args
model.cpu
cls.cls.cls.cls.cls.cls.torch.cat.long
torch.float32.grid_dimX.torch.arange.unsqueeze
_ext.nms.gpu_nms
i.rpn_bbox_pred_level1.view
NotImplementedError
instance_id.mesh_vert_instances.sum
self.compute_frustum_bounds
self.logger_train.scalar_summary
rpn_bbox_targets_level1.float
numpy.abs
self.net.named_parameters
i.gt_bbox.anchors_level2.bbox_overlap.numpy.argmax
numpy.round
numpy.expand_dims.astype
numpy.oneos_like
map_label_image
lin_ind_volume.cuda.size
acc.if_ignore
idx.self.scenes.os.path.basename.split
self.ss_paths.append
open.close
x.torch.from_numpy.expand
i.gt_bbox.anchors_level1.bbox_overlap.numpy
roi.cpu.numpy
os.path.isfile
A.K.anchors_level1.reshape.astype
bbox_targets_level1.reshape.reshape
numpy.random.choice
pad_size.input.data.new.fill_
self.rpn_net_level2
args.cfg.split
x.view
_compute_targets
i.rpn_cls_prob_level2.view
glob.glob.sort
A.K.anchors_level2.reshape.astype
PIL.Image.fromarray
self.net.named_parameters.dict.items
numpy.linspace
net_conv_level2.size
bbox_outside_weights_level1.reshape.reshape
self.eval
all_anchors_level1.size
lib.utils.config.cfg.NUM_CLASSES.cls.numel.cls.new.zero_
self.decompress_depth_zlib
interpolated_precision.append
numpy.sin
bbox_targets_level2.reshape.reshape
Padding
torch.save
os.listdir
torch.nn.MaxPool2d
self.construct_optimizer
lines.np.asarray.astype
bbox_inside_weights_batch_level1.append
torchvision.transforms.Resize
self.__update_weights
self.frames.append
self.forward_prepare
cls.mask_ind.self.net._predictions.data.cpu
camera_to_world.new.fill_
torch.nn.Conv2d
torch.float32.self.step.self.step.grid_dimX.torch.arange.unsqueeze.unsqueeze
sigma_2.abs_in_box_diff.detach
self._predictions.data.cpu.numpy
voxel_bounds_min.np.maximum.cpu
lib.utils.config.cfg_from_file
dimZ.dimY.dimX.data.np.reshape.np.expand_dims.astype
sigma_2.abs_in_box_diff.detach.float
lib.utils.evaluation.DetectionMAP.AP
format
self.__get_model
pre_nms_topN.scores.view.sort
f.read
loss_function
rpn_level3.self.rpn_bbox_pred_net_level3.permute.contiguous
cls.nonzero.view.unsqueeze
torch.nn.functional.sigmoid
torch.nn.MaxPool1d
self.net._predictions.cpu
killing_inds.append
numpy.concatenate
imageft.view.view
numpy.argsort
num_fg.int.fg_inds.numel.np.arange.np.random.choice.torch.from_numpy.long
pred_mask.append
fg_inds.cls.contiguous.view.nonzero
lib.layer_utils.anchor_target_layer.anchor_target_layer
self.lambda_func
self.CoordConvTranspose3d.super.__init__
i.rpn_cls_prob_level3.view
_get_bbox_regression_labels
i.gt_bbox.anchors_level3.bbox_overlap.numpy
lib.utils.config.cfg.BG_THRESH.max_overlaps.nonzero.view.numel
self.image_enet_trainable
self._init_backbone_classifier
self.image_enet_trainable.eval
os.path.exists
self.net._predictions.cpu.numpy
counter.item
torch.inverse
lib.utils.config.cfg.ROI_THRESH.self.net._predictions.nonzero
glob.glob
utils.print_error
i.gt_bbox.cpu
write_bbox
all_anchors_level2.size
world_to_grid.repeat
torch.load.items
keep_inds.rois.contiguous
numpy.histogram
rpn_level1.self.rpn_cls_score_net_level1.view
numpy.savetxt
torch.nn.functional.binary_cross_entropy_with_logits
self.volume_dims.voxel_bounds_max.np.minimum.cpu
create_enet.load_state_dict
numpy.meshgrid
bbox_targets.new.view
lend.lstart.hend.hstart.wend.wstart.c.data.numpy
sys.path.append
numpy.zeros_like
imageio.imwrite
self.conv1
torch.float32.grid_dimZ.torch.arange.unsqueeze.unsqueeze.expand
self.cnn_model.modules
self.reset_accumulators
torch.round
torch.index_select.le
SolverWrapper.benchmark
self._add_losses
verts_len.np.zeros.astype
self.StatefulMaxUnpool2d.super.__init__
Autoencoder.cuda.parameters
len
self.cnn_model._modules.items
image_files.append
k_dimZ.k_dimY.k_dimX.coords_channels.out_channels.torch.zeros.to
lib.utils.config.cfg.NUM_CLASSES.cls.numel.cls.new.zero_.new
rpn_level2.self.rpn_bbox_pred_net_level2.permute.contiguous.size
real_proj_mapping.append
SensorData.SensorData
alignment.split.a_.a_.float.np.array.reshape.split
self._scene_info.pred_box.clip_boxes.numpy
acc.inc_predictions
fg_inds.cls.contiguous
math.fmod
torch.cuda.empty_cache
blobs.cuda
self.decode
i.blobs.cuda
proposals_batch.append
torch.ge
bbox_targets.shape.cls.new.zero_
_import_symbols
self.cuda
nn_search
heights.unsqueeze
torch.float32.self.step.self.step.grid_dimZ.torch.arange.unsqueeze
io.BytesIO
bbox_targets_batch.append
argparse.ArgumentParser.set_defaults
lib.nets.enet.create_enet_for_3d
numpy.linalg.inv
lib.utils.bbox_transform.clip_boxes
utils.export_ids
instance_id.instance_mask.append
self.conv_layer
lib.layer_utils.nms.pth_nms.pth_nms
reprint.output
os.path.dirname
pool5.view
self.deconv1
self.ignore_class.append
cls.mask_ind.self.net._predictions.data.cpu.numpy
out_fn
lib.utils.config.cfg.NET.net_module.getattr.load_state_dict
label_to_segs.items
batch_size.i.batch_size.i.levelInds.nonzero.numel
recon_x.view
torch.index_select
self.get_label_id
numpy.cross
numpy.nanmean
torch.stack
Autoencoder.cuda
SolverWrapper.test
torch.utils.ffi.create_extension.build
torch.nn.Parameter
lib.utils.evaluation.DetectionMAP.mAP
lib.layer_utils.nms_wrapper.nms
torch.ones
masks_batch.append
masks.append
labels_batch_level2.append
i.gt_bbox.anchors_level3.bbox_overlap.numpy.argmax
generate_anchors_for_single_voxel
bbox_inside_weights_batch.append
lib.utils.logger.Logger
matched_gt.append
box_reg.torch.from_numpy.float
lib.datasets.BinaryReader.BinaryReader.close
numpy.vstack
self._imageft.cuda
levelInds_combined_list.append
self.compute_ap
main
lstart.ind_l.hstart.ind_h.wstart.ind_w.np.array.torch.from_numpy.float
query_boxes.t
bbox_targets.new.view.expand_as
lin_ind_volume.cuda.cpu
numpy.uint16.depth_data.np.fromstring.reshape
util.read_label_mapping
lib.utils.overlap.bbox_overlap
numpy.logical_not
self.geometry2
i.gt_bbox.cuda
numpy.transpose
inspect.getfile
load_pred
numpy.arccos
line.split
self.optimizer.step
box_ares.view
lib.utils.config.cfg.ROI_THRESH.self.net._predictions.nonzero.size
self.Bottleneck.super.__init__
self.load
proposals_combined_list.append
i.split
numpy.max
round
LambdaReduce
rpn_bbox_outsise_weights_level1.float
loss_function.backward
lengths.unsqueeze
self.scale_lr
numpy.logical_and
TP.append
keep_inds.labels.contiguous.unsqueeze
pickle.load
map
tqdm.tqdm
random_shift.torch.float32.self.step.self.step.grid_dimZ.torch.arange.unsqueeze.unsqueeze.k.expand
torch.stack.append
numpy.save
new_image_dims.new_image_dims.transforms.CenterCrop
lib.utils.config.cfg.FG_THRESH.max_overlaps.nonzero.view
all_anchors_level3.size
self.np_paths.append
proj2d.cpu.cpu
torch.float32.grid_dimY.torch.arange.unsqueeze
self._anchors_level1.cuda
grad_output.data.contiguous
argparse.ArgumentParser.add_argument
coords.resize_.resize_
lib.datasets.BinaryReader.BinaryReader
Dropout2d
torch.nn.ReLU
torch.nn.MaxUnpool2d
rois_batch.append
get_bbox_edges
roi_score.cpu
torchvision.transforms.Compose
self.initialize
zlib.decompress
numpy.sum
os.mkdir
labels_level_list.append
lib.utils.evaluation.DetectionMAP.evaluate
lib.utils.evaluation.DetectionMAP.finalize
math.sin
numpy.count_nonzero
self.predictions.append
self.load_pose
lib.utils.config.cfg.NET.net_module.getattr.mask_backbone
print_results
self.deconv2
scipy.misc.toimage
torch.nn.PReLU
numpy.dot
math.ceil
bbox_targets.cuda
rpn_bbox_targets_level3.float
l.__str__
net._predictions.cpu
net._predictions.data.cpu.numpy
rpn_bbox_pred_level2.size.rpn_bbox_pred_level2.size.rpn_bbox_pred_level2.size.cfg.NUM_ANCHORS_LEVEL2.self.batch_size.rpn_level2.self.rpn_cls_score_net_level2.view.permute.contiguous
self.img_transform
frameids.append
lib.utils.config.cfg.BG_THRESH.max_overlaps.nonzero
torch.ByteTensor
pred_classes.astype.astype
os.path.commonprefix
numpy.greater
lib.utils.config.cfg.MASK_THRESH.mask.np.where.astype
evaluate_matches
self.snapshot
instance_key.instances.nonzero.astype
ctr_y.unsqueeze
feat_stride.pool_size.pool_size.pool_size.RoIPoolFunction
utils.read_label_mapping
rpn_labels_level2.long
f.write
x.np.stack.astype
easydict.EasyDict
matplotlib.use
fg_inds.unsqueeze.expand
lib.layer_utils.projection.ProjectionHelper
Autoencoder.cuda.load_state_dict
__all__.append
lib.utils.evaluation.DetectionMAP.evaluate_mask
torch.nn.functional.softmax
p.torch.round.long
write_mask
tools.visualization.write_ply
cv2.resize
self.depth_to_skeleton
torch.nn.Sequential.parameters
self.Autoencoder.super.__init__
FP.append
torch.float32.grid_dimZ.torch.arange.unsqueeze.unsqueeze
output.cuda.cuda
os.path.isdir
numpy.minimum
blobs.numpy
dets.size.np.arange.torch.from_numpy.long.cuda
_unmap
math.sqrt
tools.visualization.write_bbox
numpy.zeros
gt_inst.copy
collections.deque.popleft
label.new
filename.open.read.splitlines
writelines.append
imageft.view.permute
open.writelines
self.deconv3
self.remove_snapshot
ctx.save_for_backward
torch.optim.RMSprop.zero_grad
torch.device
hist.bucket.append
tensorflow.Summary
alignment_file.open.readlines
SensorData.SensorData.export_color_images
rpn_bbox_pred_level1.size.rpn_bbox_pred_level1.size.rpn_bbox_pred_level1.size.cfg.NUM_ANCHORS_LEVEL1.self.batch_size.rpn_level1.self.rpn_cls_score_net_level1.view.permute.contiguous
export
numpy.cos
write_result_file
torchvision.transforms.Normalize
lib.utils.timer.Timer.total_time
lib.layer_utils.proposal_layer.proposal_layer
torch.index_select.ge
Instance
torch.zeros.cuda
self._predictions.cpu
dimZ.dimY.dimX.data.np.reshape.astype
numpy.ones_like
object_id_to_segs.items
torch.load
parse_args
Instance.to_dict
self._modules.values
d.cuda
SensorData.SensorData.export_poses
self.unpooling.forward
torch.nn.functional.relu
SolverWrapper.train_model
scene_info.torch.ByteTensor.zero_
anchor.strip
print_error
DetectionMAP.compute_IoU_mask
numpy.stack
self.file.close
self.rpn_cls_score_net_level2
torchvision.utils.save_image
ipdb.set_trace
camera_to_world.new
lib.utils.config.cfg_to_file
torch.cat.append
functools.reduce
acc.precision_recall
ctr_z.unsqueeze
torch.unsqueeze
self.from_snapshot
numpy.in1d
os.path.realpath
label.label_to_segs.extend
cls.nonzero.view
lib.layer_utils.roi_pooling.roi_pool.RoIPoolFunction
cls.nonzero.view.size
self._classifier
gt_mask.append
os.remove
reader.read.np.reshape.np.transpose.astype
torch.float32.self.step.self.step.grid_dimY.torch.arange.unsqueeze
bbox_outside_weights_batch_level1.append
rpn_level2.self.rpn_bbox_pred_net_level2.permute
numpy.cumsum
min
self.file.read
torch.nn.Parameter.size
lib.layer_utils.projection.Projection.apply
struct.calcsize
numpy.isclose
coords_multiplication
i.self._gt_bbox.size.torch.ones.cuda
torch.utils.data.DataLoader
labels_batch_level3.append
utils.load_ids
str
numpy.load
torch.cat
self.volume_dims.voxel_bounds_max.np.minimum.cuda
self._roi_pool_layer
lend.lstart.hend.hstart.wend.wstart.data.torch.max.torch.max.torch.max.view
deltas.detach
self.net._losses.backward
rpn_bbox_pred_level3.size.rpn_bbox_pred_level3.size.rpn_bbox_pred_level3.size.cfg.NUM_ANCHORS_LEVEL3.self.batch_size.rpn_level3.self.rpn_cls_score_net_level3.view.permute.contiguous
collections.deque.append
lin_ind_update.new
self.Dropout.super.forward
self.geometry1
rpn_level1.self.rpn_bbox_pred_net_level1.permute.contiguous.size
torch.nn.ConvTranspose2d
value.numel
sys.stdout.write
rpn_bbox_outsise_weights_level2.float
numpy.unique
self.CoordConv3d.super.__init__
bbox_inside_weights_level1.reshape.reshape
os.path.isabs
argparse.ArgumentParser
numpy.unravel_index
acc.ranking
save_scannet_benchmark
loss_box.mean.mean
torch.optim.RMSprop.step
tools.visualization.read_ply
lib.utils.bbox_transform.bbox_transform
torch.bmm
torch.IntTensor
lib.utils.timer.Timer.average_time
self.rpn_net_level1
torch.min
numpy.array.extend
lib.layer_utils.projection.ProjectionHelper.compute_projection
self.MaskBackbone.super.__init__
torch._utils._rebuild_tensor
levelInds_batch.append
acc.inc_not_predicted
labels_batch_level1.append
lib.utils.timer.Timer.clean_total_time
x.strip
rpn_label_level1.data.nonzero
math.fabs

@developer
Could please help me check this issue?
May I pull a request to fix it?
Thank you very much.

when i run main.py , it occour, but i really put scannetv2_enet.pth put in the directory.
but i dosen't work

thank you for the readMe update.
I have successfully run the code in ubuntu 16.04 and cuda8 !

if I want to use the data taken by my 3d-camera, to predict the results with your model.
how can I do?
I don't know file .scene and world2grid.txt means.
world2grid is camera parameters?
How to generate these files?

Thank you for the great work!

I have a question about the comparative experiments.
As mentioned in the Sec. 7 of the paper, To compare to previous approaches that operate on single RGB or RGB-D frames, we first obtain predictions on each individual frame, and then merge all predictions together in the 3D space of the scene, does it mean that for each frame, we forward the model and get a 3D bounding box prediction? But I cannot find the correspondence annotations between 2d bounding boxes and 3d boxes. I wonder how to train these models without such annotations, namely, for an object in a rgbd frame, how to obtain the groundtruth 3d bounding box? Or you just extract the 2d detection results and project them to 3d space via camera parameters?

Would you mind provide some processing scripts?
Thanks in advance.

Traceback (most recent call last):
File "main.py", line 84, in
benchmark(args)
File "/Users/santhosh/Documents/projects/3D-SIS/lib/model/trainval.py", line 68, in benchmark
net_module = importlib.import_module("lib.nets.backbones")
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 677, in _load_unlocked
File "", line 728, in exec_module
File "", line 219, in _call_with_frames_removed
File "/Users/santhosh/Documents/projects/3D-SIS/lib/nets/backbones.py", line 8, in
from lib.nets.network import Network
File "/Users/santhosh/Documents/projects/3D-SIS/lib/nets/network.py", line 11, in
from lib.layer_utils.proposal_layer import proposal_layer
File "/Users/santhosh/Documents/projects/3D-SIS/lib/layer_utils/proposal_layer.py", line 5, in
from lib.layer_utils.nms_wrapper import nms
File "/Users/santhosh/Documents/projects/3D-SIS/lib/layer_utils/nms_wrapper.py", line 3, in
from lib.layer_utils.nms.pth_nms import cpu_nms
File "/Users/santhosh/Documents/projects/3D-SIS/lib/layer_utils/nms/pth_nms.py", line 4, in
from ._ext import nms
File "/Users/santhosh/Documents/projects/3D-SIS/lib/layer_utils/nms/_ext/nms/init.py", line 3, in
from ._nms import lib as _lib, ffi as _ffi
ImportError: dlopen(/Users/santhosh/Documents/projects/3D-SIS/lib/layer_utils/nms/_ext/nms/_nms.so, 2): no suitable image found. Did find:
/Users/santhosh/Documents/projects/3D-SIS/lib/layer_utils/nms/_ext/nms/_nms.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x00
/Users/santhosh/Documents/projects/3D-SIS/lib/layer_utils/nms/_ext/nms/_nms.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x00

Thanks for your help!
I test your scannet v2 recently,and the test results of the data you provided are very good, but my own data test prompts:
mAP of classification :0
mAP of mask :0

Do you know how this problem come?

Thank you for your contribution. I have a question, from the semantic label of mesh, how do you label each voxel? Especially in this case, many vertices with different classes correspond to the same voxel

Hi, I have read through the data generation instruction you have provided but still I am confused.

So, I have few indoor 3d data in ply format and I want to 3D-SIS on it? I have tried to go through data generation instruction but still, I am confused.

I only have 3d data in ply format, what would be the baby step?

Hi,
Thanks for sharing such an amazing work, I want to have a try train the model, but the whole Scannet is 1.5T too large to down load, could you please share the Scannet train data?
Best wishes!

when i run build.py in the lib\layer_utils\nms,
This error will occour
cffi.error.VerificationError: LinkError: command 'C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\link.exe' failed with exit status 1107
how can i solve it

Hello，thank you for your contribution.I have a question about enet.Whether pretrained ENet you provide completes training in scannetv2 train set which don’t contain val set？

Thank you for the code. I was able to execute example.sh file however when I try to use the ply file generated I get an error like
RPly: Error reading 'red' of 'vertex' number 0 ] 0% Read PLY failed: unable to read file: example_result/visualization/scene0144_00.ply PointCloud with 142719 points.

I have tried 3 different libraries,Open3d, pyntcloud, and pytorch3D (separately) to visualize the ply file but I keep on getting the same error.
I could visualize the ScanNetv2 ply data with all those 3 libraries BTW.

Checkpoint and dataset has been download.
The torch version I use is changed into 0.4.1.post2, since 0.4.1 will cause error.
Train mode is OK, but when I tried test mode, an error occur and I don't know how to fix.

It shows the following error:
File "......./3D-SIS-master/lib/model/trainval.py", line 91, in test
net.load_state_dict(torch.load(saved_model))
File "....../python3.7/site-packages/torch/nn/modules/module.py", line 719, in load_state_dict
self.class.name, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for ScanNet_Backbone:
Unexpected key(s) in state_dict: "mask_backbone.geometry.0.weight", "mask_backbone.geometry.2.weight", "mask_backbone.geometry.4.weight", "mask_backbone.geometry.6.weight", "mask_backbone.geometry.8.weight", "mask_backbone.geometry.10.weight".

I ran the example.sh as guided but I got an error message:
RuntimeError: storage has wrong size: expected -8935141660703063632 got 110592
Any help will be appreciated :)
�]0;IPython: pointmatching/3D-SIS�Called with args: Namespace(cfg='ScanNet/example', debug=False, epochs=100000, gpu='0', mode='benchmark', num_workers=1, output_dir='example_result/scannet_benchmark_checkpoint', step='1205541', tag=None) Using configs: {'ALLOW_BORDER': 0, 'ANCHORS_TYPE_LEVEL1': 'scannet14_3.txt', 'ANCHORS_TYPE_LEVEL2': 'scannet14_11.txt', 'ANCHORS_TYPE_LEVEL3': '', 'BASE_IMAGE_PATH': 'example_result/scannet_benchmark_example_data/images', 'BATCH_SIZE': 1, 'BG_THRESH': 0.2, 'BG_THRESH_LO': 0.0, 'BIAS_DECAY': False, 'CLASS_BATCH_SIZE': 16, 'CLASS_POOLING_SIZE': 4, 'CLASS_THRESH': 0.5, 'COLOR_MEAN': [0.496342, 0.466664, 0.440796], 'COLOR_STD': [0.277856, 0.28623, 0.291129], 'DEBUG': False, 'DEPTH_SHAPE': [41, 32], 'DISPLAY': 10, 'DOUBLE_BIAS': True, 'FG_FRACTION': 0.5, 'FG_THRESH': 0.5, 'FILTER_ANCHOR_LEVEL1': '', 'FILTER_ANCHOR_LEVEL2': '', 'FILTER_ANCHOR_LEVEL3': '', 'FIRST_TIME_ANCHORS': True, 'FIX_BACKBONE': False, 'FIX_CLASS': False, 'FIX_ENET': True, 'FIX_RPN': False, 'FLIP_TSDF': False, 'GAMMA': 0.1, 'IMAGE_EXT': '.jpg', 'IMAGE_SHAPE': [328, 256], 'IMAGE_TYPE': 'color', 'INIT_TRUNCATED': True, 'INTRINSIC': [[37.01983, 0, 20, 0], [0, 38.5247, 15.5, 0], [0, 0, 1, 0], [0, 0, 0, 1]], 'KEEP_THRESH': 1.0, 'LABEL_MAP': 'datagen/fileLists/nyu40labels_scannet.csv', 'LEARNING_RATE': 5e-06, 'LOAD_BACKBONE': True, 'LOAD_CLASS': True, 'LOAD_RPN': True, 'LOG_TSDF': False, 'MAP_THRESH': 0.5, 'MASK_BACKBONE': 'MaskBackbone', 'MASK_FG_THRESH': 0.5, 'MASK_ONLY_IMAGES': False, 'MASK_THRESH': 0.5, 'MASK_USE_IMAGES': False, 'MAX_IMAGE': 400, 'MAX_VOLUME': 2000000, 'MODE': '', 'MOMENTUM': 0.9, 'NET': 'ScanNet_Backbone', 'NORMALIZE_WEIGHTS': [0.3280746813009404, 3.9644974086960434, 5.459494152836571, 2.241522691584157, 4.820655512680854, 3.565918577548873, 3.538498341919445, 4.636521236560596, 5.445050937449535, 5.079250281008131, 6.2030429647735845, 4.622662494840168, 5.956294301248057, 5.459141107819665, 6.724871661883906, 5.832442848923174, 5.064773947290611, 6.738988357113375, 3.375217918833916], 'NUM_2D_CLASSES': 41, 'NUM_ANCHORS_LEVEL1': 3, 'NUM_ANCHORS_LEVEL2': 11, 'NUM_ANCHORS_LEVEL3': 0, 'NUM_CLASSES': 19, 'NUM_IMAGES': 5, 'NUM_IMAGE_CHANNELS': 128, 'NYUV2_FINETUNE': False, 'ONLY_IMAGES': False, 'PRETRAINED_ENET_PATH': 'example_result/scannet_enet_checkpoint/scannetv2_enet.pth', 'PROJ_DEPTH_MAX': 4.0, 'PROJ_DEPTH_MIN': 0.1, 'RANDOM_NUM_IMAGES': False, 'ROI_THRESH': 0.9, 'RPN_BATCHSIZE': 64, 'RPN_CHANNELS': 256, 'RPN_FG_FRACTION': 0.5, 'RPN_NEGATIVE_OVERLAP': 0.15, 'RPN_POSITIVE_OVERLAP': 0.35, 'SNAPSHOT_KEPT': 0, 'STEPSIZE': [], 'TEST': {'RPN_NMS_THRESH': 0.1, 'RPN_POST_NMS_TOP_N': 200, 'RPN_PRE_NMS_TOP_N': 400}, 'TEST_FILELIST': 'experiments/filelists/ScanNet/v2/example_scene.txt', 'TEST_SAVE_DIR': 'example_result/test', 'TRAIN': {'RPN_NMS_THRESH': 0.5, 'RPN_POST_NMS_TOP_N': 500, 'RPN_PRE_NMS_TOP_N': 1000}, 'TRAINVAL_FILELIST': 'experiments/filelists/ScanNet/v2/trainval.txt', 'TRAIN_FILELIST': 'experiments/filelists/ScanNet/v2/train.txt', 'TRUNCATED': 3.0, 'USE_BACKBONE': True, 'USE_CLASS': True, 'USE_GT': True, 'USE_IMAGES': True, 'USE_IMAGES_GT': False, 'USE_MASK': True, 'USE_RPN': True, 'VAL_FILELIST': 'experiments/filelists/ScanNet/v2/val_chunk.txt', 'VAL_NUM': 20, 'VAL_SAVE_DIR': 'example_result/val', 'VAL_TIME': 2.0, 'VOXEL_SIZE': 0.046875, 'WEIGHT_DECAY': 0.0005} Traceback (most recent call last): File "main.py", line 84, in <module> benchmark(args) File "/home/neil/leondelee/pointmatching/3D-SIS/lib/model/trainval.py", line 71, in benchmark net.load_state_dict(torch.load(saved_model)) File "/home/neil/anaconda3/envs/3dsis/lib/python3.6/site-packages/torch/serialization.py", line 358, in load return _load(f, map_location, pickle_module) File "/home/neil/anaconda3/envs/3dsis/lib/python3.6/site-packages/torch/serialization.py", line 549, in _load deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly) RuntimeError: storage has wrong size: expected -8935141660703063632 got 110592

Hi,
I have recently encountered an issue when I tried to inference on scannetv2_test_data that you provided. However, I have configured everything as you guided, but got an error like this:
[Errno 2] No such file or directory: '/home/neil/leondelee/pointmatching/3D-SIS/data/ScanNet/scannetv2_test_data/images/scene0462_00/depth'
It seems that there is a file called "scene0462_00_0_.scene" in the folder "scannetv2_test_data/scenes" but there isn't a folder like "scene0462_00/" in the folder "scannetv2_test_data/images/".
Any help will be appreciated!

Hi, your work is awesome.

I have been trying to install all the requirement but I am currently having a problem while running the example.sh.

Can you tell me, the python version and the Cuda version for this work?

Thanks. And great work.

Hello，thank yor for sharing your codes.I have a question.
Why use the feature vector of 41x32 size instead of maintaining the original size?

Greetings!

Your example.sh doesn't seem to be working. Some issues I've found:

Called with args:
Namespace(cfg='ScanNet/example', debug=False, epochs=100000, gpu='0', mode='benchmark', num_workers=1, output_dir='example_result/scannet_benchmark_checkpoint', step='1205541', tag=None)
... ...

Traceback (most recent call last):
File "main.py", line 84, in
benchmark(args)
File "F:\3D SIS\code\lib\model\trainval.py", line 64, in benchmark
logger = Logger(os.path.join(log_dir, 'test'))
File "F:\3D SIS\code\lib\utils\logger.py", line 15, in init
self.writer = tf.summary.FileWriter(log_dir)
File "D:\Anaconda3\Anaconda3c\envs\python368\lib\site-packages\tensorflow\python\summary\writer\writer.py", line 336, in init
filename_suffix)
File "D:\Anaconda3\Anaconda3c\envs\python368\lib\site-packages\tensorflow\python\summary\writer\event_file_writer.py", line 67, in init
gfile.MakeDirs(self._logdir)
File "D:\Anaconda3\Anaconda3c\envs\python368\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 368, in recursive_create_dir
pywrap_tensorflow.RecursivelyCreateDir(compat.as_bytes(dirname), status)
File "D:\Anaconda3\Anaconda3c\envs\python368\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: Failed to create a directory: example_result/scannet_benchmark_checkpoint\ScanNet/example\logs\test; No such file or directory

The example.sh code is not working with me

I get these errors:

in Vox2mesh: examples_results/test not found not found directory or file
and in evaluate map: examples_results/benchmark_results not found directory or file

Hope you can help me out.
Thanks

Hi @Sekunde ,
Thanks for your contribution！we were able to get the demo set up and running nicely!But when we tested our own RGBD data, we encountered a problem.We successfully generated the .scene files, but the final result is strange, it seems that the whole is misaligned. Did you use the Alignment in the Scannet code to align the scene when you test?

This is a wonderful work. Thank you for your contribution. This work is very helpful to me. I have a problem. I'm doing research on semantic category segmentation. I want to know how to use “datagen” code to mark semantic categories for voxels of each chunk？（20 class）

when I was running example, I got:

Traceback (most recent call last):
File "main.py", line 84, in
benchmark(args)
File "/home/sidonie/3D-SIS-master/lib/model/trainval.py", line 68, in benchmark
net_module = importlib.import_module("lib.nets.backbones")
File "/usr/lib/python3.5/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 986, in _gcd_import
File "", line 969, in _find_and_load
File "", line 958, in _find_and_load_unlocked
File "", line 673, in _load_unlocked
File "", line 665, in exec_module
File "", line 222, in _call_with_frames_removed
File "/home/sidonie/3D-SIS-master/lib/nets/backbones.py", line 8, in
from lib.nets.network import Network
File "/home/sidonie/3D-SIS-master/lib/nets/network.py", line 11, in
from lib.layer_utils.proposal_layer import proposal_layer
File "/home/sidonie/3D-SIS-master/lib/layer_utils/proposal_layer.py", line 5, in
from lib.layer_utils.nms_wrapper import nms
File "/home/sidonie/3D-SIS-master/lib/layer_utils/nms_wrapper.py", line 3, in
from lib.layer_utils.nms.pth_nms import cpu_nms
File "/home/sidonie/3D-SIS-master/lib/layer_utils/nms/pth_nms.py", line 4, in
from ._ext import nms
File "/home/sidonie/3D-SIS-master/lib/layer_utils/nms/_ext/nms/init.py", line 3, in
from ._nms import lib as _lib, ffi as _ffi
ImportError: /home/sidonie/3D-SIS-master/lib/layer_utils/nms/_ext/nms/_nms.so: undefined symbol: state

Hello there, thanks for your awesome work!

I'm not encountering any problems. Just wondering how to generate the visualization results shown in https://github.com/Sekunde/3D-SIS/blob/master/images/teaser.jpg and the YouTube video.

Are you still using Meshlab? How can I add the bounding boxes and labels to the result files?

Thank you in advance.

I was trying to run the demo, and called
"python main.py --cfg ScanNet/example --step 1205541 --output_dir example_result/scannet_benchmark_checkpoint --mode benchmark --gpu 0" in the shell.
(after set up those 3 src as required in example.sh)

Then I got:
Traceback (most recent call last):
File "main.py", line 84, in
benchmark(args)
File "/home/user/Team/YLLuo/3D-SIS-master/lib/model/trainval.py", line 73, in benchmark
SolverWrapper.benchmark(net, dataloader, logger)
File "/home/user/Team/YLLuo/3D-SIS-master/lib/model/trainval.py", line 682, in benchmark
proj_mapping0, proj_mapping1 = zip(*real_proj_mapping)
ValueError: not enough values to unpack (expected 2, got 0)

I traceback the variables(trainval.py/benchmark()), found:
<real_proj_mapping>==[]
which is loaded from:
<proj_mapping> == [None, None, None,........] (all None's in it)
which is loaded from: < blobs> , and it seems fine...

Any idea help with this?
Thanks!

Thank you for this project! Can your code worked in windows?
I got error in run "python build.py ",
i have already installed the all of requirements.txt.

this is error message:
C:\python\3D-SIS-master\3D-SIS-master\lib\layer_utils\roi_pooling\src\cuda\roi_pooling_kernel.cu.o : fatal error LNK1107: 檔案無效或損毀: 無法在 0x82A8 讀取
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\envs\torch041\lib\distutils_msvccompiler.py", line 519, in link
self.spawn([self.linker] + ld_args)
File "C:\ProgramData\Anaconda3\envs\torch041\lib\distutils_msvccompiler.py", line 542, in spawn
return super().spawn(cmd)
File "C:\ProgramData\Anaconda3\envs\torch041\lib\distutils\ccompiler.py", line 909, in spawn
spawn(cmd, dry_run=self.dry_run)
File "C:\ProgramData\Anaconda3\envs\torch041\lib\distutils\spawn.py", line 38, in spawn
_spawn_nt(cmd, search_path, dry_run=dry_run)
File "C:\ProgramData\Anaconda3\envs\torch041\lib\distutils\spawn.py", line 81, in _spawn_nt
"command %r failed with exit status %d" % (cmd, rc))
distutils.errors.DistutilsExecError: command 'C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\bin\HostX86\x64\link.exe' failed with exit status 1107

When I run: python tools/visualization.py --path /data/fxy/3DSIS/results/ScanNet/rpn_class_mask_5/test --mode results

I can't find the .ply files in the test folder, and the traceback doesn't show errors, it shows below :
Called with args:
Namespace(mode='results', path='/data/fxy/3DSIS/results/ScanNet/rpn_class_mask_5/test/')

Could you please tell me the reasons and what should I do to get the correct results?

Hi @Sekunde ,
Thanks for posting this, we were able to get the demo set up and running nicely, and the output was easy to interface with our ROS-based system. Like a couple of the other posters(#30, #8), we're struggling a bit to understand the workflow of getting from RGB-D camera data (ROS messages, pngs, whatever) into the .scene format in order to run it through the inference system. Could you post a baby-steps version of what you described in the "data generation" README? I think it would help people compare and cite this work more easily. Here is a list of things that we think might need clarification:

1. Generating .scene file from RGBD data, preferably from PNGs, but this is flexible. Ideally, this would be a sequence of command line instructions, including compilations if necessary.
2. How to modify the configuration file (yaml file) after generating the .scene file.

(We could use our existing OctoMap setup to do the voxelization, but it seems like your code already handles the RGBD->.scene process, so we thought we could reuse that.)

Congrats! Great work!
Thank you for making the reproduction process so easy for this repo!

I have a question regarding this line:
https://github.com/Sekunde/3D-SIS/blob/master/lib/layer_utils/projection.py#L103

When we are creating the depth mask at this line, why are we clipping against voxel_size here?

Thank you for your time in advance!

Hello!

We are trying use your model on our own dataset, while we came across some problems about the data we need to prepare as input.
We downloaded data for some scenes from ScanNet v2, and run datagen/prepare_2d_data.py on them. However, we only color/depth/pose/label from that.
We are quite confused about :
1)what do .scene files in scannet_example_data/scenes contain?
2)How can we generate those .scene files for our own dataset?
3)How should we construct the matrix in world2grid.txt in scannet_example_data/images?

Thank you!

according to the screenshot, can i do infer only with a cpu?(my local computer only has a old gpu, like gtx730...)

when I run "python main.py --cfg ScanNet/rpn_class_mask_5 --mode test --step 805146 --gpu 0" it reported the following error:
starting test on whole scan....
0it [00:00, ?it/s]Traceback (most recent call last):
File "main.py", line 81, in
test(args)
File "/home/3d-sis/3D-SIS/lib/model/trainval.py", line 95, in test
SolverWrapper.test(net, dataloader, logger)
File "/home/3d-sis/3D-SIS/lib/model/trainval.py", line 789, in test
for iter, blobs in enumerate(tqdm(data_loader)):
File "/home/miniconda3/envs/3d-sis/lib/python3.6/site-packages/tqdm/_tqdm.py", line 999, in iter
for obj in iterable:
File "/home/miniconda3/envs/3d-sis/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 501, in iter
return _DataLoaderIter(self)
File "/home/miniconda3/envs/3d-sis/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 297, in init
self._put_indices()
File "/home/miniconda3/envs/3d-sis/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 345, in _put_indices
indices = next(self.sample_iter, None)
File "/home/miniconda3/envs/3d-sis/lib/python3.6/site-packages/torch/utils/data/sampler.py", line 138, in iter
for idx in self.sampler:
File "/home/miniconda3/envs/3d-sis/lib/python3.6/site-packages/torch/utils/data/sampler.py", line 34, in iter
return iter(range(len(self.data_source)))
File "/home/3d-sis/3D-SIS/lib/datasets/dataset.py", line 43, in len
return len(self.scenes)
AttributeError: 'Dataset' object has no attribute 'scenes'

by the way I can successfully run demo code

when i run main.py, error
ImportError: cannot import name 'AsyncGenerator'.
how can i sovle it

Traceback (most recent call last):
File "main.py", line 81, in
test(args)
File "/data/fxy/3DSIS/code/lib/model/trainval.py", line 93, in test
SolverWrapper.test(net, dataloader, logger)
File "/data/fxy/3DSIS/code/lib/model/trainval.py", line 820, in test
blobs['proj_ind_2d'].append(torch.stack(proj_mapping1))
RuntimeError: CUDA error: out of memory

I would like to ask whether I must use multi-gpus. I use only one gpu and errors show above.

I created a docker image and hope to run the example on a Mac on CPU. When I run "python main.py --cfg ScanNet/example --step 1205541 --output_dir example_result/scannet_benchmark_checkpoint --mode benchmark --gpu 0", I get the following error:

Traceback (most recent call last):
File "main.py", line 84, in
benchmark(args)
File "/home/Dev/3D-SIS/lib/model/trainval.py", line 70, in benchmark
net.init_modules()
File "/home/Dev/3D-SIS/lib/nets/network.py", line 64, in init_modules
self.image_enet_fixed, self.image_enet_trainable, self.image_enet_classification = enet.create_enet_for_3d(cfg.NUM_2D_CLASSES, cfg.PRETRAINED_ENET_PATH, cfg.NUM_CLASSES)
File "/home/Dev/3D-SIS/lib/nets/enet.py", line 699, in create_enet_for_3d
model.load_state_dict(torch.load(model_path))
File "/home/Applications/miniconda3/lib/python3.7/site-packages/torch/serialization.py", line 358, in load
return _load(f, map_location, pickle_module)
File "/home/Applications/miniconda3/lib/python3.7/site-packages/torch/serialization.py", line 542, in _load
result = unpickler.load()
File "/home/Applications/miniconda3/lib/python3.7/site-packages/torch/serialization.py", line 508, in persistent_load
data_type(size), location)
File "/home/Applications/miniconda3/lib/python3.7/site-packages/torch/serialization.py", line 104, in default_restore_location
result = fn(storage, location)
File "/home/Applications/miniconda3/lib/python3.7/site-packages/torch/serialization.py", line 75, in _cuda_deserialize
raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location='cpu' to map your storages to the CPU.

Is it possible to perform inference on CPU at all? I don't see the environment variable CUDA_VISIBLE_DEVICES used anywhere.

Thank you

When i run main.py
No such file or directory :' experiments/cfgs/classification_rpnloss.yml'
how can i sovle it?

Hi @Sekunde
Thanks for your work and post the code on github, what a wonderful work!
I have a question about the batch size you set when you ran the project. Based on the paper you only used about 24h for 10 epochs, what the batch size you set for the dataloder? And the mask batch size is 16, however I can not find the parameter setting in the config.py file. I ran the project incredibly slow when batch size in 1(GPU is 2080 Ti) about 20h for 1 epoch, and If a increase the batch size it will report error.

Thank you!

hello, I'm studying your thesis with the your code.
Thank you so much for sharing the code so that I can understand your thesis.
But I need a more detailed explanation, because There are many difficulties in running the demo shell.
I followed the instructions, but I couldn't get any results.(ply files that visualizations will be stored in TEST_SAVE_DIR )

1. Should I do this part of inference-on-scannetv2 before I turn on the demo?
2. The downloaded scannetv2_test_datazip does not have an images folder. What exactly should BASE_IMAGE_PATH be? And I'm curious about the structure of BASE_IMAGE_PATH.
   3.I make dir as folder-structure created folders and unzipped file according to this structure. The paths used in cfg for the demo shell were placed in absolute paths along the structure, but they didn't work properly.

If you don't mind, Could you explain in more detail so I can run the demo?

best regards,

Hello, I want to generate the. scene file of my own scene. What software should I use to generate it? Thank you.

I see you're using plyfile for I/O. Using meshio (one of my projects) which recently gained support for PLY files, would extend the range of supported mesh formats by a great deal.

Hi!
I am running test on SUNCG test data (followed the instructions in README/Inference on SUNCG dataset).
Q1: Should in Inference on SUNCG dataset/3, should we modify experiments/cfgs/ScanNet/benchmark.yml or experiments/cfgs/SUNCG/rpn_class_mask_5.yml?

Then I modified both and try to run the test.
Q2: Those images files cannot be accessed since scene_name(defined in dataset.py/151) refers to first chunk(before"0") of scene_name, while all those real scene_names (as in suncg_test_data/images; suncg_test_data/scenes downloaded from given link) include the "0".

Then I tried either modify the directory names for those scenes in suncg_test_data/images or just change the definition of scene_name to include the "0".
Q3:
But then, after "starting test on whole scan..."
I got:
error: no invalid depth
error: no invalid depth
error: no invalid depth
[path to the scene file]:(invalid sample: no valid projection)
and this seems further caused "CUDA error:out of memory" issue

Is there anything I can do to solve this?
Thanks!

When I run "python main.py --cfg ScanNet/example --step 1205541 --output_dir example_result/scannet_benchmark_checkpoint --mode benchmark --gpu 0" from " example.sh.".
Ｉ have the problem:
...
loaded mode from example_result/scannet_benchmark_checkpoint/ScanNet/example/checkpoints/step_1205541.pth
starting test on whole scan....
0%| | 0/5 [00:00<?, ?it/s]/media/zgh/winz/3D/3D-SIS/lib/nets/network.py:546: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
rpn_cls_prob_level1 = F.softmax(rpn_cls_score_level1)
/media/zgh/winz/3D/3D-SIS/lib/nets/network.py:559: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
rpn_cls_prob_level2 = F.softmax(rpn_cls_score_level2)
cudaCheckError() failed : no kernel image is available for execution on the device

I do not how to solve it? and I re-run the build.py in nms and roi_pooling, it does not work.
Can you help me ？　Thank you very much.