Comments (10)
You are correct. Here's how I did it:
def placeholder_inputs_rgb(batch_size, num_point):
pointclouds_pl = tf.placeholder(tf.float32, shape=(batch_size, num_point, 3))
pointclouds_rgb_pl = tf.placeholder(tf.float32, shape=(batch_size, num_point, 6))
labels_pl = tf.placeholder(tf.int32, shape=(batch_size,num_point))
return pointclouds_pl, pointclouds_rgb_pl, labels_pl
def get_model(point_cloud, point_cloud_rgb, is_training, bn_decay=None):
""" Segmentation PointNet, input is BxNx3 and BxNx6, output BxNx13 """
batch_size = point_cloud.get_shape()[0].value
num_point = point_cloud.get_shape()[1].value
end_points = {}
with tf.variable_scope('transform_net1') as sc:
transform = input_transform_net(point_cloud, is_training, bn_decay, K=3)
point_cloud_transformed = tf.matmul(point_cloud, transform)
input_image = tf.expand_dims(point_cloud_transformed, -1)
net = tf_util.conv2d(input_image, 64, [1,3],
padding='VALID', stride=[1,1],
bn=True, is_training=is_training,
scope='conv1', bn_decay=bn_decay)
net = tf_util.conv2d(net, 64, [1,1],
padding='VALID', stride=[1,1],
bn=True, is_training=is_training,
scope='conv2', bn_decay=bn_decay)
with tf.variable_scope('transform_net2') as sc:
transform = feature_transform_net(net, is_training, bn_decay, K=64)
end_points['transform'] = transform
net_transformed = tf.matmul(tf.squeeze(net, axis=2), transform)
point_feat = tf.expand_dims(net_transformed, [2])
point_cloud_rgb = tf.expand_dims(point_cloud_rgb, [2])
concat_rgb = tf.concat(axis=3, values=[point_feat, point_cloud_rgb])
net = tf_util.conv2d(concat_rgb, 64, [1,1],
padding='VALID', stride=[1,1],
bn=True, is_training=is_training,
scope='conv3', bn_decay=bn_decay)
from pointnet.
@vaidarnav that's surely a reasonable way to do it! You can experiment with different variations though to see the different and gain more insights.
@Logrus Thanks for sharing your code :) looks good to me
from pointnet.
Hi @soulslicer
RGB or normals can be included in the network. The most naive way is to add more channels to input points -- currently it's 3-channel of XYZ, you can make it 9-channel as XYZ, RGB and normal.
From our experience the network is quite robust to inconsistent sampling density :)
from pointnet.
I'm also interested in adding rgb and normals to the network. Right now the n-point input has dimension (n X 3 X 1), would we need to change it to (n X 9 X 1)? Or would we add rgb and normal data as separate channels and make it (n X 3 X 3)
from pointnet.
Hi @vaidarnav, you should add all channels that you want as you described, like (n X 9 X 1). You also need to change the dimension of the first kernel to account for this.
from pointnet.
Thanks @Logrus .
Another question I had was about the affine transformation step involving the T-net. As far as I understand, the input of this network should only be the xyz data (of dim n X 3), and the output (dim 3 X 3) should only be multiplied with the xyz data, not affecting the rgb or normal values. Is this correct?
from pointnet.
@vaidarnav, that is correct. You should have a couple of placeholders (one nX3 for geometry and another nX6 for rgb+normals). Then you apply couple of the transformer networks on geometry and concatenate nX64 with your rgb placeholder and get nX70. After that, the usual architecture follows.
Unfortunately in this repo there's no example of this, but it isn't hard to do.
By the way, for the dataset from paper (building parsing) spacial transformer nets don't help much, but I saw +6% MIoU increase in another dataset where the rotations were very important. So keep that in mind.
from pointnet.
So, to be completely clear, you should feed XYZ from "input transform" all the way to "feature transform", and then append RGB/norm data to the output of "feature transform"?
from pointnet.
Hello @charlesq34 , very nice work! I'm wondering is the archirecture (global pooling + local transformation + dimension expansion) able to learn the geodesic distance between any pair of points in a point cloud, i.e., isometric embedding in a parametric manner? You showd a demo of normal vector prediction in the paper, but the success seems almost gauranteed to me since normal is a rather local quantity, while geodesics is way more global? Anyway, I'm not sure about this, and appreciate your comments very much!
from pointnet.
Honestly I'm not sure how well the network can perform in learning geodesic distance. It's a very interesting direction though -- I can imagine some siamese network with geodesic loss will be a good starting point for the project :)
from pointnet.
Related Issues (20)
- meshlab
- PointNet for angle regression
- Area_5/hallway_6 HOT 1
- Issued certificate has expired HOT 1
- Annotation HOT 1
- A lightweight Cylinder3D model with much higher performance is now available!!!
- Isn't this section incorrect? HOT 1
- __init__() missing 1 required positional argument: 'dtype'
- How much memory needed for sem_seg training HOT 2
- How to visualize the semantic segmentation results through ROS
- how to use PointNet model in live inference HOT 1
- cant download dataset with HDF5 data. Help! HOT 1
- Segment point clouds with different point numbers
- Cannot get modelnet40 from server HOT 3
- Conv2D x Conv1D
- Input transform and point cloud with features on the points
- Is normalization to unit sphere mandatory?
- Tensorflow-TensorRT Integration error
- How does the the T-Net achieve translation invariance?
- how to use cpu
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pointnet.