pbattaglia / physicsvision Goto Github PK

1. Pushing the 2D detection (vertices) through 3D scene, and searching out the 3D support and contact points.
2. Operation: Add/Delete, Change depth(size), Stretch down to create/satisfy support.

1. A good evaluation set
2. how to go beyond the block world revisit paper

Our plan is to add new functionality to Yibiao's existing visual parsing module that uses physical constraints to improve 3D scene interpretations. The visual parser will use physical constraints in 2 ways: 1) by respecting a "physics prior", in which objects that are unstable or interpenetrating other objects are penalized, and 2) by incorporating physically plausible proposals into the parser's search scheme, in order to invest the bulk of the effort on scene states that will not be penalized by the physics prior. Both of these uses of physical constraints capitalize on a forward simulation of physical dynamics, using a computerized physics engine (Bullet).

We have three goals for this week:

1. Penalize the parsing results using the feedback of physics engine. We plan to simply add a physical term (e.g. an exponential function of displacement/velocity) to the energy of posterior. That term is derived by inputting a scene proposal to the physics engine, and asking it to calculate the objects' velocities and transformations for each time step. These data will then be used to compute a summary score that reflects how physically plausible the scene is.

*** Tentative
When evaluating the physics score, we can factor the penalize on a per-object basis. Then, subsequent physical-adjustment proposals will selectively target those objects that scored poorly in physical plausibility.

1. Generate 3D proposals: In order to make more meaningful 3D proposals regarding objects' arrangements in the scene, we will have three operations (transition kernels) that modify existing scene state samples in concert with physical constraints:
   a. add/delete an object
   b. change its depth (and size simultaneously) according to 3D contact detection
   c. change the local location/pose/scale
   d. Stretch an existing object down to create/satisfy support, or insert an invisible object as a support for an existing object.

Proposal details:
a. Given the parser's existing 2D detection, we will develop a physical contact detection function, which pushes the parser's 2D detection through 3D scene and searches out the 3D support and contact points. This "push" will be aligned with a conical projection outwards from the camera, which inscribes the object, and allows the proposal to select a z-coordinate for which the object is supported by (contact from below) or attached to (contact from a side), an existing object in the scene. The delete move is a reversal of an add, for reversibility of the sampler.
b. For existing objects in the scene, their depth can be modified using using a similar procedure as (a).
c. For existing objects, we can change the location or pose by "bumping" the object, so that the resultant proposed state is at a physically plausible static equilibrium.
d. We assume the floating blocks are very unlikely (with probability 0). The valid proposal should be either supported from its bottom, either itself or by a second, lower object, or be attached to another object in the scene.

1. Evaluate our algorithm:
   The methodology:
   a. Evaluate the 3D detection rate by finding correspondences between parsing results and ground truths
   b. Evaluate the 2D segmentation accuracy by comparing 2D projected segmentations of inferred sample scenes with 2D projected segmentations of ground truth, where each projection is taken from various camera angles around the scene.

   The dataset:
   We can generate the synthesized red/yellow scene and tower scene with ground truth.
   We also plan to build up some real block scene with toy wood blocks, and take pictures of them. They can be used in the paper to evaluate qualitatively.