Physion inference with less than 8 condition frames about vdt

[aweitz]

Thank you for the detailed response!

I think you could try the application of our unified mask modeling, than VDT is capable of performing zero-shot extrapolation across any spatial-temporal dimension.

To clarify, the unified mask approach must be applied during training to enable the various tasks (including zero-shot extrapolation) during inference, correct? Do you plan to release details on how you modulated the spatial-temporal mask (e.g. probability of frame dropout, etc.)?

[RERV]

Thank you for your interest in our VDT. The observed decrease in performance when fewer frames are used as a condition stems from the fact that our released model was only trained with a fixed number of frames (8 frames) for conditioning. And we have discovered that it is feasible to extend the model's capabilities to conditions involving more than 8 frames, as demonstrated in Appendix Figure 8. The term "any length" may have been somewhat ambiguous; it specifically refers to any quantity exceeding 8 frames. We have made revisions to clarify this point.
For second question "Would the model need to be trained with only 1 conditioning frame", I think you could try the application of our unified mask modeling, than VDT is capable of performing zero-shot extrapolation across any spatial-temporal dimension.

[aweitz]

Additionally, can you please clarify if the unified mask modeling is applied in the image or latent (VAE) space? It seems your notation suggests it is applied in the image domain (${\mathcal{M} \in R^{F \times H \times W \times C}}$). Thank you!

[RERV]

Hi everyone, my apologies for the late reply. I was quite busy earlier and couldn't get to it. I've now updated the mask modeling, and you can find the necessary code in it. Have fun!