Is there a relationship between video encoding standards and video compression in video generation? about minisora

[fanqiNO1]

On the one hand, the video compression model takes the latest decoded frame as a reference when encoding and decoding current frame, and the architecture of the video compression model is similar to the Variational Auto-Encoder (VAE). Therefore, if we replace the entropy model and byte stream part with a generative model, such as DiT, then the process of video generation will occur in the latent space of the video compression encoder. At the same time, the whole model can be regarded as an autoregressive video generation model referenced only to the previous generated frame.

On the other hand, as another typical spatio-temporal consistent model, the video compression model includes spatio-temporal information without relying on 3D convolution or spatio-temporal patches, although this is not in line with the technique mentioned by Sora in the technical report. But the method of combining spatio-temporal information to achieve the spatio-temporal consistency is still part of the learning process. Meanwhile, a video generation model based on the video compression model architecture may also be an alternative to Sora in low-computing-power scenarios.

I hope my answer helps.

[chg0901]

I will provide a brief answer to your question. Prior to the use of deep learning or AGI for video generation, researchers employed these methods for video compression. In fact, the design of some video compression modules also mimics these standards to a certain extent.

Could you please answer his question in details or in other aspects @fanqiNO1 ?